Algorithms for searching and analysing information from open sources in the context of cyber threats

Abstract

The article presented the development of an algorithm for searching and analysing information open sources in the context of cyber threats. The proposed algorithm is an effective tool for detecting, monitoring, assessing and neutralising threats in the digital environment. The work described the main stages of the algorithm, which include: quick access to relevant information, assessment of data reliability, trend analysis, identification of connections between objects, and prediction of potential threats. The development of the algorithm involved searching for and analysing information open sources; noise filtering; contextual analysis and cross-checking to improve the reliability of results; and constructing relationship graphs to identify dependencies between objects and determine their potential danger. These tasks were implemented by collecting data through API (Application Programming Interface), web scraping (BeautifulSoup), using search operators, processing data with NLP (Natural Language Processing) tools, and classification using machine learning models and regular expressions. The article analysed the information obtained using relationship graphs, identifies key objects and evaluates the reliability of sources. The developed algorithm reduced the time required for searching and analysing information, increased the relevance and accuracy of the data obtained, and provides effective support for cybersecurity decisions. As an example, the algorithm was applied to monitor suspicious job postings on the LinkedIn platform, phishing ads containing invalid links or false information were detected. The use of the LinkedIn API and web scraping made it possible to automate the collection of job postings and compare them with a database of known phishing websites. The developed algorithm reduced the time spent searching for and analysing information compared to manual methods. The implementation of such solutions helps prevent cyber threats and ensure security in the digital environment. The algorithm significantly improves the efficiency of working with open sources, providing an automated process for collecting, processing, and analysing data for further threat assessment.