Архітектура каскадних парсерів гетерогених джерел даних для автоматизованої побудови датасетів для інтелектуальних систем моніторингу

Abstract

The paper presents the architecture of cascade parsers of heterogeneous data sources for automated construction of datasets of intelligent monitoring systems. It is proposed to combine disparate input data by types of information sources (web resources in search results, static reporting files and integration via API with thematic data providers) in a single pipeline for data collection, normalization and validation. For each type of data, its own strategy for creating a pipeline for preparing data to a unified format suitable for further analytical processing is provided.