Алгоритм виявлення дублікатів вакансій на основі семантичної схожості

Abstract

The paper explores the problem of intelligent filtering and data deduplication of job vacancies collected from unstructured sources. The algorithm for semantic analysis of job postings using high-dimensional numerical vectors is considered. A solution based on the OpenAI cloud API for generating embeddings and the pgvector extension for PostgreSQL DBMS is proposed, which allows detecting hidden duplicates of vacancies using the cosine similarity metric. Experimental validation showed a reduction in information noise by almost twice.