Проблематика консистентності великих мовних моделей при навчанні з підкріпленням

Abstract

Reinforcement learning (RL), particularly methods like RLHF and DPO, has become the standard for creating agents based on large language models (LLMs). However, these approaches face two fundamental problems: inconsistency in agent behavior and significant computational inefficiency. As an alternative to current approaches, training based on multidimensional rewards and differentiated critique is proposed. This approach enables targeted policy updates, systematically correcting specific aspects of behavior. This not only enhances agent consistency but also radically improves computational efficiency, as each training iteration is significantly more information-dense. Thus, this opens the path toward creating more reliable, logically consistent, and training-efficient autonomous agents.