Logarithmic transformation of input activations in convolutional network neurons: An approach and analysis of effectiveness using MNIST classification as an example

Abstract

Convolutional neural networks attract considerable attention because they can learn directly input data, automatically forming the features required for classification. The purpose of the study was to improve the architecture of a convolutional neural network by applying a logarithmic transformation of input activations in the neurons of convolutional layers and examining its influence on classification accuracy and the robustness of the neural network to changes in image brightness. The mathematical model of a modified neuron was considered; it evaluates the input signal through a logarithmic transformation of the form γ ·ln(x+1), γ is an additional trainable parameter that determines the degree of nonlinear signal compression. This approach increased the sensitivity of the neuron to relative changes in input signals, reduces the influence of large activation values, and can improve the stability of the training process. An analysis of gradients for the trainable parameters of the neural network was conducted within the backpropagation method. The analysis indicated that logarithmic transformation changes the pattern of weight s, strengthening the gradient at small signal values and weakening it at large values. Particular attention was paid to the γ parameter, which acts as a global scale coefficient for the neural layer and enables the network to adjust the level of nonlinearity independently. Experimental verification of the effect of logarithmic transformation of input activations in convolutional layers was conducted using the MNIST handwritten digit classification task with a LeNet-like architecture. The experimental results demonstrated that, for standard images, the logarithmic model provides only a slight increase in accuracy compared with the model with classical linear neurons, averaging 99.17% versus 99.11%. However, for the same images with reduced brightness, the logarithmic model demonstrated high robustness and much higher accuracy. In particular, when brightness was reduced sevenfold, classification accuracy for the model with classical neurons decreased on average to 27.11%, as the logarithmic model maintained a level of approximately 90.98%. The results confirmed the appropriateness of using logarithmic transformation as a mechanism for improving the stability of neural networks in computer vision tasks under insufficient or variable illumination.