Activation function parameterization improvement method based on recurrent neural network

Abstract

The invention discloses an activation function parameterization improvement method based on a recurrent neural network, and the method comprises the steps: step 1, constructing a bidirectional long short-term memory network Bi-LSTM on the basis of a long short-term memory network; step 2, connecting all hidden layers in the Bi-LSTM network in series, adding an average pooling layer behind the lasthidden layer in the network, connecting a normalized exponential function layer behind the average pooling layer, and establishing a densely connected bidirectional long short-term memory network DC-Bi-LSTM; and step 3, training on the data set by applying the parameterized Sigmoid activation function, recording the sentence classification accuracy of the densely connected bidirectional long-short-term memory network, and obtaining the parameterized activation function corresponding to the optimal accuracy. According to the invention, through the parameterized activation function module, theunsaturated region of the S-shaped activation function is expanded, the derivative of the function is prevented from being too small, and the gradient disappearance phenomenon is prevented.

Description

technical field [0001] The present invention relates to the technical field of natural language processing and text classification, in particular to an activation function parameterization improvement method based on a recurrent neural network. Background technique [0002] Deep neural networks are widely used in computer vision, however, stacked recurrent neural networks suffer from vanishing gradient and overfitting problems. Therefore, on this basis, some new types of cyclic neural networks have emerged, and the bidirectional long-term short-term memory network based on dense connections is a very effective cyclic neural network. [0003] The activation function module is a basic module of the neural network. In general, activation functions have the following properties: [0004] (1) Nonlinearity: Almost all functions can be expressed by a two-layer neural network under the condition that the activation function is nonlinear; [0005] (2) Differentiability: The optimi...

AI Technical Summary

Problems solved by technology

[0008] In order to solve the deficiencies in the prior art, the present invention provides an activation function parameterization improvement method based on a recurrent neural network, which solves the two-way saturation characteristic of the S-type activation function, which easily leads to the disappearance of the gradient and cannot effectively correct the weight value, and in When performing reverse error propagation, it is easy to cause the activation function to fall into the saturation region, causing the gradient to disappear, which increases the difficulty of training the recurrent neural network.

Images