Method, apparatus, storage medium and apparatus for generating network representation of neural network

Abstract

The invention relates to a network representation generation method, an apparatus, a storage medium and an apparatus of a neural network. The method comprises the following steps: obtaining a source vector representation sequence corresponding to an input sequence; a linear transformation being carried out on the source vector representation sequence to obtain a request vector sequence, a key vector sequence and a value vector sequence correspond to the source vector representation sequence; calculating a logical similarity between a request vector sequence and a key vector sequence; constructing a local reinforcement matrix according to a request vector sequence; based on logical similarity and local reinforcement matrix, the distribution of attention weights of local reinforcement corresponding to each element being obtained by nonlinear transformation. According to the distribution of attention weights, the value vectors in the value vector sequence are fused to obtain the network representation sequence corresponding to the input sequence. The scheme provided herein generates a network representation sequence that not only enhances local information, but also preserves connections between long-distance elements in the input sequence.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a neural network network representation generation method, device, computer-readable storage medium and computer equipment. Background technique [0002] Attention Mechanism (Attention Mechanism) is a method for building a model for the dependency relationship between the hidden state of the encoder and the decoder in the neural network. The attention mechanism is widely used in deep learning-based natural language processing (NLP, Natural Language Processing) in each task. [0003] SAN (Self-Attention Network, self-attention neural network) is a neural network model based on the self-attention mechanism, which belongs to a kind of attention model, and can calculate an attention weight for each element pair in the input sequence, so that Long-distance dependencies can be captured, and the network representation corresponding to each element will not be affected by th...

AI Technical Summary

Problems solved by technology

[0004] Based on this, it is necessary to provide a neural network network representation generation method, device, Computer-readable storage media and computer equipment

Images