Chinese text data word vector representation method based on BIE position word list

Abstract

The invention relates to a Chinese text data word vector representation method based on a BIE position word list, and relates to the field of deep learning and named entity recognition, and the method comprises the following steps: S1, generating a total word set and a strong correlation word set, and constructing the BIE position word list; s2, constructing a position-independent word vector by using the original representation of the word vector; s3, condensing word vector representations in the word set based on a word frequency weighted average pooling algorithm; and S4, weighting the BIE position word vector of the word and splicing the weighted BIE position word vector with the original word vector to generate a word vector containing vocabulary position information. According to the method, the position information of the highly correlated vocabularies can be highlighted while the total position information of the vocabularies is fused into the word vectors. And the character vector representation dimension is expanded, so that the Chinese entity recognition result has higher accuracy.

Description

technical field [0001] The invention belongs to the field of deep learning and named entity recognition, and relates to a Chinese text data word vector representation method based on a BIE position word list. Background technique [0002] Named Entity Recognition (NER) is a basic work in the field of natural language processing, and is a subtask of tasks such as information retrieval, relation extraction, and question answering systems. Unlike natural word segmentation in English text, the number of characters in a word in Chinese text is not fixed, and there is no word segmentation identifier. This makes it difficult to learn lexical boundary information for Chinese named entity recognition tasks. Therefore, it is necessary to incorporate lexical position information into the Chinese word vector representation to improve the accuracy of Chinese named entity recognition. [0003] At present, the widely used vocabulary enhancement methods are vocabulary enhancement methods ...

AI Technical Summary

Problems solved by technology

However, the effect of lexical information integration is strongly related to the Embedding strategy. The WC-LSTM structure has the problem of lexical information loss, and the Multi-digraph structure relies on dictionary labels, making it difficult to be widely used.

Images