Chinese word segmentation method and device, electronic equipment and storage medium

Abstract

The invention provides a Chinese word segmentation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed text sequence which comprises a plurality of characters arranged in sequence; extracting a feature vector corresponding to each character in the to-be-processed text sequence to obtain a feature vector group; mapping each feature vector in the feature vector group into a two-dimensional vector, with the two-dimensional vector comprising a first dimensional value and a second dimensional value; and determining whether the corresponding character interval is a word boundary or not through the first dimension value and the second dimension value. The multi-classification problem of Chinese characters and words is simplified to the two-classification problem of word boundaries, that is, the character interval is the word boundary or not the word boundary, so that when the system performs word segmentation on the to-be-processed text sequence, the calculated amount is greatly reduced, the consumed time is naturally and greatly shortened, and the working efficiency of whole Chinese word segmentation processing isimproved.

Description

technical field [0001] The invention relates to the field of natural language processing, in particular to a Chinese word segmentation method, device, electronic equipment and storage medium. Background technique [0002] As one of the basic tasks in natural language processing, Chinese word segmentation is an essential and key preprocessing link for many natural language processing tasks. The performance of the results will directly affect the final performance of subsequent tasks. Therefore, processing Chinese word segmentation accurately and efficiently can effectively help other Chinese natural language processing tasks. [0003] In recent years, with the development of neural networks, sequence tagging models based on deep learning have made breakthrough progress in the performance of Chinese word segmentation, but many sequence tagging models have too high time complexity when processing Chinese word segmentation, for example, Characters are usually classified into: s...

AI Technical Summary

Problems solved by technology

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the excessive time complexity defect in word segmentation processing in the prior art, thereby providing a Chinese word segmentation method, including the following steps: obtaining the text sequence to be processed, the text sequence to be processed Including a plurality of characters arranged in sequence; extracting the feature vector corresponding to each character in the text sequence to be processed to obtain a feature vector group; mapping each feature vector in the feature vector group to a two-dimensional vector, wherein , the two-dimensional vector includes a first dimension value and a second dimension value, the first dimension value can affect the judgment parameter that the corresponding character interval is not a word boundary, and the second dimension value can affect the corresponding character interval as The judgment parameter of the word boundary; determine whether the corresponding character interval is a word boundary by the first dimension value and the second dimension value

Images