Text word segmentation processing method and device, equipment and medium

Abstract

The invention discloses a text word segmentation processing method and device, equipment and a medium. The method comprises the following steps: collecting a text to be subjected to word segmentation, the text to be subjected to word segmentation comprises a plurality of suspected words which are connected in series, and the suspected words are composed of pronunciation characters; all characters in the text to be subjected to word segmentation are sequentially traversed, redundant characters formed by continuous repetition in the suspected words are ignored in the traversing process, the redundant characters are converted into words in a dictionary tree diagram, the words are sequentially added into a result list, the dictionary tree diagram comprises a plurality of paths starting from a root node of the dictionary tree diagram and respectively reaching different tail end nodes, and the word sequence of the dictionary tree diagram is obtained; nodes through which each path passes store each character of the single word in sequence; and outputting the words in the result list in sequence as word segmentation results. According to the word segmentation device, word segmentation processing is carried out according to the tree diagram, abnormal repeated characters can be processed in the word segmentation process, redundant characters in the text to be subjected to word segmentation are ignored, and words contained in the text are extracted accurately and accurately.

Description

technical field [0001] The present application relates to the technical field of computer word segmentation, in particular to a text word segmentation processing method, and also relates to the corresponding device, equipment and non-volatile storage medium of the method. Background technique [0002] Word segmentation, that is, splitting text paragraphs or sentences into several words, is one of the most basic parts in text-related "Natural Language Processing" (NLP), and it also plays a very important role. Accurately and quickly obtaining the word-segmented results will help ensure the accuracy of subsequent NLP and improve work efficiency. [0003] After several tokens are obtained based on the format of the text itself (such as spaces between words) and punctuation marks, repetitions are often introduced between multiple words (such as He-Who-Must-Not-Be-Named) and normal words The letters (such as g-o-o-o-o-o-d) will not be reasonably segmented or restored to normal w...

AI Technical Summary

Problems solved by technology

[0006] First, it is not efficient to directly match and search for a word in a dictionary list with a large number of elements. At the same time, it is not efficient to enumerate all possible word combinations and then search sequentially.

[0007] Second, traversing from the back to the front, the time complexity is at least O(n2) level. As the length of the token continues to increase, the time to extract the segmented words will increase significantly, which will greatly affect the overall work efficiency.

Images