Key-based segmentation method and device for character sequences

Abstract

The invention relates to a key-based segmentation method and a device for character sequences. The method includes loading keys and establishing a key list; sequencing the keys according to priority of the key attributes in the key list; and according to the sequencing result and taking the key as the smallest segmentation unit, and performing segmentation on the character sequences according to predetermined word segmentation. According to the method and the device, the priority of the keys is defined, the key list is established, the sequence is formed according to the priority of the keys in the key list, the key serves as the smallest segmentation unit, the segmentation is performed on the character sequences according to predetermined word segmentation, therefore word segmentation accuracy and new word identifying ability are improved, word segmentation ambiguity is avoided, and requirements of application scenes such as formula, function verification and analysis which have a strict requirements for word segmentation can be satisfied.

Description

technical field [0001] The present invention relates to the technical field of character sequence word segmentation, in particular to a keyword-based character sequence segmentation method and device. Background technique [0002] Currently commonly used word segmentation algorithms mainly fall into the following three categories: [0003] 1. Word segmentation based on string matching; 2. Word segmentation based on understanding; 3. Word segmentation based on statistics. The comparison results of the above three branch algorithms are shown in Table 1 below: [0004] word segmentation method Word segmentation based on string matching participle based on comprehension word segmentation based on statistics Ambiguity recognition Difference powerful powerful new word recognition Difference powerful powerful need a dictionary need unnecessary unnecessary need corpus no no ...

AI Technical Summary

Problems solved by technology

[0007] 1. The accuracy of word segmentation cannot meet the requirements: For some applications that require very high accuracy of word segmentation, such as expressions and formula word segmentation, the accuracy of word segmentation must reach 100%, otherwise it will cause errors in calculation results

[0008] 2. There is ambiguity in word segmentation: there is ambiguity in the recognition of words

For example, in formulas or expressions, when sequences such as +, +=, if, else if, etc. appear, they cannot be recognized well

[0009] 3. Insufficient ability to recognize new words: When new entries appear in the system, the algorithm cannot recognize new words well.

Images