Chinese word segmentation algorithm based on reverse maximum matching

Abstract

The invention discloses a Chinese word segmentation algorithm based on reverse maximum matching, which comprises the following steps: initializing three objects in a memory; inputting the contents of a text which needs word segmentation; splitting characters in the text into different types according to character codes; directly adding characters which are not Chinese characters to word segmentation results according to the character codes after the text is segmented into short sentences; splitting the short sentences into character sets according to a character string matching and decision-making mechanism; matching the character sets with character sets in a word segmentation dictionary based on the reverse maximum matching algorithm; storing matched character sets into a word segmentation result set; combining consecutive unmatched characters; and adding the consecutive unmatched characters to the word segmentation results to complete word segmentation. A quick word segmentation algorithm based on dictionaries is provided, and the dictionary loading efficiency and the word segmentation efficiency are greatly improved while word segmentation accuracy is ensured.

Description

technical field [0001] The invention relates to a text analysis technology in the field of artificial intelligence, in particular to a classification technology for data mining in the field of artificial intelligence, which is applied to functions such as search engines and data mining in Internet products. Background technique [0002] Today, when the amount of information is soaring and gradually showing a trend of bursting, the Internet industry, which receives and disseminates the largest amount of information, has been plagued by a problem, that is, how to make users quickly respond to the colorful information on the website? Accurately search and locate the resources you need. Currently widely used in Internet products is the Chinese word segmentation technology, which splits a piece of text into multiple words by splitting and matching dictionaries to help computers "understand" the core content of the text. For example, the realization of functions such as search en...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a Chinese word segmentation algorithm based on reverse maximum matching, aiming at problems such as low word segmentation accuracy, inaccurate recognition of unregistered words, and low performance in the process of searching and locating resources required by products on the Internet.

Images