Word segmentation method and device

Abstract

The invention discloses a word segmentation method and a word segmentation device, and belongs to the field of information processing. The method comprises the following steps of performing word segmentation on a text to be subjected to the word segmentation to obtain a first text by using an nth-order Markov model; when the first text comprises a target word string, adding the target word string to a dictionary to obtain an updated dictionary, wherein the target word string is not stored in the dictionary, and the dictionary is used for storing all word strings and corresponding estimated probabilities; performing word segmentation on the first text to obtain a second text and a third text according to the updated dictionary and a preset algorithm by utilizing a forward maximum matching word segmentation method and a backward maximum matching word segmentation method respectively; selecting the text of which the word length expectation and the word length variance are consistent with preset rules from the second and third texts as a word segmentation result. According to the method, a conventional dictionary is updated only by adding new words into the conventional dictionary, so that the new words can be flexibly added on the premise of not increasing the word segmentation ambiguities.

Description

technical field [0001] The invention relates to the field of information processing, in particular to a word segmentation method and device. Background technique [0002] Chinese word segmentation refers to dividing a sequence of Chinese characters into individual words. Chinese word segmentation plays an important role in the fields of information retrieval, machine translation and speech recognition, and is an indispensable link in the process of Chinese speech processing. Generally, due to the problem of word segmentation ambiguity, the accuracy of traditional dictionary-based mechanical word segmentation methods cannot reach 100%. For example, "Nanjing Yangtze River Bridge" can be divided into "Nanjing Yangtze River Bridge", and can also be divided into "Nanjing Yangtze River Bridge". Both word segmentation methods seem reasonable if you do not rely on other knowledge. [0003] In order to solve the above problem of word segmentation ambiguity, in the prior art, an n-...

AI Technical Summary

Problems solved by technology

[0005] If you want to add new words to the n-gram language model, you need to estimate the probability for the new words, which leads to the need to collect corpus containing new words and rebuild the n-gram language model, which has certain limitations.

Images