Method and system for finding out new words

Abstract

The invention provides a method and system for finding out new words. The method comprises the following steps of: based on a bigram language model, respectively extracting bigram elements of a foreground corpus; respectively obtaining statistical information of the foreground corpus; filtering the bigram elements according to the statistical information and a first pre-set rule; expanding the remained bigram elements in the foreground corpus by using an n-gram language model and a second pre-set rule, wherein re-counting a background corpus is unnecessary during the updating of n-gram elements; preventing from re-finding out existing new words in the background corpus; judging boundaries of the new words according to the second pre-set rule; and removing garbage bigram elements and n-gram elements. The method is used simply and easily. The manual correction burden is reduced.

Description

technical field [0001] The invention relates to the field of text information processing, in particular to a new word discovery method and system. Background technique [0002] Chinese (and Asian languages ​​such as Japanese) do not use spaces to denote word boundaries like Western languages, so word segmentation is the primary task of Chinese language processing. However, with the rapid development of Internet content services (such as Weibo, novels), new words on the Internet continue to emerge, and the word segmentation models used in systems such as Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and search engines need to be constantly updated. Neologisms are never outdated, so neologism discovery has recently become a research hotspot. At present, there are roughly three problems faced by new word discovery: [0003] 1. Lack of effective basis. There is no clear definition of new words at present. In the prior art, dictionaries (as background corpus) are g...

AI Technical Summary

Problems solved by technology

However, since the composition of new words varies widely, in many cases there is no general rule, such as the translation of names in novels, magic names, racial names, and dictionaries and rules often fail to achieve good results.

[0004] 2. Word boundaries are difficult to determine

[0005] 3. Difficult to update

New word discovery that relies on word segmentation often requires combining the background corpus and foreground corpus to retrain the word segmentation model to discover new words on a rolling basis. The process is complicated and time-consuming

Images