Method and device for recognizing ambiguous words with combinatorial ambiguities

Abstract

The invention discloses a method and device for recognizing ambiguous words with combinatorial ambiguities. The method includes the steps that coarse-grained segmentation is conducted on first-language sentences through a core word list; candidate ambiguous words capable of being split into a plurality of words of smaller granularity are detected from word segmentation results through the core word list after coarse-grained segmentation is finished; fine-grained segmentation is conducted on the first-language sentences by splitting the candidate ambiguous words; translations of the candidate ambiguous words and translations of the words, of the smaller granularity, formed by splitting the candidate ambiguous words are respectively extracted from second-language sentences corresponding to the first-language sentences; whether the extracted translations of the candidate ambiguous words and the extracted translations of the words, of the smaller granularity, appear in translations, of the candidate ambiguous words and the smaller-granularity words, acquired through a first-language dictionary and a second-language dictionary or not is judged to determine whether the candidate ambiguous words are true ambiguous words or not.

Description

technical field [0001] The present application generally relates to the field of natural language processing, and in particular to a method and device for identifying ambiguous words with combination ambiguity. Background technique [0002] In natural language processing, word segmentation is one of the basic topics. Most natural language processing is based on the results of word segmentation, so the quality of word segmentation directly affects the accuracy of subsequent work. Due to the characteristics of natural language itself, in the process of word segmentation of natural language, the problem of word segmentation ambiguity will be encountered. Taking Chinese as an example, Chinese word segmentation ambiguity mainly includes the following two types: intersection ambiguity and combination ambiguity. Generally, assuming that A, X and B are word strings respectively, if the word string AXB composed of them satisfies the condition that AX and XB are words at the same ti...

AI Technical Summary

Problems solved by technology

However, existing solutions have many disadvantages

For example, it is difficult to manually summarize the potential ambiguity in the corpus, and the manual method requires a lot of time and manpower, and the cost is very high

Moreover, combined ambiguity has domain-related characteristics, that is, different ambiguities will occur in different domains, and it is difficult to predict artificially.

In addition, manually labeling the corpus is also a costly process. If the word segmentation system is applied in a new field, it is necessary to re-label the new corpus

Images