Method and system for realizing cross-domain Chinese text error correction

Abstract

The invention provides a method for realizing cross-domain Chinese text error correction. The method comprises the following steps: carrying out error detection by adopting a sequence labeled error detection model and combining a supervision data training model of a general field; performing error retrieval in the pinyin library of the word list through the editing distance or the Jaccard distance to obtain an error replacement set; and sequentially replacing the words in the error replacement set with errors, adopting an rnnlm language model to carry out confusion degree calculation on the sentence after error replacement, determining correct words in the error replacement set according to the calculated sentence confusion degree, and completing Chinese text error correction. The invention provides the method for realizing cross-domain Chinese text error correction, namely a set of error detection-candidate recall-error correction sorting model, the error correction problem of cross-domain texts can be processed more universally, texts are recalled through a language model of deep learning training, the confusion degree of the recalled texts can be improved, the models are mutually decoupled, and the error correction efficiency is improved. And the efficiency is improved.

Description

technical field [0001] The invention relates to the field of text error correction, in particular to a method and system for realizing cross-field Chinese text error correction. Background technique [0002] In our daily life, when we use social tools such as WeChat and Weibo, typos often appear when we browse the web and read articles on official accounts, which leads to ambiguity in the meaning of the text. Chinese text error correction technology is an important technology for automatic checking and automatic error correction of Chinese sentences through natural language processing algorithms. Its purpose is to improve the correctness of language and enhance the efficiency and value of text interaction. The existing mainstream text error correction technologies are mainly divided into two types: one is to find the text error position through sequence learning, and then correct the pipeline method of text error information by sorting. The other is an end-to-end model base...

AI Technical Summary

Problems solved by technology

[0003] However, the former is relatively inefficient for sorting and recalling the algorithm for correcting wrong text, and the given correct text has a limited scope of application because the candidate set is a finite set, which may also lead to ambiguity.

The latter end-to-end method requires a large number of supervised training sets, and the model complexity is very high, and the performance cannot be embedded in many downstream applications as a basic module, which is too inefficient

Images