Text detection and correction method based on Pinyin similarity and language model

Abstract

The invention discloses a text detection and correction method based on Pinyin similarity and a language model. The method comprises the steps: collecting a large number of correct instruction text statements to serve as training statements; selecting words of a professional field from the training statements, and constructing a custom dictionary; carrying out word segmentation on the training statements by utilizing a HanLP language processing toolkit and a custom dictionary; counting the occurrence frequency of each word and each word combination in the word segmentation result in all the training statements, and constructing a Bi-Gram language model; converting the to-be-corrected statement into corresponding to-be-corrected pinyin, and converting words of the custom dictionary into corresponding dictionary pinyin; and correcting the to-be-corrected statement according to the pinyin similarity between the to-be-corrected pinyin and the dictionary pinyin in combination with the sentence rationality of the to-be-corrected statement to obtain a corrected statement. Through word pinyin similarity calculation and sentence rationality analysis, semantic information and contexts of sentences are considered, wrong words in the sentences can be detected, and the correction accuracy is improved.

Description

technical field [0001] The invention relates to the technical field of text detection, in particular to a text detection and correction method based on pinyin similarity and language model. Background technique [0002] When the speech recognition in the open field is directly applied to the professional field, due to the interference of noise, user accent and lack of professional vocabulary, the text after speech recognition has errors, which reduces the analyzability of the text. Chinese correction technology is an important technology to realize the automatic checking and correcting of Chinese sentences. Its purpose is to improve the correctness of the language and reduce the cost of manual verification. Most of the existing research on text correction is oriented to normative texts in open fields such as newspapers, books and periodicals. In specific domains, speech recognition engines in general domains have a low recognition rate for sentences in specific domains, and...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a text detection and correction method based on pinyin similarity and language model to solve the problem that the existing text correction method cannot fully consider the semantic information and context of the sentence, and relies too much on the example library of the sentence, resulting in text corrected for the low accuracy

Images