Corpus noise reduction method and device, electronic equipment and storage medium

Abstract

The invention relates to a corpus noise reduction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring estimated label distribution of an initial corpus set; obtaining a confidence matrix according to the estimated label distribution, wherein the confidence matrix is used for describing label noise distribution under a category condition; obtaining a noise corpus in the initial corpus set based on the confidence matrix; and processing the noise corpus in the initial corpus set to obtain a target corpus set. In the embodiment, the confidence matrix can be established through the prediction probability of the label and the labeling label, the noise corpus in the initial corpus set is identified through the confidence matrix, after the noise corpus is processed, the proportion of the noise corpus in the target corpus and ambiguous information can be reduced, the boundary of the target corpus is clearer, and the training frequency of the vertical domain model is reduced, so that computing resources and consumed time required by training are reduced, and the training efficiency is improved.

Description

technical field [0001] The present disclosure relates to the technical field of corpus noise reduction, and in particular, to a corpus noise reduction method and device, electronic equipment, and a storage medium. Background technique [0002] With the improvement of semantic understanding ability of intelligent voice assistant, it has become an important application of human-computer intelligent interaction. Existing intelligent voice assistants usually use multi-vertical-domain competition to realize intelligent services, that is, intelligent voice assistants initiate requests to multiple preset vertical-domain service models, and each vertical-domain service model analyzes the above requests and provides the service Feedback to the intelligent voice assistant, and inform the confidence level of the service it provides; the intelligent voice assistant will feedback the service with the highest confidence level to the user. Therefore, the quality of service provided by eac...

AI Technical Summary

Problems solved by technology

[0003] In practical applications, the corpus of the vertical domain service model comes from a variety of sources, and the expressions of different application scenarios are quite different, which is easy to introduce noise data; when a certain vertical domain has rich semantics, there will be blurred semantic boundaries and ambiguous information, resulting in Decrease in labeling accuracy; there are differences in the understanding of corpus by labelers, which leads to the introduction of noise in the labeling process

Images