Corpus tagging method and a corpus tagging device

Abstract

The invention discloses a corpus tagging method and a corpus tagging device. The corpus to be processed is vectorized to obtain a text vector of the corpus. According to the text vector of the corpus,clustering the corpus by using a DBSCAN clustering algorithm to obtain a long tail corpus and a corpus to be annotated; Returning a re-clustering process for the long tail class corpus; As for that corpus to be annotate, a label is set to obtain the annotated corpus; Finally, all the tagged corpus are combined to get the final tagged corpus, without adjusting the number of clusters many times, the algorithm is simpler, the tagging efficiency is higher, and the reliability is better.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a corpus tagging method and a device for applying the method. Background technique [0002] Corpus is the basic resource of corpus linguistics research and the main resource of empirical language research methods. Traditional corpora are mainly used in dictionary compilation, language teaching, traditional language research, statistical or case-based research in natural language processing, etc. With the development of Internet big data and artificial intelligence technology, corpus has also been widely used. [0003] The corpus stores the language materials that have actually appeared in the actual use of the language, such as user messages and customer service dialogues obtained directly from the webpage; the corpus is the basic resource for carrying language knowledge, but it is not equal to language knowledge; real corpus It needs to be processed before ...

AI Technical Summary

Problems solved by technology

[0006] The second is to use an unsupervised algorithm (clustering algorithm) to cluster the labeled data, and then label each category; this method can directly label the corpus without relying on too much prior information, but some manual intervention is required later

[0007] For the second labeling method, the most common one is to pre-label the corpus with the k-means algorithm as the core algorithm, but the disadvantage is that because k-means needs to achieve a given number of clusters, the possible result is that it depends on experience first Specify a number of clusters, and then adjust the value of the number of clusters continuously according to the clustering effect until the appropriate value is adjusted. The whole process is too redundant

Images