Microblog text normalizing, word segmenting and part-speech tagging method and system

Abstract

The invention relates to a microblog text normalizing, word segmenting and part-speech tagging method. The microblog text normalizing, word segmenting and part-speech tagging method comprises the steps that firstly, a tagged corpus is established, and tagged corpora in the tagged corpus is divided into a training set, a development set and a testing set; secondly, a microblog dictionary is established through SVM model training and learning; thirdly, through the training set, the development set and the microblog dictionary, a text normalizing, word segmenting and part-speech tagging combined model is formed through training and learning with a BeamSearch method; fourthly, through the combined model, text normalizing, word segmenting and part-speech tagging are conducted on a microblog text to be processed at the same time, and the performance of the combined model is tested. According to the method, a large number of microblogs with tagged sentences are used as the training corpus, a candidate result is expanded through the mciroblog dictionary, the established combined model can act on three tasks at the same time, the three tasks influence each other, so that the performance of each task is improved, and therefore the overall performance is improved.

Description

technical field [0001] The invention relates to the field of natural language processing, in particular to a BeamSearch-based microblog text normalization method. Background technique [0002] Commonly used natural language processing techniques such as word segmentation, part-of-speech tagging, and syntactic analysis are all based on normalized text. However, these technologies are not very good at processing non-normalized text such as Weibo. Therefore, models trained using traditional corpus cannot be directly applied to microblog texts, and new research is needed on the processing of microblog texts. [0003] Due to many non-standard language phenomena in Weibo texts, especially the large use of non-standard words. E.g: [0004] 1) I just saw Chen Laoshi's scarf and found out that friend c has come to Ningbo. [0005] 2) Children who do not hand in their homework will have safflowers. [0006] If the traditional model is used for word segmentation and part-of-speech...

AI Technical Summary

Problems solved by technology

[0005] 2) Children who do not hand in their homework will have safflowers

[0010] The traditional method of microblog processing is generally serial processing. Text normalization is done first, and then other processing such as word segmentation and part-of-speech tagging is performed. This processing is firstly inefficient and secondly, errors are propagated. If normalization is wrong, It will inevitably lead to an increase in the error rate of subsequent tasks.

Images