News keyword abstraction method based on word frequency and multi-component grammar

Abstract

A method to extract new keywords based on word frequency and multiple grammars is provided, which belongs to the technology field of a natural language processing, and is characterized by extracting the potential models of part of speech of the multiple grammars of the keywords by researching characteristic part of speech of the keywords and adopting computer to assist excavation and taking the models as the basis of the keywords to extract arithmetic. When extracting the new keywords, firstly excavating the multiple phrases in text in accordance with the potential models of part of speech and extract candidate word set of the keywords, and then excavating potential keywords not loading from titles and add the potential keywords to the candidate keyword set. The application brings forward an improved single text word frequency / inverse text frequency value (tf / idf) format, introduces target-oriented characteristics, grades the candidate keywords, obtains the order of the candidate keywords and gives the keywords of news document after optimizing the results. Compared with the traditional keyword extraction method based on single text word frequency / inverse text frequency value (tf / idf), the method has higher recall rate under the condition of the same precision.

Description

technical field [0001] The invention belongs to the field of text mining, in particular to news text mining. Background technique [0002] Keyword extraction is an important research topic in text information retrieval. The keyword extraction of Chinese news plays an extremely important role in understanding the important content of news and realizing the precise retrieval of related news events. Text keywords refer to several words or phrases that can summarize the text and are related to the semantic content of the text. Through keywords, people can quickly find the information they need. Furthermore, keywords can also provide rich semantic information for deeper text mining applications, such as text classification, text clustering, text retrieval, and topic mining. [0003] At present, there are many keyword extraction methods at home and abroad, and they have been widely used. However, most of the research work is focused on improving the accuracy of keyword extract...

AI Technical Summary

Problems solved by technology

However, most of the research work is focused on improving the accuracy of keyword extraction, without carefully studying what words are keywords, resulting in inconsistent standards for keyword definitions, and it is difficult to compare various methods

In addition, the results of keyword extraction are greatly affected by the quality of the word segmentation system. In keyword extraction, most keyword extraction methods use word segmentation as the first step in processing, and the missed detection and errors in the word segmentation process directly cause text Difficulties in follow-up processing, and for this reason, the keyword extraction method also needs to solve the problem of extracting unregistered words

Images