Lexical classification method and system and realization method

Abstract

The invention discloses a lexical classification method and system and a realization method. The lexical classification method comprises the steps as follows: firstly, obtaining a document classification training set comprising documents and document type information belongs to the documents; secondly, preprocessing all the documents to form lexes to be classified; thirdly, obtaining the TF (lexical frequency) value and the IDF (inverse document frequency) value of each lexis to be classified in one document type, and summating the TF value and the IDF value to obtain a TF-IDF value; and lastly, taking a quotient obtained by division of the TF-IDF value by the sum of the TF-IDF values of the TF values and the IDF values of the lexes to be classified in all the document types belonging to the documents as a probability that the lexes to be classified belong to the document types, and generating a lexical classification database comprising the lexes to be classified, the types corresponding to the lexes to be classified and the probability that the lexes to be classified belong to the corresponding types. Due to the adoption of the technical scheme, the lexical classification operation can be automatically finished, the input cost is very low, and the effect is more accurate.

Description

technical field [0001] The present invention relates to the technical field of document analysis, in particular to a vocabulary classification method, system and implementation method. Background technique [0002] In the field of document analysis technology, thesaurus is a relatively important technology, which can be used in various purposes and fields. For example, in user behavior analysis, the user's basic category can be determined by using the user's input of keywords. Collect the search keywords entered by the user, and the category of the keyword can be obtained through the thesaurus, and then the user category is marked. [0003] Table 1 [0004] vocabulary category confidence probability Yao Ming physical education 90％ entertainment 10％ fund finance 72％ public welfare 28％ [0005] As shown in Table 1, vocabulary classification can be used in dictionary editing, sema...

AI Technical Summary

Problems solved by technology

For example, in the classification of Sina, the first-level classification is sports, and the second-level classification changes with current events. The two classifications of football and the World Cup will exist at the same time, and some web pages will be repeated. This situation has not been solved by the existing technology. method

[0013] 3. The degree of contribution of the existing web pages to the classification is not clearly indicated, but the importance of the web pages to the classification is available and has great application value

Images