Extraction method and device for Internet-oriented meaningful strings

Abstract

The invention provides an extraction method and a device for Internet-oriented meaningful strings. The extraction method comprises the following steps: extracting repeated character strings and filtering the character strings sequentially by in-string analysis, out-string analysis and among-string analysis; and the extraction device comprises a repeated string discovery module, an in-string analysis module, an out-string analysis module and an among-string analysis module which are successively connected in series. The invention can effectively extract meaningful strings on news pages and forums, and can be widely used in the fields of network public opinion management, Internet intelligent information processing and the like.

Description

technical field [0001] The present invention relates to a technology that utilizes computer technology to assist intelligent analysis of network information or management of public opinion, specifically a method and system for quickly, accurately and efficiently extracting meaningful strings from massive Internet web pages and forum information. Background technique [0002] Text representation is the first step in content-based text processing. Feature items in text representation are important factors affecting text classification and clustering results. Currently commonly used text feature items mainly include words, words, phrases, and semantics. Theoretically speaking, semantic concept (semantic set) is higher than phrase (syntax set), phrase is higher than word (word set), and word is higher than word (character set). Usually, semantic concepts can be obtained by means of semantic dictionaries (synonyms, synonyms dictionaries, etc.) or latent semantic indexes. Howev...

AI Technical Summary

Problems solved by technology

The disadvantage of using words as features is that it simply considers whether a word appears in a document and its frequency of occurrence, and regards features as independent existence, while completely ignoring the semantic relationship between text contexts, and does not consider features order of precedence between

[0005] In a nutshell, the existing meaningful string extraction algorithms have the following disadvantages: 1) Using mutual information as a feature in the intra-string analysis cannot filter double-word strings well. The string is actually a single-character string, and it is meaningless to calculate the frequency of occurrence of a single character; 2) Both the internal analysis and the external analysis do not consider the difference between the string and the string, and there will be many string representations in the extracted meaningful string Similarity, resulting in semantic similarity and redundancy of many meaningful strings

Images