Three-folded webpage text content recognition and filtering method based on the Chinese punctuation

Abstract

A method based on Chinese website punctuation triple recognition and text content filtering. The method based on existing URL, the website information keywords in the method of filtration - filtration rate and the low rate of filtration of the whole problem, Bringing on a method for composite based on the URL and on keywords, as well as text-based knowledge representation method of vector space website text content filtering. Applying to a method Based on black-and-white list of URL filtering and Chinese punctuation statistical characteristics to effectively remove navigation information, relevant linked information, advertising linked information, copyright information and other Web content noise information to extract content of text; adopting vector space model text knowledge representation, By calculating vector text template and unhealthy information in the feature vector cosine angle, and set the threshold, compared to the text of the class. The invention can be widely used in the filtering of undesirable information network and website personalized information services.

Description

technical field [0001] The invention belongs to the field of network information security and relates to the identification and filtering of bad text information on Chinese web pages. Background technique [0002] In several existing web content security products, such as "Internet Nanny" and "Internet Dad", most of them use methods based on URL addresses and keywords to prohibit access to illegal web pages and websites. In terms of diversity and dynamics, this method of using a static address library or manually updating URLs and keywords is far from meeting people's filtering requirements. Parents look forward to the emergence of more effective and comprehensive information filtering products. [0003] Existing filtering methods for web page text content mainly revolve around the vector space model. [0004] Liu Peide and others constructed a network information filtering system (NIFS) with a feedback mechanism by using the vector space model, TC3 classification algorithm...

AI Technical Summary

Problems solved by technology

[0010] 1. Using the URL and keyword filtering method, the filtering accuracy and filtering rate are low, and the filter is easily bypassed;

[0011] 2. Using the content filtering method based on text vector space alone has a slow filtering speed and cannot meet the requirements of real-time filtering for broadband network data transmission;

[0012] 3. There are few studies on the preprocessing process of web pages, especially the research on the general web page content extraction method has not been reported in the literature, and the research on this issue can effectively improve the speed of web page data processing;

[0013] 4. The content identification and filtering methods for the characteristics of Chinese web pages have not been reported yet

Images