Method and device for judging type of webpage

Abstract

The invention discloses a method and a device for judging the type of a webpage. The method comprises the following steps: carrying out rule matching in a prestored rule table on the basis of the URL of a webpage to be judged, wherein the rule list comprises a plurality of rule records for judging the type of the webpage; if the rule matching is successful, obtaining the type of the webpage to be judged according to the successfully matched rules; if the rule matching is unsuccessful, extracting predetermined features from the URL and / or HTML source codes of the webpage to be judged, and utilizing a classifier to classify the type of the webpage to be judged so as to obtain the type of the webpage to be judged on the basis of a feature vector composed of features selected from the extracted predetermined features. In the scheme of the invention, the invention can combines the advantages of a rule recognition scheme and a recognition scheme based on statistical learning, and can realize to judge the types of various webpage such as blogs, forums, news and the like.

Description

technical field [0001] The present invention generally relates to a natural language processing technology including text classification, and in particular relates to a method and / or device for judging a webpage type. Background technique [0002] With the rapid development of computer and network technology, various factors such as the demand for personal space and the simplification of website creation have promoted the rapid increase in the number of websites. Taking China as an example, according to the "22nd Statistical Report on Internet Development in China" released by China Internet Network Information Center (CNNIC), by the end of June 2008, the total number of domain names in China reached 14.85 million, with an annual growth rate of 61.8%. . In recent years, the number of users of various network media such as network news, blog (Blog) / personal space, forum (BBS) has increased greatly. Among all network applications including basic applications, online media, d...

AI Technical Summary

Problems solved by technology

[0007] Although the above methods have achieved good results in blog identification, they separate the statistical identification method based on machine learning from the rule identification method based on the specific characteristics of the blog platform or webpage provided by the blog provider. Combining the two methods

Compared with the statistical identification method, the rule identification method generally has the advantages of fast speed and high precision. However, due to the continuous increase in the number of websites, the rule identification method is difficult to cover all situations, and the statistical identification method can combine various features to make a comprehensive analysis of the type of web pages. Judgment, in principle, is applicable to any webpage, but the speed is generally not as fast as the rule recognition method, and there is also the possibility of misjudgment

Moreover, the features used in the existing recognition process are not rich enough, and the unique features of forums, news and other types of pages are not considered in the feature extraction

Images