System and method for identifying webpage types

Abstract

The invention relates to the field of network information retrieval and mining, in particular to a system and method for identifying webpage types. The method comprises the following steps that a heuristic rule is predefined, and a heuristic rule list is generated; a predetermined feature is extracted from a training webpage to form a standard feature vector which is optimized twice to form a simplified feature set, a classifier and a feature extractor are established, and a classification model is generated through the classifier; based on an URL and a source code of a webpage to be identified, rule matching is carried out on the heuristic rule list; if matching succeeds, the webpage type of the webpage to be identified is output; if matching fails, the classifier is used for carrying out webpage type classification on the webpage to be identified. The system and method for identifying webpage types are flexible and convenient to use, high in identifying speed and high in identifying accuracy, big change is of no need when a cross language webpage is identified, identifying efficiency is high, and high actual use value is achieved.

Description

technical field [0001] The invention relates to the field of network information retrieval and mining, in particular to a system and method for identifying web page types. Background technique [0002] With the increase of network information, it is sometimes difficult to retrieve the information documents that users want through search engines. At the same time, how to express the search results of search engines to users has also attracted more and more attention. Most traditional search systems return a large collection of web documents that can match user queries. However, the high recall rate and low precision of search engine processing result documents make it more and more difficult to find useful information for users. In recent years, researchers have done a lot of research on the method of classifying documents according to topics, and achieved good results. However, although documents can be successfully classified according to topics, there are still a large n...

AI Technical Summary

Problems solved by technology

[0008] The technical problem to be solved by the present invention is to provide a system and method for identifying web page types, which solves the problem of poor web page type identification based on heuristic rules and inappropriate feature selection of classifiers in the prior art, especially for cross-language The problem of requiring large changes and low efficiency when identifying web pages

Images