Method and system for extracting webpage information

Abstract

A method of extracting web page information includes analyzing a document object model (DOM) structure of a sample page to obtain a position of information to be extracted. A node corresponding to the position of the information to be extracted is rendered in the DOM structure as a target node. Starting from the target node, relative position information is traversed recursively until the root node is found to create candidate paths. The candidate paths are rendered as a path set. A DOM structure of a page to be extracted is analyzed, information is located in the DOM structure of the page starting from the root node in the path set, and an extracted node candidate set is obtained. A node having highest robustness from the extracted node candidate set is selected to be a final extracted node and extracted information is obtained using the extracted node.

Description

Technical field [0001] This application relates to the field of network technology, in particular to a method and system for extracting webpage information. Background technique [0002] With the rapid development of the Internet, the Internet has become the most important information release platform. However, in the face of the explosive growth of Internet information, how to quickly and effectively obtain the information users need has become an urgent problem to be solved. Traditional search engines can help people get web pages by searching keywords, but they only give links to related pages, and users still need to manually browse the web to find interesting information. On the other hand, due to the inability to customize precise queries, a large number of search results are not what users want, and precise and professional search results cannot be provided. An ideal approach is that the Internet as an information source can be queried like a database. Thus, web informa...

AI Technical Summary

Problems solved by technology

The automatically generated XPATH usually only records the label name and offset information, and the positioning information is too simple to cope with the continuous change of the web page structure

However, after the content of the webpage is updated and the elements on the XPATH path change, the problem of not being able to locate the content or locating the non-extracted content will be caused.

At the same time, because the information recorded by XPATH is too simple, XPATH cannot be used to solve the problem of repeated structure identification, and additional algorithms need to be added to realize the identification and extraction of repeated structures

[0005] In the process of implementing the present application, the inventor found at least the following problems in the prior art: web page information extraction usually uses a semi-automatic information extraction method, and the extracted information is located by analyzing the page structure. Since web page information is a type of dynamic change, For real-time updated data, after the page content is updated and the structure of the page changes, it is prone to problems such as extraction failure or inaccurate extraction results caused by location information failure

[0006] On the other hand, existing techniques cannot well address the problem of repetitive structure identification

The automatic XPATH generation method cannot use XPATH to solve the problem of repeated structure identification, and additional algorithms need to be added to realize the identification and extraction of repeated structures

Images