Cascade crawling method and device for multi-level pages based on web crawlers

Abstract

The invention relates to a cascade crawling method for multi-level pages based on web crawlers. The method comprises: grabbing an upper-level page, storing grabbed data in an upper-level page data analysis table, and setting main key values for objects needing to continue to grab a lower-level page in the upper-level page data analysis table, wherein the main key values corresponding to the objects are different; grabbing a subordinate page and storing the captured data in a subordinate page data analysis table; setting a foreign key value for the lower-level page data analysis table, obtaining a main key value of an object corresponding to a lower-level page from an upper-level page data analysis table, and taking the main key value as the foreign key value of the lower-level page data analysis table, thereby realizing associated query of an upper-level webpage and a lower-level webpage after grabbed data falls to the ground. According to the method, a data acquisition mode capable ofrestoring logics before and after the webpage is provided, the webpage capture integrity is ensured, the data is stored according to the original webpage hierarchy sequence, and the associated multi-hierarchy page data can be conveniently obtained.

Description

technical field [0001] The invention relates to a web crawler-based cascade crawling method and equipment for multi-level pages, belonging to the field of data crawling. Background technique [0002] The existing method of crawling upper and lower level pages is: first grab the upper level pages, then store the URL addresses in the upper level pages, and repeatedly grab the lower level pages according to these URL addresses, and finally identify and match the landing data through the crawler task. The crawler task ID is in one-to-one correspondence with the crawler and the data landing file captured by the crawler; when the crawler task ends and needs to match data, use the crawler task ID to parse the crawled data file into a structured file according to the logic of the original web page data. Because the crawler task ID can only play a role of one-to-one correspondence with the crawler task, the task ID does not reflect the hierarchical relationship, therefore, the origi...

AI Technical Summary

Problems solved by technology

Because the crawler task ID can only play the role of one-to-one correspondence with the crawler task, the task ID does not reflect the hierarchical relationship, therefore, the original data hierarchy cannot be restored through the crawler task ID

If there is an association between multi-level pages, when the existing crawler technology captures multi-level hierarchical data, it is difficult to verify the integrity and accuracy of the data due to the complexity of the association logic between the levels.

At the same time, due to the greater difficulty of data access, multi-level web data usage rules are more cumbersome

Images