Method and system for realizing streaming crawler

Abstract

The invention discloses a method and a system for realizing a streaming crawler. The method comprises the following steps of: constructing an initial request data stream, and configuring a construction process of a crawler data stream conversion diagram; constructing a request data stream by using an object model through a mapping relationship between a hierarchical tree model and a website structure and a data object model, and configuring the request data stream to bypass a website anti-crawling strategy; a crawler component performing conversion operation on a data stream, and constructing a crawler data stream conversion diagram from requesting the data stream to downloading a page data stream and then to a result data stream. According to the method, the thought of responsive programming is adopted, the data stream conversion diagram of the whole crawler is constructed, the asynchronous streaming crawling model based on responsive programming is generated, blocking operation in the data crawling process is processed asynchronously, and compared with a traditional crawler scheme, the development efficiency, the system throughput and the resource utilization rate are improved; and the application value is very high.

Description

technical field [0001] The invention relates to a streaming crawler implementation method and system, and belongs to the technical field of network crawlers. Background technique [0002] With the explosive growth of network data, people's demand for personalized data search in various fields has also increased, and how to obtain useful information in the network has become an important task in various fields. For large-scale acquisition of network data, we often use crawler technology to crawl web pages, and perform respective analysis on the web page structure of specific pages, and finally obtain the required structured or semi-structured data. [0003] The general crawling method of crawlers is to start from an initial URL set, download the page corresponding to the URL; and analyze the content of the page to extract the corresponding page elements and links on the page. The extracted page elements are The required data; then filter the page links to obtain the links th...

AI Technical Summary

Problems solved by technology

But in the case of heavy load, this mode will consume a lot of memory, because each crawler request maintains a thread stack

Continuous context switching can also cause a lot of CPU time loss, can not make full use of CPU resources

[0005] In the real world, in addition to the problem of low CPU utilization caused by a large number of network IOs, crawlers also have various problems, such as login verification, exception handling, customized data analysis, and crawler configuration switching, etc.

These problems greatly reduce the efficiency of crawler development and the efficiency and quality of crawled data

However, the current crawler framework rarely considers the issues of crawler development and crawler operating efficiency.

Images