Distributed data crawling system, method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a distributed data crawling system, method and device, equipment and a storage medium. The system provided by the embodiment of the invention comprises a taskqueue cluster and a data crawling cluster, wherein the task queue cluster comprises at least one terminal; wherein a starting task queue and a middle task queue are arranged in the task queue cluster; wherein the initial task queue and the intermediate task queue are respectively used for storing an initial crawling address and an intermediate crawling address; wherein the data crawling cluster comprises at least one terminal and is used for accessing the task queue cluster to obtain an initial crawling address and an intermediate crawling address and crawling a target webpage according to the initial crawling address and the intermediate crawling address. According to the system provided by the embodiment of the invention, the initial task queue and the intermediate task queue are respectively arranged in the task queue cluster, so that the task amount in the data crawling process is convenient to change, the resource scheduling difficulty is reduced, and the data crawling efficiencyis improved.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of computer applications, and in particular to a distributed data crawling system, method, device, device and storage medium. Background technique [0002] The current society has entered the era of big data, human life is increasingly inseparable from data, and the amount of data is growing explosively. Since data contains a lot of value, how to obtain data has become an urgent problem to be solved. [0003] In the prior art, web crawlers are often used to crawl network data. Common web crawler implementation methods include: using Python's Requests library or Trep library to build a crawler system, using the Scrapy framework to build a single-process multi-threaded crawler system, or using the Scrapy framework and Redis database to build In the memory-based distributed crawler system, the above-mentioned technical solution has the problems of difficulty in resource scheduling and ...

AI Technical Summary

Problems solved by technology

[0003] In the prior art, web crawlers are often used to crawl network data. Common web crawler implementation methods include: using Python's Requests library or Trep library to build a crawler system, using the Scrapy framework to build a single-process multi-threaded crawler system, or using the Scrapy framework and Redis database to build The memory-based distributed crawler system, the above-mentioned technical solution, has the problems of difficult resource scheduling and fixed job tasks, which cannot be changed during the crawling process, and the data crawling efficiency is poor.

Images