A node and system in a distributed crawler cluster

Abstract

An embodiment of the invention discloses decentralized distributed-type nodes and a distributed-type system in a crawler cluster, wherein the decentralized distributed-type nodes and the distributed-type system in the crawler cluster are based on structuralized peer-to-peer (p2p). The distributed-type nodes in the crawler cluster based on the structuralized p2p comprise a bottom covering net, a crawl module and a control center, wherein the bottom covering net is based on a p2p organization mode to execute a protocol of disseminating and receiving url between the nodes, the crawl module is based on the disseminated url to be in charge of grabbing corresponding resources from the internet, and the control center executes the function of disseminating and receiving the url. By using the features of kademlia of the structuralized p2p algorithm, the distributed-type nodes and the distributed-type system in the crawler cluster skillfully solve de-weight and load balancing problems in the distributed-type crawler system, achieve good expansibility and fault tolerance of the system, and are capable of providing a universal design method for a large-scale distributed-type crawler system.

Description

technical field [0001] The invention relates to the field of computer data mining, in particular to a distributed crawler cluster method. Background technique [0002] Crawlers are the most important tool for data collection of search engines. In today's era of information explosion, it is difficult for crawlers in the traditional client (C) / server (S) mode to capture all the data in the network; moreover, if If the number of crawlers increases, the server will bear a greater load and the system will not be able to improve service performance. In this context, the method of distributed crawler clusters has gradually entered people's field of vision. [0003] The current research on distributed crawler clusters is mainly focused on the distributed crawler system under the master-slave mode, that is, there will be some core management nodes responsible for task management, uniform resource locator url deduplication and load balancing, etc., such master-slave The mode still c...

AI Technical Summary

Problems solved by technology

[0003] The current research on distributed crawler clusters is mainly focused on the distributed crawler system under the master-slave mode, that is, there will be some core management nodes responsible for task management, uniform resource locator url deduplication and load balancing. Such master-slave The mode still cannot solve the problems of performance bottlenecks and large-scale expansion

Another distributed crawler model is a structured peer-to-peer network p2p, that is, each host running a crawler node can communicate freely. Without a control node, there is no system bottleneck and large-scale expansion is possible. However, due to lack of management, Collaboration between nodes, url deduplication and network load balancing have become difficult problems to solve

Images