Decentralized distributed data processing system

Abstract

The invention provides a decentralized distributed data processing system, and the system comprises a system construction and maintenance module, wherein a hash ring is partitioned by using a consistent hash algorithm, each partition is occupied by a key value, the key value range is 0-2160, and each node Node is mapped to a key value bit in the hash ring; a task scheduling module which is used for carrying out hash operation on the to-be-processed data to generate a corresponding hash identification value by utilizing a consistent hash algorithm when the to-be-processed data enters the system, and mapping the hash identification value to a key value bit in a hash ring; and a data processing module which is used for monitoring the to-be-processed data mapped to the interval section from the current node to the adjacent upstream node in real time, and receiving, processing and storing the to-be-processed data. According to the system, a previous master-slave distributed framework is abandoned, and a p2p decentralized distributed framework is adopted, so that the high reliability of the whole distributed operation is guaranteed.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a decentralized distributed data processing system. Background technique [0002] Because with the explosive growth of data, a single computer can no longer meet the ability to analyze and process big data. Blindly relying on improving the processing efficiency of a single computer hardware is not only costly, but also inefficient in processing big data. It will improve a lot, and thus encounter the bottleneck of data processing. [0003] Therefore, enterprises usually use hundreds of ordinary computers to network to form a distributed system to analyze and process big data. See eg figure 1 As shown in the development history of the distributed data system, in 2003, Google first proposed the distributed big data processing technology MapReduce, which is used to realize its search for global web page backup data. Yahoo developed and open sourced the Apache Hadoop system based...

AI Technical Summary

Problems solved by technology

The master-slave distributed architecture system is highly dependent on the central master node. If the central node loses connection or fails, the entire distributed system will fail and cannot process data; and when maintaining and upgrading, since there is only one central node, the system needs to be Stopping the system makes the reliability of the system very low; in the data processing of the master-slave architecture, each slave machine needs to perform data interaction with the host machine, the processing capacity of a single host machine is limited, and the bandwidth is also limited, resulting in data processing bottlenecks and affecting data processing s efficiency

Such an architecture brings performance, scalability, and reliability disadvantages

Images