A Load Balancing Method for Big Data Real-time Query System Based on Replica Selection

Abstract

The invention relates to the field of computer database processing, in particular to a big data real-time enquiry system load balancing method based on copy selection. The method comprises the processes of node load information collection and node load balancing, and the node load balancing process comprises the stages of preprocessing and the copy selection. The method has the advantages that the problems that an existing big data real-time enquiry system load balancing method is too simple and the current state of a machine is not considered are solved, the new big data real-time enquiry system load balancing method based on the copy selection is provided, the load balancing effect is superior to that of the existing big data real-time enquiry system, time complexity is low and is O (n2), and n is the number of blocks; the method is suitable for heterogeneous distributed systems and the conditions of operating other tasks in the systems.

Description

technical field [0001] The invention relates to the field of computer database processing, in particular to a load balancing method for a large data real-time query system based on copy selection. Background technique [0002] In the era of big data, it is impossible to store massive data in a single server. Existing big data real-time query systems, such as Google Dremel, Cloudera Impala, etc., all adopt distributed computing architecture to ensure real-time big data query. How to ensure the load balance of each node during operation has always been the focus of distributed systems. [0003] The database tables of existing big data real-time query systems logically consist of stored data and associated metadata describing the form of the data in the tables. Data is generally stored in a distributed file system. The existing distributed file system divides files into blocks, stores different data blocks of the same file on multiple nodes, and creates a copy of each data b...

AI Technical Summary

Problems solved by technology

The existing load balancing methods of big data real-time query systems have the following problems. First, it is impossible to obtain better load balancing

When the strategy for selecting replicas is determined each time, the degree of load balancing generated by different sequences of data blocks is varied. It is difficult to obtain better load balancing by only considering the default sequence of data blocks.

Second, the heterogeneity of the distributed system is not considered, such as the difference in the disk read rate of the machine

Images