Big data processing method and big data processing device

Abstract

The invention provides a big data processing method and a big data processing device. The big data processing method comprises the following steps: obtaining a first performance ratio among all servers in a cluster; according to the first performance ratio, selecting a plurality of target servers used for processing a current big data task in all servers according to the first performance ratio; according to a second performance ratio among a plurality of selected target servers, segmenting the current big data task to obtain a plurality of data task blocks; correspondingly distributing the plurality of data task blocks to the plurality of target servers to be independently processed; and receiving a plurality of processing results obtained by that the plurality of target servers correspondingly process the plurality of data task blocks to carry out summary output. Through the technical scheme of the big data processing method, in a multi-thread cluster, the computation efficiency of the big data can be effectively improved, and the problem of memory overflow is solved so as to improve user experience.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular, to a big data processing method and a big data processing device. Background technique [0002] At present, the traditional big data processing method is single-server and single-threaded during the running of the program. At this time, if the amount of data is large, it will not only cause a long running time, but also may cause memory overflow. For example, there are 10 million data to be calculated, and now there are 7 servers in a cluster, namely A, B, C, D, E, F and G, such as figure 1 As shown, when processing, call server E through load balancing, and directly execute 10 million data on a single machine on server E, which not only slows down the calculation efficiency, but also may cause memory overflow. [0003] Moreover, even if a single server is running with multiple threads, when the amount of data is large, running multiple threads may cause threads to wait ...

AI Technical Summary

Problems solved by technology

[0002] At present, the traditional big data processing method is single-server and single-threaded during the running of the program. At this time, if the amount of data is large, it will not only cause a long running time, but also may cause memory overflow.

For example, there are 10 million data to be calculated, and now there are 7 servers in a cluster, namely A, B, C, D, E, F and G, such as figure 1 As shown, when processing, call server E through load balancing, and directly execute 10 million data on a single machine on server E, which not only slows down the calculation efficiency, but also may cause memory overflow

[0003] Moreover, even if a single server is running with multiple threads, when the amount of data is large, running multiple threads may cause threads to wait and the calculation speed of the program to be slow, which greatly reduces the user experience.

Images