Slow-task dispatching method and slow-task dispatching device of Map Reduce system

Abstract

The invention provides a slow-task dispatching method and a slow-task dispatching device of a Map Reduce system. The slow-task dispatching method includes: respectively acquiring computing power values of computing nodes in the Map Reduce system, arraying the computing nodes into a slow-node array according to the computing power values, selecting last M computing nodes in the slow-node array as target computing nodes, acquiring pre-estimated computing power values after the target computing nodes downloads slow tasks to be dispatched, arraying the target computing nodes and other computing nodes to be a new slow-node array according to the sequence from large to small of the computing power values, presetting the Nth computing node counted from the end to the start of the slow-node array to be an evaluation reference node, and dispatching the slow tasks to be dispatched to the target computing nodes when the pre-estimated computing power values of the target computing nodes are larger than the computing power value of the evaluation referent node, wherein the N is a natural number. By the aid of the slow-task dispatching method, fluctuation during slow-task dispatch is suppressed effectively.

Description

technical field [0001] The invention relates to computing technology, in particular to a slow task scheduling method and device for a MapReduce system. Background technique [0002] As a basic computing framework, MapReduce is widely used in Internet applications such as cloud computing environments. With the development of the cloud computing environment, virtualization technology is gradually adopted, and there may be multiple virtual machines on a physical host; due to the large difference in the computing capabilities of different physical hosts, and the differences between virtual machines on the same physical host There is a large performance difference, which leads to the prominent node heterogeneity of the MapReduce system. The node heterogeneity refers to the difference in the processing speed of different computing nodes (one virtual machine is equivalent to one computing node). Computing tasks are scheduled to different computing nodes, and the response time for ...

AI Technical Summary

Problems solved by technology

The disadvantage of the above scheme is that too many computing tasks are identified as slow tasks, and the scheduling of too many slow tasks takes up more system resources; moreover, the adjacent scheduling principle may also appear that adjacent nodes run slower after scheduling. The slow task will be further scheduled, causing the slow task to be scheduled multiple times, that is, the jitter of system scheduling appears

However, it has been found in practice that LATE still cannot solve the jitter problem of system scheduling. Even if a computing node located outside the last 25% is selected for slow task scheduling, it is still possible that the computing power of the node will drop too much after loading the slow task. If there are too many slow tasks, the scheduled slow tasks are still slow tasks and will be rescheduled, resulting in system jitter in which slow tasks are scheduled multiple times

Images