A Spark energy-saving scheduling method based on energy consumption awareness

Abstract

The invention discloses a Spark energy-saving scheduling method based on energy consumption perception. According to the method, first, a big data computing energy consumption model under a Spark computing framework is constructed, a policy table of a relation between energy consumption of tasks and computing resources and execution time is established based on the model, Spark task scheduling isguided and optimized through the policy table, and therefore total computing energy consumption is effectively lowered on the premise of guaranteeing parallel computing efficiency. Through the method,the defect that an original Spark scheduling policy cannot perceive energy consumption is overcome. The method has the advantages of energy consumption perception, dynamic optimal scheduling and highextensibility, and energy consumption generated by application programs running under the Spark computing framework is effectively lowered.

Description

technical field [0001] The invention relates to the fields of big data processing and energy efficiency, in particular to a big data energy consumption model based on energy consumption perception and an energy-saving scheduling strategy under the Spark computing framework based on the model. Background technique [0002] The huge power consumption generated by big data computing has become an urgent problem to be solved in data centers. At present, many enterprises and organizations are facing the problem of large-scale data computing. While considering computing efficiency, computing cost is also an important aspect of their concern. Both enterprises and organizations hope to reduce the energy consumption of big data computing to reduce computing costs. However, the situation is not optimistic in the current era of big data, according to a recent research report [1] It shows that from 2010 to 2015, electricity consumption in German data centers increased by 15%, to 1.2 b...

AI Technical Summary

Problems solved by technology

When the physical nodes of the cluster change, the original performance analysis results need to be regenerated, which has poor flexibility and cannot be applied to solve the energy consumption problem of the Spark computing framework.

Images