Energy-saving-oriented Hadoop distributed file system storage policy in cloud computing

Abstract

The invention discloses an energy-saving-oriented Hadoop distributed file system storage policy in cloud computing. The energy-saving-oriented Hadoop distributed file system storage policy comprises the following steps: 1, performing regional division on a data node, dividing active-state data nodes in a hot area and standby-state data nodes in a cold area, and storing a created file in the hot area; 2, storing a data file, which is stored in the hot area, in the maximum data node of the preferably matched hot area according to a preferably matching policy; 3, judging the active degree of the data file, and transferring a data file in the cold area after the active degree reaches a threshold range; and 4, performing active degree judgment on the data file transferred in the cold area, and setting a node to be in a standby state if the difference between the existing time and the last access time of the data node of the cold area storing the data file is larger than a standby time threshold of the node. According to the energy-saving-oriented Hadoop distributed file system storage policy in the cloud computing, the hot nodes and the cold nodes can be effectively utilized, and the energy consumption is relatively greatly reduced.

Description

technical field [0001] The invention relates to the field of computer big data, in particular to an energy-saving Hadoop distributed file system storage strategy in cloud computing. Background technique [0002] With the continuous improvement and popularization of cloud computing technology, following the pursuit of performance, capacity, fault tolerance, security and other indicators, the concept of green energy saving has gradually become a new standard in the industry. Among the existing strategies for energy-saving management of the Hadoop distributed file system, some of them mainly reduce the energy consumption of the server during operation by classifying and learning the computing load or migrating and storing data in real time. The cost of cooling the entire data center infrastructure. Although the existing method has obvious energy saving, like the traditional Hadoop distributed file system, the system uses a rack-aware data block storage strategy to make the dis...

AI Technical Summary

Problems solved by technology

Although the existing method has obvious energy saving, like the traditional Hadoop distributed file system, the system uses a rack-aware data block storage strategy to make the distribution of data blocks in the cluster random. On the one hand, this strategy will cause the entire cluster to Unbalanced data distribution, especially when new nodes are added, will result in a waste of computing and storage capacity for new nodes; on the other hand, there are huge differences in access rules between different files. If Hadoop distributed All data nodes in the file system cluster are active, which will inevitably increase energy consumption and cause a lot of power to be wasted

Images