HDFS (Hadoop Distributed File System)-based small file combination tool and method

Abstract

The invention provides an HDFS (Hadoop Distributed File System)-based small file combination tool and method, which is applied to a server which can carry out information interaction with each node contained in the HDFS in a cluster. The HDFS-based small file combination tool comprises a setting unit, a catalogue generation unit, a transmission unit and a reading combination unit, wherein the setting unit is used for setting a small file combination rule; the catalogue generation unit is used for determining catalogue parameters and generating an input / output catalogue according to the catalogue parameters; the transmission unit is used for setting a small file threshold value, determining the small file in each external node according to the small file threshold value and storing the determined small file into the input catalogue generated by the catalogue generation unit; and the reading combination unit is used for traversing the input catalogue, reading at least two small files which conform to the combination rule set by the setting unit in the input catalogue, combining the at least two small files into at least one data file and storing the at least one data file into the output catalogue generated by the catalogue generation unit. File processing efficiency can be effectively improved.

Description

technical field [0001] The invention relates to the technical field of computer applications, in particular to an HDFS-based small file merging tool and method. Background technique [0002] Distributed file system (Hadoop, HDFS) consists of a NameNode and several DataNodes. It is an important part of the cluster and has been widely used in the field of large-scale computing due to its reliability, efficiency and scalability. Because the NameNode puts the metadata of the file system in the memory block, although the size of each small file is much smaller than the size of a block, each small file needs to occupy a block, and the storage of each block occupies about 150 bytes , then, if there are 10000000 small files, the namenode needs about 2G memory space. If 100 million files are stored, the namenode needs 20G memory space. At the same time, the access process of HDFS to small files needs to continuously jump from one small file to another. Then, with the number of small...

AI Technical Summary

Problems solved by technology

Because the NameNode puts the metadata of the file system in the memory block, although the size of each small file is much smaller than the size of a block, each small file needs to occupy a block, and the storage of each block occupies about 150 bytes , then, if there are 10000000 small files, the namenode needs about 2G memory space

If 100 million files are stored, the namenode needs 20G memory space. At the same time, HDFS needs to continuously jump from one small file to another in the access process of small files. Then, with the number of small files stored in HDFS increase, leading to a decrease in the efficiency of file processing

Images