File merging method and device for big data platform

Abstract

The invention provides a file merging method and device of a big data platform. The method comprises the following steps of monitoring the directory change of the big data platform, and judging whether the number of files under the changed directory changes or not; grouping files having similar characteristics under the changed directory in a case where the number of files under the changed directory changes; judging whether there are small files smaller than an integer multiple of a set number of data blocks in the same group of files; when the small files exist in the files of the same group, obtaining the small files of the same group, and merging the small files of the same group. Through the above scheme, small files can be reduced, memory usage of namenode can be optimized, and the large data platform can accommodate more files.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a file merging method and device for a big data platform. Background technique [0002] In a big data platform, such as a Hadoop cluster, when performing data analysis, there are often a large number of small files in the data directory. The existence of these small files puts a lot of pressure on the namenode, resulting in a decrease in the computing efficiency of the cluster by several times or even dozens of times. times. In the prior art, it is necessary to develop functional components for each group of data directories or each type of target data to merge files. [0003] However, existing file merging schemes can only allocate schedules based on time. This maintenance mode has many disadvantages: first, the development content is relatively trivial and the development cost is high; second, the scheduling plan cannot be arranged according to the actual data situation, an...

AI Technical Summary

Problems solved by technology

[0003] However, existing file merging schemes can only allocate schedules based on time

This maintenance mode has many disadvantages: first, the development content is relatively trivial and the development cost is high; second, the scheduling plan cannot be arranged according to the actual data situation, and there may not be many small files when the task is started, which wastes cluster computing resources, or the task is executed. There are new files written into the directory, and the problem of the number of small files has not been well resolved; the third is that the computing resources for each file processing cannot be dynamically applied according to the actual situation

Images