A file fragmentation method based on hbase

Abstract

The invention discloses an HBase-based file fragmentation method, which belongs to the field of computers. First read the configuration file for fragmenting large files and the configuration file for reading according to the granularity of fragmentation from the system, and fragment a certain large file. The writing and reading of file content is performed in the order of fragments, and the content of each fragment is stored in the HBase data table with an independent key value through multiple parallel threads. At the same time, the metadata information of each fragment is stored in the HBase metadata information table as a new column. Then, if the writing and reading of the file content is the specified slice, write and read the file content according to the specified fragment number; if the writing and reading of the file content is random reading, set the byte start position and The size of the read content; finally, when the writing or reading is interrupted due to an exception on the server side, just re-upload or read the fragment at the breakpoint after the service is restored. The invention makes large file storage more efficient and reliable, and has strong practicability and adaptability.

Description

technical field [0001] The invention belongs to the field of computers, in particular to an HBase-based file fragmentation method. Background technique [0002] With the development and application of Internet technology, the data generated by various industries such as social networks, mobile communications, online video, and e-commerce have shown explosive growth, and our lives are gradually entering the era of big data. In the era of big data, how to effectively store a large amount of data and then analyze and apply it has increasingly become the key to the development and advancement of enterprises in all walks of life. Especially in the face of large file storage, how to avoid excessive single-point load and how to achieve efficient and reliable storage are problems that need to be solved urgently. [0003] As a column-oriented distributed storage system for unstructured data, HBase provides a solution for distributed high-performance and reliable storage of unstructu...

AI Technical Summary

Problems solved by technology

[0004] Aiming at the problems of high single-point load, inefficient storage and low data security when storing large files, the present invention proposes a file fragmentation method based on HBase, which provides an efficient and reliable solution for fragmentation storage of large files

Images