File system, file storage method, storage device and computer readable medium

Abstract

The invention provides a file system, a file storage method of the file system, a file storage device of the file system and a computer readable medium. In the file storage method of the file system,at least two levels of indexes are used in the file system, a relation between a category of a file to be stored and block information of the file system is stored in a first level of index in the atleast two levels of indexes, and the file storage method of the file system comprises the following operations: obtaining the category according to the category information; Segmenting the file into aplurality of segments; Querying and obtaining block information associated with the category in the first-level index according to the category information; Under the condition that the correspondingblock has no residual space or the size of the residual space is smaller than that of the fragment, inserting a relation between a continuous extension category representing the same category as thecategory of the file and new block information into the first-level index, and storing the fragment in the block when the size of the fragment is greater than or equal to that of the fragment; And storing the relationship between the block information and the information of the file in other indexes except the first-level index in the at least two levels of indexes.

Description

technical field [0001] The present disclosure relates to the field of computer technology, and more specifically, to a file system, a file storage method for the file system, a file storage device for the file system, and a computer-readable medium. Background technique [0002] With the advent of the era of big data, data is exploding in an unprecedented way, and the heterogeneity of data is becoming stronger and stronger. The storage and processing of massive heterogeneous data has brought us more and more challenges. Distributed file system (Hadoop Distributed File System, HDFS) is used by many large enterprises to process massive data because of its good stability and high efficiency. [0003] However, it still has some limitations of its own, among which the problem of small files seriously restricts the further development of HDFS. [0004] Block is the most basic storage unit of HDFS. The default size of block is 64M (it can also be set to 128M). It will not occupy ...

AI Technical Summary

Problems solved by technology

HDFS does not exist to effectively process a large number of small files, but is designed for streaming access to large files. If a large number of small files are stored at the same time, HDFS will spend a lot of time saving and backing up metadata. It will put pressure on Namenode's memory, and the speed of file processing will be greatly reduced

Images