Distributed file system

Abstract

The invention relates to a distributed file system which comprises a big file storage server, a big file metadata management server and a cache server, wherein the big file storage server is used for storing split big file data blocks, and big files are files of which the size is larger than the preset size; the big file metadata management server is used for storing metadata of the big files, storing mapping information of the big file data blocks on the big file storage server, managing the namespace of the big files and processing the requesting information of a user; the cache server is used for storing small files, metadata of the small files and caching part of frequently visited big files, and the small files are the files of which the size is smaller than or equal to the preset size. According to the distributed file system, the big files and the small files are stored separately, the big files are stored on the big file storage server in a block mode, the small files are stored on the cache server and the reading and writing efficiency of the big files and the small files is improved effectively.

Description

technical field [0001] The present invention relates to the technical field of computer storage, and more specifically, to a distributed file system. Background technique [0002] With the popularization and improvement of cloud computing, more and more users store personal or enterprise data in the cloud. These data include not only large files but also small files. This type of data has the characteristics of large data volume and higher reading frequency than writing Frequency, need for quick retrieval, etc. [0003] Currently, the file systems used by cloud service providers are mainly divided into two categories: Network File System (Network File System, NFS) and Distributed File System (Hadoop Distributed File System, HDFS). The network file system means that the cloud service provider performs a virtual partition on the server, and divides a piece of disk space for the user to store files. Every time the user reads and writes a file, he needs to log in to the remote ...

AI Technical Summary

Problems solved by technology

The network file system means that the cloud service provider performs a virtual partition on the server, and divides a piece of disk space for the user to store files. Every time the user reads and writes a file, he needs to log in to the remote virtual server first, and read and write files on the virtual disk. The defect of this type of system is that all user data is stored on the same server, and if the server fails, it will have a significant impact on the normal operation of users

[0004] Distributed file system refers to a file system in which service providers use multiple servers to store data in clusters. Users need to send requests when reading and writing files. The background server processes user requests and returns the request results to users. It is currently the most widely used The most popular distributed file system is HDFS, but this system has two main defects: it cannot efficiently store a large number of small files and only a single named node for global management

The disadvantage of this method is that the name server is mainly responsible for processing user requests, and the storage space is fixed. When the amount of data becomes larger and larger, its performance will become a bottleneck restricting the development of TFS

And when the name server is severely faulted and data is lost, the backup name server needs to synchronize data with the name server while responding to user requests. At this time, the load of the backup name server is too large

In the MapR file system, file data blocks and metadata are stored on nodes at the same time, which overcomes the bottleneck of a single naming server, but storing large files and small files together at the same time wastes storage resources and is not easy to manage

[0005] The current distributed file system cannot effectively store small files and solve the problem of a single management node

Images