Architecture method of distributed file system supporting mass data access

Abstract

The invention discloses a distributed file system architectural method supporting mass data access. The method is based on a distributed Hash table, and access nodes are obtained by conducting Hash mapping on a file path. By the adoption of the fully distributed type non-centralized architectural design, new nodes can join a cluster after several times of communications. A Kademlia algorithm is adopted for addressing among the nodes, and a routing table is divided and the distance between the nodes is obtained through exclusive-or operation to achieve skip of the most adjacent nodes. A leader is selected through a PaxosLease algorithm to handle the mapping operation on the nodes so as to solve the consensus problem. Block storage of fixed size is conducted on actual data of files, and redundant backup is conducted on the multiple nodes, so that the requirements for safety and distributed calculation are met. When an architectural system generated according to the architectural method processes mass files, the processing efficiency can be remarkably improved, and the good effect can be achieved in the environment with the low latency requirement.

Description

technical field [0001] The invention relates to the field of distributed file system research, in particular to an architecture method of a distributed file system supporting mass data access. Background technique [0002] With the development of Internet technology, "cloud computing" is increasingly valued by people. It is a new oriented technology formed by the integration of traditional technologies such as distributed computing, parallel computing, utility computing, network storage, virtualization, and load balancing. User's service product concept. And "cloud storage" is one of the cloud services closest to ordinary netizens. [0003] In the early distributed file system, files and their metadata information were not backed up redundantly. Once one of the servers fails, the files stored on the server will not be available. And as the number of files increases, the system becomes larger, making it difficult to scale and manage. Modern distributed file systems pay mor...

AI Technical Summary

Problems solved by technology

But its single master node is easy to become a bottleneck, and there is a single point of failure

MogileFS supports the reading and writing of a large number of small files, and can automatically copy files, but it does not support random reading and writing of files. It relies too much on the database and also has a single point of failure.

Images