System and method for metadata management

Abstract

The invention discloses a system and a method for metadata management. The system comprises a message engine, a node monitor, a metadata management module, a metadata backup module, a directory heat monitor, a directory replication module and a directory migration module. Metadata of the system are distributed in nodes of a metadata cluster instead of a single capacity limited in memory capacity, so that the memory capacity can be expanded horizontally; a dynamic distribution policy is adopted in the distribution of the metadata, so that high availability and high memory capacity of a distributed file system required in enterprise application environment and high expansibility of the memory capacity and access performance are improved.

Description

technical field [0001] The invention relates to the technical field of metadata management in a distributed file system, in particular to a metadata management system and method. Background technique [0002] In the current distributed file system supporting big data, a single-node metadata service model is adopted. First of all, if the metadata node fails due to hardware failure or software error, the metadata information will not be obtained when performing file operations, and the file system will not be able to continue to be used, resulting in the paralysis of the entire system. Although some technical solutions use the Active-Backup (backup algorithm) mode to add backup nodes, when the primary node fails, the process of switching the metadata service to the backup node takes a long time, which cannot meet the high requirements of enterprise applications. availability. Secondly, because the metadata is stored in the memory of the metadata node, and the memory of the m...

AI Technical Summary

Problems solved by technology

First of all, if the metadata node fails due to hardware failure or software error, the metadata information will not be obtained when performing file operations, and the file system will not be able to continue to be used, resulting in the paralysis of the entire system

Although some technical solutions use the Active-Backup (backup algorithm) mode to add backup nodes, when the primary node fails, the process of switching the metadata service to the backup node takes a long time, which cannot meet the high requirements of enterprise applications. availability

Secondly, because the metadata is stored in the memory of the metadata node, and the memory of the metadata node is limited, in the application of big data, the scale of the file system continues to increase, and the size of the metadata increases rapidly. The memory will not be able to accommodate so much metadata, thereby limiting the storage capacity of the file system; in addition, when accessing data in the file system, all access needs to send requests to the metadata node

When the access volume of the file system increases, the metadata node becomes the performance bottleneck of file access

Images