Cloud storage data synchronizing framework and implementing method thereof

Abstract

The invention provides a cloud storage data synchronizing framework based on analyzing the shortcoming of a single point failure of central server nodes Namenode, which exists in a hadoop distributed file system (HDFS) module in the current Hadoop framework. The cloud storage data synchronizing framework adopts a double-center-server architecture, and double center servers serve on line simultaneously, so that data consistency is realized; and a three-machine Paxos algorithm aiming at the double center servers and based on a distribution consistency Paxos algorithm is designed, so that a data synchronizing framework Quorum is formed, and read-write operation on the architecture is standardized. By the data synchronizing framework Quorum, the problem of service interruption under the condition of the signal point failure of the nodes Namenode is well solved, so that a system can provide correct data read-write access outward under the condition that a certain server fails, and final consistency of data can be ensured.

Description

technical field [0001] The invention belongs to the field of data cloud storage, and in particular relates to the design and realization of a data synchronization framework. Background technique [0002] With the rapid development of the Internet industry, the storage and real-time processing of massive data has become an urgent problem to be solved in the computer industry. Traditional relational databases can no longer handle the growing unstructured data in massive data, while distributed data solutions represented by Hadoop have begun to become the focus of the industry. [0003] The Hadoop framework has become the preferred framework for massive data processing, and is even known as the "golden key to connect massive data processing in the 21st century". As the basic module of Hadoop, HDFS provides users with a distributed file system. HDFS adopts the classic master / slave architecture. A cluster built with HDFS is usually composed of a Namenode node as the master and ...

AI Technical Summary

Problems solved by technology

However, this structure of HDFS also has disadvantages: as the central server of the master in its master / slave architecture, the Namenode node is a single node, which means that if the Namenode server fails, it will cause the collapse of the entire file system

In large-scale cluster applications, it is still the bottleneck of system efficiency;

[0010] 2. Due to the need to synchronize data between the master and backup machines, and the frequency of synchronization ranges from a few seconds to a few minutes, some data must be lost after the Namenode fails;

[0011] 3. Manual intervention is required for active-standby switchover. There must be a certain time interval between the system failure alarm and the manual switchover of the standby machine. During this period, the system is also inaccessible.

Images