A big data backup method based on virtual shared directory

Abstract

A big data backup method based on a virtual shared directory relates to the technical field of data backup. Through local storage on a media server, a file sharing protocol interface is provided, a virtual shared directory will be created, if the interface is provided to a big data platform A that needs to be backed up, when the big data platform A needs to be backed up, then the sharing right ofthe virtual directory can be obtained by mounting the partition locally. After the backup is finished, the partition is disconnected, and the partition can fall back to the media server. At the same time, the partition can provide the shared directory service to another storage server. Through file replication, the backup of the big data file is realized very concisely.

Description

technical field [0001] The invention belongs to the technical field of data backup, and in particular relates to a big data backup method for improving the efficiency of big data backup. Background technique [0002] The value of data in the era of big data is more critical, and the security of data running on big data needs to be guaranteed, so a faster and more general backup technology is needed to realize data backup of various big data platforms and ensure backup efficiency and compatibility. [0003] At present, the method for data backup of big data generally follows the following architecture, which includes the following parts: backup agent (ie agent), media server, storage medium [0004] The specific implementation details can be roughly divided into the following two types: [0005] (1) Client agent→HTTP→media server→ISCSI→storage medium [0006] The backup agent is installed on the big data host to be backed up, collects the backup data, and transmits the dat...

AI Technical Summary

Problems solved by technology

[0009] In the prior art (1), a corresponding collection client is required for different backup objects, and an agent is required to transfer data from a real data source (such as hadoop namenode) to a local temporary storage directory (on the host), Then the data in this directory is processed by cutting into blocks (for example, one 64K data block each time), and then each data block is transmitted to the media server through the HTTP protocol. After the media server receives it, it undergoes a series of deduplication and compression processing Finally, the data will be transmitted to a special storage medium (such as disk) through the FC network through the ISCSI protocol. The entire process data will go through four key time-consuming steps (namely, agent local temporary storage, local switching, network transmission to the media server, Media server network transmission to storage media), the efficiency of data backup is difficult to be guaranteed, and too many links also increase the operating risk of the system;

[0010] Compared with technology (1), the difference is that after the data is transmitted to the media server, the data is not directly transmitted to the storage medium through the ISCSI protocol, but is cut into blocks again through the HTTP protocol, and the data is transmitted to the object storage through the HTTP protocol Medium (object storage), compared with technology (1), technology (2) is only different in the back-end storage protocol, and the overall storage efficiency and risks are not effectively avoided. At the same time, it is aimed at the collection of multiple types of big data platforms It is also necessary to develop the corresponding client agent agent, and the complexity and compatibility of the backup system have not been improved.

Images