Forest type storage structure and method for distributed erasure code hybrid storage based on multiple storage media

Abstract

The invention discloses a forest type storage structure and a method for a distributed erasure code hybrid storage based on multiple storage media, which are characterized in that data block data of erasure codes are placed in a solid state disk for storage in a distributed storage system, and data check block data of the erasure codes are placed in a mechanical hard disk for storage. The method comprises the following steps: (1) classifying data storage media in a distributed storage system, and establishing a forest type hybrid storage structure; (2) classifying erasure code data in the distributed storage system into data block data and check block data, and marking the data block data and the check block data; and (3) placing the classified erasure code data on a specific tree of a forest type storage structure for distribution and disk falling. In this way, hybrid architecture storage of erasure code data on distributed storage based on multiple storage media is achieved. According to the invention, excessive wear of SSD caused by erasure code write amplification can be solved, the system performance is improved with lower cost, the service life is prolonged, and the reliability is enhanced.

Description

technical field [0001] The invention relates to the field of distributed storage, in particular to a forest-type storage structure and method based on multiple storage media for distributed erasure code hybrid storage. Background technique [0002] Large-capacity, low-cost, high-performance storage system design has always been a hotspot in the field of storage research. As humans have entered the era of big data, the explosive growth of data volume has put forward higher requirements for storage systems. Traditional data management The method has also encountered great challenges, and big data technology has gradually emerged. A very important point in big data technology is how to store and manage big data quickly and efficiently. On the one hand, the storage system must be able to achieve large-capacity storage at low cost, and on the other hand, the performance gap between storage and computing is constantly expanding, which requires high-performance data that matches t...

AI Technical Summary

Problems solved by technology

[0006] Currently, SSDs are more expensive than HDDs due to high material costs and complex manufacturing.

The pure SSD system has excellent performance but the unit storage price is expensive, and the traditional three-copy storage technology leads to only 1 / 3 of the actual data storage utilization rate, and the high price of SSD cannot bear the cost of three copies at all; distributed storage The storage utilization rate of the erasure code storage method is relatively high, which can adapt to the expensive cost of SSD

However, due to the inherent coding properties of erasure codes, erasure codes will generate large write amplification during the use of distributed storage, resulting in excessive wear and tear of SSDs, reduced life expectancy, and reduced reliability.

Images