A data processing method and related equipment

Abstract

The embodiment of the present application provides a data processing method and related equipment, which can comprehensively utilize the advantages of solid-state equipment, tile recording disks, and log-structured merge trees to optimize the stored data. The method includes: acquiring target data; determining a target ordered key-value pair table corresponding to the target data; determining a second ordered key-value pair table according to the target ordered key-value pair table; The value pair table determines the first ordered key-value pair table; merges the second ordered key-value pair table into the first ordered key-value pair table; when the merged first ordered key-value pair table When the table meets the preset condition, write the merged first ordered key-value pair table and the third ordered key-value pair table as an array set into the target dynamic area whose size dynamically changes in the tile record disk .

Description

technical field [0001] The present application relates to the field of storage, and in particular to a data processing method and related equipment. Background technique [0002] With the rapid development of the information age, data in various fields has shown explosive growth. According to the research report of IDC, from 2010 to 2020, the amount of global data will increase by 50 times, and the annual growth rate of the total amount of global data will maintain about 50% in the future. Among them, the amount of data in my country will reach 8.6ZB, accounting for about 21%. In addition, delay-sensitive interactive Internet applications (such as Weibo, Twitter, and Facebook) are gradually increasing. While requiring high response speed, such applications also encourage users to publish and share files and pictures. Therefore, the storage system has been challenged unprecedentedly in terms of capacity and performance. [0003] In terms of hardware, among the various capa...

AI Technical Summary

Problems solved by technology

In terms of software, under the background of big data, the traditional file system or block storage can no longer meet the storage performance requirements of most applications, and the key-value storage based on the log-structured merge tree has become a large-scale data center due to its excellent service quality. The main storage method of the application

[0004] Although solid-state drives, tile-record disks, and log-structure merged tree structures all have such obvious advantages, they also have their own defects, for example: the number of erasing and writing of solid-state disks is limited, first erase and then write; Amplification problem; and log-structured merge tree huge merge overhead

Images