Elastic key-value key-value pair data storage method based on rdma and htm

Abstract

The invention provides an elastic Key-Value key-value pair data storage method based on RDMA and HTM, the design follows the server-client architecture mode, including: on the server side, an improved G-Cuckoo is provided in combination with a hash graph Hash data management mode, avoiding the problem of kick-out infinite loop between hash tables caused by searching for free positions during data insertion; analyzing the traditional network transmission of messages between the client and the server that requires back and forth round responses Key-value pair storage performance bottleneck problem, use Infiniband remote direct memory access RDMA technology, design RDMA network communication engine, receive data access requests and send back data request results; use hardware transactional memory HTM technology, realize two-stage lock protocol lock operation, guarantee Atomic characteristics of data operations; using key-value pairs for data self-verification check codes to ensure data consistency. The implementation of the present invention can greatly improve the basic operation speed of key-value pair data.

Description

technical field [0001] The invention belongs to the field of computer systems and data storage systems, in particular to an elastic Key-Value key-value pair data storage method based on RDMA and HTM, which can provide fast and scalable data services. Background technique [0002] At this stage, the demand for data-intensive computing is increasing day by day, and the data provided for data-intensive computing are generally stored in storage systems. In addition to storage systems such as relational databases, more and more key-value storage systems are emerging, which can also meet the needs of fast data access. For key-value storage systems, how to obtain low latency and high throughput of data access has become a topic worthy of in-depth study and discussion. [0003] In a distributed cluster environment, it is often necessary to carry out remote data access to key-value storage systems through traditional networks. In this case, both the server side responsible for data...

AI Technical Summary

Problems solved by technology

The advantage is that the query efficiency is high; the disadvantage is that a large number of random writes can easily cause node fragmentation, resulting in performance degradation

(2) Based on Hash table implementation, such as Redis, the advantage of Hash table-based implementation is high query efficiency, but the disadvantage is that it cannot support range query. When a large number of Hash value collisions occur, the reading and writing efficiency will decrease.

(3) Implementation based on Log-Structured-Merge tree (LSM tree), such as levelDB, which has the advantage of high write performance, but the disadvantage of poor read performance and easy to cause serious write amplification

[0010] Generally speaking, the elastic Key-Value key-value pair data storage method based on RDMA and HTM lacks strong software and program support such as high-performance data transmission network protocols, hardware transaction memory management that guarantees the atomic nature of data operations, etc.; On the other hand, there are still problems such as low efficiency of data management mode, no design of data cache to store recently accessed data, poor data consistency mechanism, etc., and the degree of parallelism of data operations needs to be further improved

Images