Database storage engine acceleration method and system based on an SPDK

A storage engine and acceleration system technology, applied in file systems, electrical digital data processing, digital data information retrieval, etc., can solve low-latency IO storage software stack long data path performance bottlenecks, lock competition, and cross-core access CPU scheduling inconsistencies. In time, the performance improvement effect cannot be achieved, which can optimize the read service scheduling mechanism, facilitate debugging and analysis, and shorten the critical path.

Active Publication Date: 2021-11-23
上海沄熹科技有限公司
View PDF12 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] ①The traditional storage software stack adopts a stacked modular design, which can reflect the flexibility of storage software stack design and implementation when facing mechanical disks; however, when facing high-performance storage media, some specific optimization strategies such as PageCache cannot give full play to their advantages. Delayed IO makes the long data path of the storage software stack a serious performance bottleneck;
[0005] ②The traditional file system is designed based on the idea of ​​"storage is an IO-intensive application", and the mechanical disk is limited by the structure itself without considering CPU multi-core processing. However, NVMe multi-queue IO processing technology requires the support of multiple processors, so lock competition, cross- Core access and untimely CPU scheduling become new bottlenecks
Therefore, for the storage engine of the LSM-Tree structure type, if the user-mode file system and user-mode NVMe SSD driver of SPDK are only directly applied, the expected performance improvement effect cannot be achieved; Doesn't perform as well as kernel-mode filesystems

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Database storage engine acceleration method and system based on an SPDK
  • Database storage engine acceleration method and system based on an SPDK
  • Database storage engine acceleration method and system based on an SPDK

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0074] as attached figure 1 Shown, the SPDK-based database storage engine acceleration method of the present invention, the method is as follows:

[0075] S1. Bind the NVMe SSD device to the user mode driver of SPDK;

[0076] S2. Format the NVMe SSD device using the user mode file system format tool;

[0077] S3. Encapsulate the interface of the storage engine to operate the user mode file system, and perform normal storage access.

[0078] The user-mode file system and user-mode drivers access NVMe SSD devices, which can reduce the depth of the data IO scheduling stack, avoid context switching and user-mode kernel mode switching overhead caused by system scheduling, and save about 50% of the IO scheduling delay time. The theoretical scheduling delay is only 10us.

[0079] The user state file system of step S3 in the present embodiment is specifically as follows:

[0080] S301, Logical division management of NVMe SSD device space: Divide the physical space of NVMe SSD devi...

Embodiment 2

[0104] The database storage engine acceleration system based on SPDK of the present invention, the system includes,

[0105] The binding unit is used to bind the NVMe SSD device and the user mode driver of SPDK;

[0106] The formatting unit is used to format the NVMe SSD device using the user mode file system grid tool;

[0107] The encapsulation unit is used to encapsulate the interface of the storage engine to operate the user mode file system and perform normal storage access.

[0108] The user status file system in this embodiment includes,

[0109] The division unit is used to divide the physical space of the NVMe SSD device into multiple logical spaces, and each logical space is regarded as an NVMe SSD device for management;

[0110] The pre-allocation unit is used to pre-allocate the file space of the logical NVMe SSD device partition according to the storage engine file size configuration parameters, and set different pre-allocated space sizes for different logical p...

Embodiment 3

[0125] The embodiment of the present invention also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the SPDK-based database storage engine acceleration method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium may be provided, on which a software program code for realizing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU of the system or device) ) to read and execute the program code stored in the storage medium.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a database storage engine acceleration method and system based on an SPDK, belongs to the technical field of databases and storage, and aims to solve the technical problems of how to give full play to the original writing advantage of supporting high throughput of a storage engine, optimizing a read service scheduling mechanism, reducing unnecessary mutual exclusion and improving read concurrency, reducing random access of data, and accelerating the overall performance of a database storage engine. According to the technical scheme, the method comprises the following steps that NVMe SSD equipment is bound with a user mode driver of an SPDK; the NVMe SSD equipment is formatted by using a user mode file system lattice disk tool; and the packaging storage engine operates an interface of the user mode file system and performs normal storage access. The system comprises a binding unit, a formatting unit and a packaging unit.

Description

technical field [0001] The invention relates to the technical field of database and storage, in particular to a method and system for accelerating a database storage engine based on SPDK. Background technique [0002] With the continuous advancement of solid-state drive (SSD) technology and the continuous decline in the price of SSD disks, more and more enterprises are looking at Turning to mixed flash or all-flash storage, it has become the general trend for solid-state drives to replace mechanical hard drives. On NVMe SSD media, due to its multi-queue parallel design, the single-threaded model can no longer take advantage of disk performance. System interruptions and network card interruptions have become new bottlenecks for the CPU. Original storage applications cannot fully utilize the multi-queue capabilities of NVMe SSDs. Parallel processing capability; at the same time, the low-latency read and write access brought by NVMe SSD makes the overhead of application proces...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/13G06F16/16G06F16/172G06F3/06G06F9/52
CPCG06F16/13G06F16/162G06F16/172G06F9/526G06F3/0643G06F3/0644G06F3/0652G06F3/0679G06F3/061Y02D10/00
Inventor 刘芳蕾刘佳星张炜刚
Owner 上海沄熹科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap