Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data association query method and system in RocksDB-based key-value storage mode

A technology of data association and storage mode, applied in the field of operator optimization, to avoid overhead, improve computing efficiency, and improve efficiency

Pending Publication Date: 2020-10-09
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF0 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical task of the present invention is to provide a data association query method and system based on the key-value storage mode of RocksDB, to solve the problem of how to improve the traditional row-by-line read and serial hashjoin calculation efficiency, and improve the efficiency of association query question

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data association query method and system in RocksDB-based key-value storage mode
  • Data association query method and system in RocksDB-based key-value storage mode
  • Data association query method and system in RocksDB-based key-value storage mode

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0054] as attached image 3 As shown, the data association query method based on the key-value storage mode of RocksDB of the present invention, the method is to store each line of data through the channel of the go language under the key-value storage system and then read it piece by piece. Realize block reading, that is, process data in units of blocks. After receiving the data block, distribute the data to different Join Workers for parallel computing to improve the computing performance of the Hash Join operator; the details are as follows:

[0055] S1. Read data block by block instead of line by line; the details are as follows:

[0056] S101. Store the read data into the channel without performing calculations;

[0057] S102. When the amount of data in the channel satisfies the shipping conditions, divide the channel into blocks for calculation.

[0058] S2. Each module communicates through the channel cache, so that each module is executed asynchronously, and the exec...

Embodiment 2

[0077] The present invention is based on the data correlation query system under the key-value storage mode of RocksDB, and this system comprises,

[0078] The main thread is used to read all the small table data and construct a hash table; then start the Table Reader and JoinWorker to start background work, and finally return the Join result calculated by the Join Worker;

[0079] Table Reader thread, used to read the data of the Outer table and distribute it to each Join Worker;

[0080] Multiple Join Worker operators are used to look up the data in the hash table, Join matching Inner and Outer tables, and pass the results to the Main Thread for parallel computing.

Embodiment 3

[0082] The embodiment of the present invention also provides an electronic device, including: a memory and a processor;

[0083] Wherein, the memory stores computer-executable instructions;

[0084] The one processor executes the computer-executed instructions stored in the memory, so that the one processor executes the data association query method in the key-value storage mode of RocksDB as in Embodiment 1.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data association query method and system in a RocksDB-based key-value storage mode and belongs to the technical field of operator optimization. The technical problem to be solved by the invention is how to improve the efficiency of traditional hash join calculation based on line-by-line reading and serial operation. The associated query efficiency is improved, the adoptedtechnical scheme is as follows: the method comprises the following steps: ; the method is applied to a key-value storage system. Each row of data is stored through a channel of go language, and thenthe data is read one by one to realize block reading, i.e., the data is processed by taking a block as a unit, and after the data block is received, the data is distributed to different Join Workers for parallel computing, so that the computing performance of a Hash Join operator is improved; the method specifically comprises the following steps: S1, reading data block by block; s2, all the modules are in cache communication through channels, all the modules are made to be executed asynchronously, and all the modules are made to end execution by sending end marks; and S3, calculating the mutual concurrent execution of the Join Workers.

Description

technical field [0001] The invention relates to the technical field of operator optimization, in particular to a data association query method and system based on a RocksDB-based key-value storage mode. Background technique [0002] RocksDB uses a log-structured database engine, which is written in C++ for better performance. Key and value are byte streams of arbitrary size. RocksDB is specially optimized for fast and low-latency storage devices (such as flash memory or high-speed hard disk). RocksDB will maximize the high rate read and write performance of flash memory and RAM. RocksDB is suitable for many different workload types. From data storage engines like MyRocks, to application data caches, and even some embedded workloads, RocksDB can calmly face these different data workload requirements. RocksDB provides some basic operations, such as opening and closing the database. Read and write support is also provided for advanced operations such as merge and compressi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/2453G06F16/2455
CPCG06F16/24532G06F16/24552
Inventor 牟冠学史大义柴毅
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com