Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Distributed indexing method and system for efficiently querying streaming data based on LSM

A streaming data and distributed technology, applied in the field of streaming data, can solve problems such as inability to add indexes, space waste, etc., and achieve the effect of maintaining consistency, improving efficiency, and improving efficiency

Active Publication Date: 2021-08-27
WUHAN UNIV OF TECH
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Since this will make it impossible to add indexes later, indexes will be created for all non-primary key columns at the beginning, resulting in wasted space

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed indexing method and system for efficiently querying streaming data based on LSM

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] In order to facilitate those of ordinary skill in the art to understand and implement the present invention, the present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described here are only used to illustrate and explain the present invention, and are not intended to limit this invention.

[0023] please see figure 1 , a kind of distributed indexing method based on LSM high-efficiency query flow data provided by the present invention, comprises the following steps:

[0024] Step 1: Batch update the data;

[0025] The data in memory is incremental data, and the data in disk is baseline data. When the amount of data in memory reaches a certain threshold, incremental data will be continuously merged to disk to generate new baseline data and divide intervals;

[0026] In this embodiment, the data is updated in batches based on the LSM-Tree method,...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed indexing method and system for efficiently querying streaming data based on an LSM. The method comprises the following steps of firstly, updating data in batches, then sorting the sampling data blocks and dividing intervals, establishing a local index for each data partition by adopting an improved R tree generation algorithm, and finally, distributing the data in each data partition and the local index thereof to each child node for storage by the main node, establishing a global index by the main node according to a root node of each local index, and directly storing the global index in the main node. In the process of establishing the local index, a method for improving the R tree is adopted and is changed into a dynamic data insertion mode, and the global index is frequently used, has small data volume and is stored in the main node, so that the data query efficiency can be greatly improved.

Description

technical field [0001] The invention belongs to the technical field of streaming data, and relates to a distributed index method and system, in particular to a log-structure merge tree (LSM)-based distributed index method and system. [0002] technical background [0003] Streaming data is different from the large-scale offline data processed by the Hadoop platform. Streaming data is continuous and real-time data without boundaries. There are roughly two methods for establishing existing distributed indexes. One is to use the Mapreduce parallel computing framework, which is created for query data Establish the corresponding map mapping table, and the client will query through the map table. For distributed databases like NoSql, in order to achieve atomicity in distributed transactions, they need to coordinate with each other. After the system writes data, but the index is not updated in time, there will be consequences caused by inconsistency between the data and the index. ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/13G06F16/182G06F16/22G06F16/23G06F3/06
CPCG06F16/134G06F16/182G06F16/2246G06F16/2365G06F3/061G06F3/0643G06F3/067
Inventor 邹承明冯丹
Owner WUHAN UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com