Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Multi-pipeline scheduling method for distributed memory database

A pipelined and distributed technology, applied in the computer field, can solve the problems of delay in response time, high cost, and inability to preempt SQL queries, and achieve the effect of speeding up the response time and shortening the execution time.

Inactive Publication Date: 2018-12-25
EAST CHINA NORMAL UNIV
View PDF3 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] (2) Non-preemptive scheduling will result in a suboptimal execution order, affecting the response time of high-priority SQL queries
A high-priority SQL query can be adjusted in the waiting queue, but it cannot preempt the currently running SQL query
Unless the system forcibly kills the currently running SQL query, but this will bring expensive costs
Once you let a high-priority SQL query wait, its response time will be delayed
In addition, even in a running SQL query, if the pipeline on the critical path is not allowed to preempt resources, it will affect the completion time of the SQL statement

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-pipeline scheduling method for distributed memory database
  • Multi-pipeline scheduling method for distributed memory database
  • Multi-pipeline scheduling method for distributed memory database

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0079] Embodiment 1: We illustrate how the algorithm in the present invention shortens the execution time of the SQL query by running the third query statement (Q3) in the TPC-H benchmark test set. A physical execution plan for Q3 such as figure 2 shown. It contains 3 pipelines (Pipeline, referred to as P), and each pipeline is divided into multiple stages (Stage, referred to as S) by the data transmission operator (exchange). The instantiation of each stage on different data slices is a task. There is a dependency relationship among the three pipelines, that is, P 2 depends on P 0 and P 1 , but P 0 and P 1 There is no dependency between them, which means that the two pipelines can be executed in parallel. And, since P 0 It involves network transmission and belongs to the pipeline sensitive to network resources. On the contrary, P 1 It is a computationally sensitive pipeline. Here, we assume that P 0 has priority over P 1 . To illustrate the situation, let's assu...

Embodiment 2

[0082] Embodiment 2: In order to describe the preemption of multiple SQL query statements in detail, we introduce the first query statement (Q1) of TPC-H. It contains only one computationally intensive pipeline, denoted as P 3 . we assume it Figure 4a and 4b Medium P 0 and P 1 Enter the system when finished, and compare the two cases of whether preemption occurs at this moment. Figure 4a Indicates that no preemption occurs. At the instant t of the Q1 input 1 , its P 3 As a secondary pipeline to fill the outstanding P in Q3 2 idle resources. at P 2 After completion, P 3 If you monopolize the resources in the cluster, some CPU resources will also not be fully utilized. By analogy, these idle resources can be filled by the pipeline in other SQL statements. Figure 4a Indicates that Q1 is given a higher priority, then it will seize the running resources of Q3 after inputting into the system, where P 2 becomes a secondary pipeline, while P 3 is the main pipeline. ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

An object of the present invention is to provide a multi-pipeline scheduling method for a distributed memory database, the invention dynamically schedules the pipeline with the highest priority amongthe high priority SQL queries according to the available resources in the cluster, moreover, some pipelines of the SQL statement are scheduled to fill the idle resources, which can improve the resource utilization of the whole cluster to reduce the execution time and improve the response time of the high-priority SQL query statements in the distributed memory database. The invention is easy to realize and can achieve considerable practical effects.

Description

technical field [0001] The invention relates to the field of computers, in particular to a multi-pipeline scheduling method for distributed memory databases. Background technique [0002] With the development of computer hardware technology, the main memory of a single commercial server continues to increase. The maximum memory capacity of a single existing server can reach TB level, which can accommodate more data and reduce the cost of traditional disk I / O. However, the memory and computing power of a single server are limited after all, which cannot meet the needs of big data applications. For example, online query processing (OLAP) applications need to process SQL query statement requests on massive data in real time, and these massive data far exceed the memory capacity of a server. Therefore, distributed memory clusters have become a common platform for big data processing, especially for applications with high real-time requirements similar to OLAP. It usually incl...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
Inventor 翁楚良方祝和
Owner EAST CHINA NORMAL UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com