A Task Scheduling Method for Heterogeneous Storage

A heterogeneous storage and task scheduling technology, applied in the field of data processing, can solve problems such as inability to fully utilize the advantages of heterogeneous storage resources, and achieve the effect of reducing job execution time

Active Publication Date: 2020-03-17
INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

It can be seen that although HDFS has distinguished heterogeneous storage resources, the traditional task scheduling method cannot fully utilize the advantages brought by heterogeneous storage resources.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Task Scheduling Method for Heterogeneous Storage
  • A Task Scheduling Method for Heterogeneous Storage
  • A Task Scheduling Method for Heterogeneous Storage

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0042] The present invention will be described in detail below with reference to the drawings and specific embodiments.

[0043] The inventor conducted the following experiments on the basis of studying the prior art.

[0044] First, tested the execution time of local tasks and remote tasks on HDD and SSD.

[0045] figure 1 (a) shows the execution time of local tasks and remote tasks on HDD in 10 Gigabit network. It can be seen that when the data is stored on the HDD, the network delay has less impact on the remote data reading, and the average time of the tasks performed locally and remotely is basically the same. In contrast, figure 1 (b) shows the execution time of local tasks and remote tasks on the SSD in the 10G network. It can be seen that when data is stored on the SSD, the average time consumed by tasks executed locally is less than the average time consumed by tasks executed remotely.

[0046] In addition, the inventor also tested and compared the "Hot" solution when all H...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention provides a task scheduling method for heterogeneous storage, including: 1) determining each task that needs to be executed according to the user job; 2) for each node with computing resources, according to the storage medium of the data of each task , judging the task type of the task if the task is executed on the node; 3) according to the priorities of various task types, setting the priority order of executing the task for each node having computing resources.

Description

Technical field [0001] The invention relates to data processing, in particular to task scheduling of data access. Background technique [0002] With the popularity of computers, mobile phones, tablet computers and other terminal devices, a huge amount of data is generated during people's use of these terminal devices and the Internet. The collection of these data as available information assets is also called big data. (Big Data). In order to respond to the demand for effective capture, management, and processing of massively increasing big data, a big data processing platform came into being. [0003] The big data processing platform is used according to the resources on each node in the current cluster (such as the idle status of the CPU, the remaining status of the memory) and according to the quality of service (Quality of Service, QoS) requirements of each user's job, in the resource and job Make the optimal configuration between. In other words, the aforementioned scheduli...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48G06F9/50H04L29/08
CPCH04L67/10H04L67/1097G06F9/4806G06F9/5038H04L67/61H04L67/63
Inventor 潘锋烽熊劲
Owner INST OF COMPUTING TECH CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap