Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

An Adaptive Configuration Method for Cluster Scheduling of Short-term Changing Big Data Jobs

A cluster scheduling and configuration method technology, applied in multi-programming devices, neural learning methods, electrical digital data processing, etc., can solve problems such as manual adjustment, unified optimal scheduling, etc., to improve training efficiency and final effect, good heterogeneity Effects of Load, Increased Efficiency and Final Effect

Active Publication Date: 2022-02-08
BEIJING INSTITUTE OF TECHNOLOGYGY
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The scheduling of the cluster needs to be adjusted in time according to the changing jobs. Therefore, it is necessary for the operator of the cluster to manually adjust the scheduler and there is no unified optimal scheduling applicable to all situations. two big problems

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • An Adaptive Configuration Method for Cluster Scheduling of Short-term Changing Big Data Jobs
  • An Adaptive Configuration Method for Cluster Scheduling of Short-term Changing Big Data Jobs
  • An Adaptive Configuration Method for Cluster Scheduling of Short-term Changing Big Data Jobs

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0122] Embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. like figure 1 As shown, the method of the present invention can be divided into four parts, which are the history log-driven sample generator, the experience playback memory, the Agent module based on reinforcement learning and the Env module responsible for interacting with the environment. image 3 It is a flowchart of the method of the present invention; as Figure 1-5 Shown, method of the present invention comprises the following steps:

[0123] Step 1, initialize each module:

[0124] The Controller module is used to initialize the Env module and the Agent module and control the execution of the code in the pre-training phase, the formal operation phase, and the evaluation phase; the Controller module is divided into two subcategories according to the requirements of the system when it is running;

[0125] The Controller module is used in the nor...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses an adaptive configuration method for cluster scheduling of short-term variable big data operations. The method is aimed at cluster scheduler configuration optimization of cloud platform heterogeneity and dynamic load, and an adaptive cluster scheduler configuration optimization method proposed for cloud platform load heterogeneity and short-term changeable characteristics; Platform load can be divided into service applications and analysis applications. Different classifications have different requirements on resource consumption and time. The method of the present invention adjusts the configuration of the cluster scheduler according to the status information of the job and the information of the cluster environment, so that it is always in the best scheduling configuration, thereby improving job performance, reducing job delay, and better adapting to the different cloud platforms. It can better find the optimized configuration items corresponding to the current cluster state, which minimizes the waiting time of cluster jobs, improves job efficiency, and effectively schedules short-term and changeable big data jobs.

Description

technical field [0001] The invention belongs to the technical field of cluster scheduling, and in particular relates to an adaptive configuration method for cluster scheduling of short-term variable big data operations. Background technique [0002] Currently, cluster scheduling is a necessary prerequisite for performance optimization and resource management of cloud computing systems. A good scheduler can effectively improve the utilization of the cluster and save the user's investment cost, so cluster scheduling has always been one of the hot research directions. Cluster scheduling for short-term big data jobs faces three major challenges: 1. The heterogeneity and dynamics of short-term jobs; 2. How to configure the scheduler, which will determine the performance of the job; 3. No one is suitable for all situations the optimal configuration. For cloud platforms, cluster jobs can be divided into two types: [0003] 1. Service applications: such as search engines (Search ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50G06F9/48G06N3/08H04L67/10H04L67/60
CPCG06F9/5038G06F9/5072G06F9/4881G06N3/084H04L67/10G06F2209/5021G06F2209/484H04L67/60
Inventor 韩锐刘驰刘子峰李泽清
Owner BEIJING INSTITUTE OF TECHNOLOGYGY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com