Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A Hadoop-based cluster task scheduling method and computer equipment

A technology of Hadoop cluster and computer equipment, applied in the field of task scheduling method and computer equipment based on Hadoop cluster, can solve the problems such as failure of resource scheduling, failure, and low cluster processing efficiency when tasks get together

Active Publication Date: 2022-04-12
福建星瑞格软件有限公司
View PDF14 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] As the business continues to increase, the Hadoop-based big data platform handles more and more tasks. The existing Hadoop cluster-based big data platform can split tasks into cluster nodes for distributed processing, but does not prioritize tasks Distinguished scheduling and concurrent quantity control
Since any task can be scheduled, limited by the bottleneck of cluster software and hardware resources, tasks cannot be scheduled due to clustering of tasks, resulting in delays or failures of some high-priority services, and some low-priority tasks are always scheduled, improving cluster processing efficiency Low

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Hadoop-based cluster task scheduling method and computer equipment
  • A Hadoop-based cluster task scheduling method and computer equipment
  • A Hadoop-based cluster task scheduling method and computer equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] Such as figure 1 with figure 2 Shown, a kind of based on Hadoop cluster task scheduling method of the present invention, comprises the steps:

[0030] Step 10: Carry out priority grouping, set the business to have N(1,2..N-1,N) priorities, for example, the priorities can be set in descending order from small to large, and each priority corresponds to a Scheduling queue, each task specifies the corresponding priority according to the importance of the business, enters the corresponding priority scheduling queue, and waits in line according to the time sequence, where N is a positive integer, for example, the value of N is 8, such as figure 2 shown;

[0031] Step 20, perform queuing scheduling, set the maximum number of concurrent tasks in the system and the maximum number of concurrent tasks in the queue corresponding to each priority (used to control the number of tasks with long running times in each queue), and go to each interval according to the order of priorit...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention provides a method for scheduling tasks based on Hadoop clusters, including the following steps: Step 10, setting the business with N priorities, each priority corresponding to a scheduling queue, and specifying the corresponding priority for each task according to the importance of the business, entering The scheduling queue corresponding to the priority is queued and waited according to the time sequence, where N is a positive integer; step 20, set the maximum number of concurrent tasks in the system and the maximum number of concurrent tasks in the queue corresponding to each priority, and each interval is based on the priority Scheduling queue scheduling tasks in high and low order; step 30, automatically assigning queued tasks to task idle priority queue scheduling according to the order of priority; step 40, re-entering the original priority for tasks that fail to run and tasks that exceed the predetermined maximum running time Level queue queuing scheduling. The invention also provides a computer device, which realizes task priority grouping, queuing scheduling, routing strategy and failover, and greatly improves cluster task scheduling efficiency.

Description

technical field [0001] The invention relates to the field of computer distributed system data processing, in particular to a Hadoop-based cluster task scheduling method and computer equipment. Background technique [0002] As the business continues to increase, the Hadoop-based big data platform handles more and more tasks. The existing Hadoop cluster-based big data platform can split tasks into cluster nodes for distributed processing, but does not prioritize tasks Distinguish between scheduling and concurrent quantity control. Since any task can be scheduled, limited by the bottleneck of cluster software and hardware resources, tasks cannot be scheduled due to clustering, resulting in delays or failures of some high-priority services, and some low-priority tasks are always scheduled, improving cluster processing efficiency Low. Contents of the invention [0003] One of the technical problems to be solved by the present invention is to provide a Hadoop-based cluster tas...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48
CPCG06F9/4881
Inventor 蔡伟群
Owner 福建星瑞格软件有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com