Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Decentralized big data job flow scheduling method and device

A technology of decentralization and scheduling method, which is applied in the direction of structured data retrieval, multi-program device, electronic digital data processing, etc. It can solve the problems of system paralysis, the impact of job flow scheduling processing, and the failure of working nodes to perform normally, etc., to achieve Improved stability and robustness effects

Inactive Publication Date: 2021-03-30
京信数据科技有限公司
View PDF4 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The above implementation methods all have the risk of centralization of the master node. Once the master node fails, all working nodes will not be able to operate normally, resulting in the paralysis of the entire system.
All big data job scheduling tasks will not run normally, and job flow scheduling processing will be greatly affected

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Decentralized big data job flow scheduling method and device
  • Decentralized big data job flow scheduling method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus their repeated descriptions will be omitted.

[0032] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one o...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a decentralized big data job flow scheduling method and device. The workflow scheduling method comprises the following steps: S1, a workflow is created, the workflow is converted into a process instance, an operation command is executed on the process instance, and the operation command is written into a workflow command table; S2, an Master cluster competes to obtain commands in the workflow command table, analyzes corresponding process instances, splits the process instances into task instances, and writes the task instances into a task queue; and S3, the Worker cluster competes to obtain the batch tasks in the task queue, instantiates the batch tasks and then executes the batch tasks. The Master cluster and the Worker cluster are arranged in a decentralized mode, meanwhile, one server is selected as a manager to execute a task by adopting a distributed lock, all nodes can serve as Maste / Worker functions, a single manager different from other nodes does not exist, and the reliability of the system is improved. The situation that the whole big data job scheduling is stopped due to the fact that the centralized Master node breaks down is avoided, and the stability and robustness of the system are improved.

Description

technical field [0001] The invention relates to a big data job flow processing technology, in particular to a decentralized big data job flow scheduling method and device. Background technique [0002] In the era of big data, no matter what industry it is, it is greatly affected by big data. With the wide application and popularization of big data technology, more and more tasks such as data processing and data analysis need to use big data clusters such as Hadoop and Spark to complete calculations. At the same time, a data analysis transaction needs to be assembled and completed by multiple computing subtasks in the form of workflow, and a strict scheduling execution strategy is formulated according to the sequence of workflow configuration. In order to further improve the flexibility of programming and the efficiency of job execution, the management of big data job scheduling is usually carried out by building a unified big data job flow management platform, and all needs...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F9/52G06F16/21G06F16/27
CPCG06F9/4881G06F9/524G06F16/21G06F16/27
Inventor 王济平黎刚汤克云周健雄高俊杰
Owner 京信数据科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com