Airflow-based distributed asynchronous task construction and scheduling system and method
A technology of distributed tasks and asynchronous tasks, which is applied in the field of distributed asynchronous task construction and scheduling system, can solve the problems that the task scheduling platform cannot meet the business system, the timed task management and configuration are chaotic, and the integration of machine resources is difficult, etc., so as to improve the output. Stability, simple structure, and the effect of improving the efficiency of task execution
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0036] Example 1: Constructing distributed asynchronous tasks based on Airflow.
[0037] Such as figure 2 As shown, its workflow is as follows:
[0038] S1: The interface call module API Server receives an image transcoding task request sent by the user, and parses the request parameters of the task creation request. The request parameters include: specify the image transcoding task template image-template, and the scheduling period is @once (ie Execute immediately after reading the script, and execute once), task command parameters, task queue image, failed retries are 3, task weight is 3 (the larger the number, the higher the weight), among them, the task command parameters are a set of JSON data Object, which can obtain task command execution data sets through data operations such as serialization and deserialization;
[0039] S2: Transfer the request parameters obtained in S1 to the Caster task construction and distribution module through the RPC transmission protocol. ...
Embodiment 2
[0046] Such as image 3 As shown, its working process is as follows:
[0047] S1: The task scheduling module Scheduler periodically polls whether the registered DAG tasks in the metadata database need to be executed;
[0048] S2: Let the interface be called to generate a picture transcoding task. At this time, there is a picture transcoding task script (DAG) in the task script directory. When the task scheduling module Scheduler reads the DAG script, it will create a DagRun instance. And generate a DagID to associate with this instance; a DAG task is composed of multiple tasks with dependencies, and each task has a taskID. Here, our image transcoding task has three tasks, which are the transcoding task execute and task Successful execution callback success_callback, failure callback failure_callback; among them, the executed functions of the two tasks of success_callback and failure_callback are both a callback function; tasks will be sorted according to the task weight and e...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com