Universal description language data system for directed acyclic graph type automatic task flow

A general description and language data technology, applied in the field of cloud computing, can solve problems such as lack of data reuse, lack of parallel description primitives, and inability to provide automatic parallel functions for single-point tasks

Pending Publication Date: 2021-01-01
SHENZHEN JINGTAI TECH CO LTD
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The description granularity of a single task in the existing general description language is very rough. The user needs to define the input parameter group and the acquisition method, but the language does not involve the type and detailed structure of the data, so that the user needs to deeply understand the specific characteristics of the task when using it. Even if the wrong data type and structure are provided, data checking and verification cannot be provided
[0008] 2. The knowledge threshold in the computer field is high, and it is inconvenient for non-computer professionals to organize:
It cannot meet the needs of algorithm writers (computer engineers) and algorithm users (scientists) at the same time
[0010] 3. Huge information and lack of data reuse
[0011] When the existing general description language is applied to high-performance scientific computing, due to its versatility, it is necessary to define a large number of parameters brought about by the characteristics of the scientific computing field, and the parameters need to be repeatedly input, without data templates and data coverage filling capabilities
[0012] 4. Lack of automatic parallel source language
The existing general-purpose description language lacks parallel description primitives like map-reduce or scatter-gather, and cannot provide automatic parallelization functions for single-point tasks

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Universal description language data system for directed acyclic graph type automatic task flow
  • Universal description language data system for directed acyclic graph type automatic task flow
  • Universal description language data system for directed acyclic graph type automatic task flow

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0071] figure 1 Explain the specific hierarchical structure and reference relationship of the language. The specific description of workflow is carried out through four levels, which are TypeDef definition layer, Step definition layer, Workflow definition layer and Template definition layer. The following are examples and introductions of each definition layer:

[0072] The TypeDef layer is not necessary. If users need to use special custom types, they need to write the TypeDef layer. This part mainly abstracts the definition of common or complex composite types, which is convenient for reference and management; the Step layer is a single The description of the task, for the input and output declaration of a docker image or other executors, needs to specifically declare the name, type, document, parameters and other information of each input and output item; the Workflow layer is a workflow composed of one or more Steps, which needs to be Define the dependent topology of the...

Embodiment 2

[0074] xwlVersion describes the version of the description language, which is used to distinguish version iterations brought about by the continuous addition of functions; class describes the type of this document, and there are four types (TypeDef, Step, Workflow, Template); version describes the definition version; author describes the author's information; doc describes the annotations for the document; name describes the name of the document, and the author needs to keep the name unique when writing the same type of document;

[0075] In the description language, a substructure called TypeAndValueDefine is defined, which contains type, name, value and several attributes, and is used to define a variable in detail. The following are three representative TypeAndValueDefine examples:

[0076] $name:

[0077] type: int[]

[0078] const: 1

[0079] value: 1

[0080] ref: $xxx / $xxx

[0081] required: true

[0082] doc: this is a type and value definition demo

[0083] sym...

Embodiment 3

[0102] The following is an example of a TypeDef definition named common (the general information section will not repeat it):

[0103] xwlVersion: 1.0.0

[0104] class: TypeDef

[0105] doc: a structure type def

[0106] author: ziqi.jiang

[0107] name: common

[0108] version: 1.0.0

[0109] typeDefs:

[0110] $jobArgs:

[0111] doc: Contains some info about compute args

[0112] type: record

[0113] fields:

[0114] :cores:

[0115] type: int

[0116] :memory:

[0117] type: int

[0118] The specific steps are:

[0119] Step 1: Define the typeDefs keyword in the outermost layer, and the typeDefs keyword contains some TypeAndValueDefine substructures. For example, the definition declares a record data named struct, and fields is a subkey declaration of record type, which contains two attributes cores and memory;

[0120] Step 2: In the TypeAndValueDefine substructure that uses the type definition, declare it in the type through a fixed-format link. The follow...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a universal description language data system for directed acyclic graph type automatic task flow. The universal description language data system comprises a Step definition layer, a Workflow definition layer and a Template definition layer, The Step layer is description of a single task, and information such as names, types, documents and parameters of input and output itemsneeds to be declared specifically for input and output declarations of a docker mirror image or other actuators; the Workflow layer is a workflow composed of one or more Steps, the dependency topology of the Steps needs to be defined, and shared parameters can also be defined; the Template layer is based on a Workflow definition, and the template is used for presetting parameters, explaining supplementary parameters, and defining an inspector or a data source. The data center and a task execution tool are used, and a programming language needs to be used for achieving a corresponding tool. Adata center needs to store each definition document and indexes the definition documents to the corresponding documents through reference links, and an interpreter needs to read all definition contents and assign corresponding data to a definition structure according to the reference links.

Description

technical field [0001] The invention belongs to the field of cloud computing, in particular to a general description language data system of directed acyclic graph automatic task flow. Background technique [0002] Nowadays, the professional subdivision trend in the field of scientific computing is obvious, and the development of subdivided problem-solving algorithms and practical value-oriented engineering application have been divided into two directions of development. How to combine and use subdivided professional methods to achieve goals has become an indispensable requirement. The smaller and smaller method granularity also makes the learning cost of understanding a large number of methods and the labor cost of combining them higher and higher. Automated workflow This technology is also widely used in various fields. In the field of scientific computing: SBP (Seven Bridge Platform) developed by SBG (Seven Bridges Genomics) in the United States is regarded as the compa...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F8/30
CPCG06F8/315
Inventor 姜子麒温书豪谈樑刘阳马健范陕姗赖力鹏
Owner SHENZHEN JINGTAI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap