Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Visualized big data workflow configuration method and system based on WEB

A workflow and big data technology, applied in the field of big data processing, can solve problems such as long startup time, achieve the effects of reducing personnel costs, improving system performance, and improving scheduling

Inactive Publication Date: 2018-05-15
北京博宇通达科技有限公司
View PDF3 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

All configuration data and program files of the open source workflow engine are stored on HDFS. When the task is running, it needs to be downloaded to the local hard disk and then started. It takes a long time to start

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Visualized big data workflow configuration method and system based on WEB
  • Visualized big data workflow configuration method and system based on WEB
  • Visualized big data workflow configuration method and system based on WEB

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0060] figure 2 Shows the flow chart of the WEB-based visualized big data workflow configuration method of this embodiment, refer to figure 1 , the method may include:

[0061] Step 11, the interface layer sets up a visual process design component, and receives the big data processing business process designed through the process design component and the corresponding data processing node.

[0062] In this embodiment, the interface layer mainly provides a user-friendly visual interface and provides corresponding design components. These design components can be pushed and pulled at will and combined arbitrarily according to user needs and operations. According to the user's combination and design, the big data processing business processes are formed, and these business processes correspond to the corresponding data processing nodes. That is to say, users control and combine data processing nodes by operating design components, and form a big data processing business proces...

Embodiment 2

[0080] Such as image 3 As shown, a WEB-based visualized big data workflow configuration system is disclosed, wherein,

[0081] The interface layer 21 is used to set the visual process design component, receive the big data processing business process designed through the design component and the corresponding data processing node; display the execution result of the big data processing business process;

[0082] The engine layer 22 is used to sequentially schedule the data processing nodes according to the big data processing business process; package the data processing nodes into a distributed program; receive and organize the execution results of the distributed program; The execution result is returned to the interface layer 21;

[0083] The execution layer 23 is configured to execute the distributed program in the distributed program running environment and return the execution result to the engine layer 22 .

[0084] Further, the engine layer 22 includes a workflow en...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a visualized big data workflow configuration method and system based on the WEB. Workflow configuration is achieved through a browser by the drag method and an wizard interface, the development efficiency is improved, and the personnel cost is reduced. Meanwhile, complete MapReduce / Hive / Spark support is provided, the current mainstream big-data processing programs refer toHiveSQL, MapReduce, Spark and SparkSQL, and the method attempts to provide thorough support to the big-data processing programs. Moreover, according to the scheme, the running speed can be improved, an lightweight distributed workflow engine is achieved, the scheduling speed and running speed of the distributed programs are improved, and performance of the system is improved.

Description

technical field [0001] The present disclosure relates to the technical field of big data processing, in particular to a WEB-based visualized big data workflow configuration method and system. Background technique [0002] In the big data environment, the data sources are very rich and the data types are diverse. The amount of data stored and analyzed and mined is huge, and the requirements for data display are high, and the efficiency and availability of data processing are very important. [0003] Traditional data collection has a single source, and the amount of data stored, managed and analyzed is relatively small, and most of them can be processed by relational databases and parallel data warehouses. In terms of relying on parallel computing to improve data processing speed, traditional parallel database technology pursues high consistency and fault tolerance. According to the CAP theory, it is difficult to guarantee its availability and scalability. [0004] The tradit...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F8/34G06F9/445G06Q10/10
CPCG06F8/34G06F9/44505G06Q10/103
Inventor 张祺君黄超曹正凤
Owner 北京博宇通达科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com