Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Streaming computing method and device based on DAG interaction

A stream computing and operator technology, applied in computing, multi-programming devices, program control design, etc., can solve problems such as large network and time overhead, affecting system performance, and inability to satisfy customers in operating scenarios, and improve processing efficiency Effect

Pending Publication Date: 2020-10-16
BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF0 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although it can solve some streaming computing services, for some computing scenarios with large data volume and high requirements for real-time and precision, such as event level scenarios and stateful computing scenarios, the computing accuracy and performance of SparkStreaming will be reduced. has declined, and for scenarios where calculations are distributed on multiple machines and data transmission is delayed, SparkStreaming's running scenarios cannot meet customer needs
On the other hand, the program of each node in DAG is encapsulated in different computing logic, so in complex business logic, the initialization of job flow and its nodes and related class loading will inevitably bring large network and time overhead , affecting system performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Streaming computing method and device based on DAG interaction
  • Streaming computing method and device based on DAG interaction
  • Streaming computing method and device based on DAG interaction

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0019] Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0020] In the present disclosure, unless otherwise stated, using the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship or importance relationship of these elements, and such terms are only used for Distinguishes one feature from another. In some examples, the first element and the seco...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a streaming computing method and device based on DAG interaction, and relates to the field of data processing and cloud computing. The streaming computing method based on directed acyclic graph (DAG) interaction comprises the following steps: generating first DAG job flow description information according to a first DAG node graph consisting of DAG nodes belonging to a first type set; converting the first DAG workflow description information into second DAG workflow description information by adapting DAG nodes belonging to a first type set into DAG nodes belonging toa second type set and suitable for an Flink engine; packaging the second DAG workflow description information into a DAG execution packet, wherein the DAG execution packet comprises the second DAG workflow description information and operational logic of nodes in the second DAG workflow description information; and sending the DAG execution packet to the job running cluster.

Description

technical field [0001] The present disclosure relates to data processing and cloud computing, and in particular, to a DAG interaction-based streaming computing method and device. Background technique [0002] Streaming computing is a computing method different from batch computing. It can analyze large-scale continuous data in real time during the changing movement process, extract useful information, and quickly transmit the computing results to the downstream. node. After the user configures the batch interval (batch interval), checkpoint (checkpoint) and other information, and composes the DAG of job flow execution by dragging and dropping, the DAG graph is processed with the help of the idea of ​​SparkStreaming microbatch to convert it into an executable The program logic is then submitted to the computing cluster for execution. In addition, in the current real-time workflow computing solution, each node in the DAG adopts decentralized management, that is, each node co...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F9/50G06F9/54
CPCG06F9/4881G06F9/5027G06F9/5072G06F9/546G06F9/541G06F16/9024
Inventor 陈伟旭叶青
Owner BEIJING BAIDU NETCOM SCI & TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com