Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Real-time data processing system and method supporting multiple engines

A real-time data processing and multi-engine technology, applied in the field of big data processing, can solve problems such as difficulty in adding new engines, large amount of maintenance, single processing engine selection, etc., to improve operation and maintenance and monitoring capabilities, improve processing capabilities, and apply to scenarios wide effect

Active Publication Date: 2022-04-12
中电福富信息科技有限公司
View PDF5 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

With the development of business and the continuous improvement of new technologies, it is possible to use new engines in the enterprise system platform. However, due to the limitations of the architecture, it is usually impossible to quickly integrate with the original modules based on the status quo of the system, and it takes a lot of manpower to refactor the code.
From the perspective of a single application, in real-time data processing, there are generally situations such as diverse data and complex business scenarios, which usually need to be processed according to the characteristics of the current data and business needs, with high customization; and real-time applications are usually provided by the corresponding personnel. There are problems such as decentralized operation and maintenance and multi-node management for task startup and shutdown.
[0003] The prior art has the following problems: (1) The choice of processing engine is single, and only one processing engine is supported for a certain real-time data processing business, which cannot be freely selected and has low flexibility
(2) The use of multiple engines is limited, and a certain type of engine is limited to a special processing category or a specific field. The use range is narrow, and it is inconvenient to update and iterate
(3) It is difficult to add a new engine. When a new engine needs to be introduced, the existing architecture needs to be greatly modified, and it is difficult to adapt the old process to the new engine.
(4) High customization of real-time processing flow, low code reusability, large maintenance, and poor versatility
(5) Real-time task management lacks support for multiple engines, task processing is scattered, and there is no unified task management module

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time data processing system and method supporting multiple engines
  • Real-time data processing system and method supporting multiple engines
  • Real-time data processing system and method supporting multiple engines

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0033] Such as Figures 1 to 3 As shown in one, the present invention discloses a real-time data processing system supporting multiple engines, which includes an engine management module, a plug-in management module, a process arrangement module and a real-time task scheduling module; the engine management module provides engine design for real-time data processing, To define the basic information, execution information and parameter information of the engine; the engine management module provides the engine bridge design to realize the specific plug-in processing logic according to the API of the engine framework; after the engine is successfully built, it ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a real-time data processing system and method supporting multiple engines, service processing is disassembled, a plug-in for realizing a certain specific function of a service is generated as required, and the plug-in and the engines have no direct relationship; analyzing service requirements to perform processing flow arrangement, dragging a corresponding plug-in to confirm a data processing method and a data flow direction, and obtaining flow arrangement information to construct a data execution link diagram; and obtaining execution engine information according to a data execution link diagram generated in the process arrangement, obtaining actual processing logic through an execution bridge, and calling and executing the actual processing logic. According to the method, plug-in type flow arrangement is supported based on multiple engines, each processing link is allowed to be defined, the flexibility is high, and the use is convenient.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a real-time data processing system and method supporting multiple engines. Background technique [0002] With the development of science and technology, massive data are generated in daily life and economic construction, and enterprises have higher and higher requirements for data timeliness. Making good use of these data in the first place can more effectively improve the competitiveness of enterprises. At present, the real-time data processing technology is becoming more and more mature, and the mainstream processing engines include Flink, Spark Stream, Storm, etc. Usually, the logic code of the real-time processing module is integrated with the processing engine, and minor adjustments have a wide impact on changes to the engine or business logic. With the development of business and the continuous improvement of new technologies, it is possible to use new engines ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/445G06F9/48G06F16/25
CPCY02D10/00
Inventor 林照吴强
Owner 中电福富信息科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products