Big data stream processing method and device based on cloud native architecture

Abstract

The embodiment of the invention discloses a big data stream processing method and device based on a cloud native architecture, electronic equipment and a storage medium, relates to the technical fieldof big data stream processing, and is used for avoiding resource occupation and plug-in conflict of big data stream processing task execution and improving platform operation safety and stability. The method comprises the following steps: receiving a big data stream processing task sent by a server, wherein the task comprises required plug-in mirror image information, Git warehouse address information and input and output information; informing the Docker Engine to pull and run the corresponding plug-in mirror image from the mirror image warehouse, so as to enable the plug-in to start to run;enabling the plug-in to access a Git warehouse after running to obtain task context data; enabling the plug-in to acquire data from the data source and perform data processing, and record a task processing result and log information into a Git warehouse and update task context data after processing is completed; and informing the server that the task is executed completely, informing the Docker Engine to destroy the running plug-in instance, and releasing occupied resources.

Description

technical field [0001] The present invention relates to the technical field of big data flow processing, and in particular to a big data flow processing method, device, electronic equipment and storage medium of a cloud-native architecture. Background technique [0002] In the process of rapid development of enterprise informatization, the integrated processing of data streams has always been an area of ​​focus. With the advent of the new technology era represented by cloud computing, big data and artificial intelligence, the original data integration platform software is facing Faced with greater challenges, facing more emerging data source types from relational databases to NoSQL databases, NewSQL databases, message middleware, etc., transmission protocols also coexist from JDBC to REST, RPC, etc., which need to be dealt with The amount of data has also changed from the unit of GB for relational data to the unit of TB or even PB in the era of big data, and the resources (C...

AI Technical Summary

Problems solved by technology

Under the traditional technical framework, the data integration processing platform realizes the access and processing of various data by increasing the functions of its own software and the plug-in modules provided by itself. It often faces the problem that the database driver is not upgraded and updated in time, and the driver version used between the plug-ins A series of problems such as conflicts, excessive resource usage, and resource inability to be released. In addition, allowing third-party plug-ins to directly interact with the platform during the process can easily lead to security issues, affecting the security and stability of the platform itself.

Images