Distributed data management system of novel scheduling algorithm based on pipelines

Abstract

The invention relates to the technical field of data management, in particular to a distributed data management system of a novel scheduling algorithm based on pipelines. The system comprises an infrastructure unit, a pipeline scheduling unit, a unified service unit and a data management unit; the infrastructure unit is used for managing a network architecture supporting system operation; the pipeline scheduling unit is used for managing scheduling of inter-process communication pipelines; the unified service unit is used for performing unified standardized management on the data; and the data management unit is used for comprehensively managing the data. According to the design, distributed data management can be realized, the fusion degree of the system is improved, and the ductility of the system is enhanced; the transmission effect in the data exchange process can be improved, and the integrity and safety of data are improved; through centralized scheduling management of the data, the architecture facilitates node expansion of the system, can support high-concurrency access of large users, can realize visual management of the full life cycle of the data, shortens the data management cycle, and reduces the cost waste.

Description

technical field [0001] The invention relates to the technical field of data governance, in particular to a distributed data governance system based on a pipeline-based new scheduling algorithm. Background technique [0002] For enterprises, the most important asset is data, that is, the core value of data can be understood as core business value. Realizing data value requires business data analysis and value mining, and only clean and complete data can well present data value. Most traditional enterprises have no awareness of quality management of data resources, and do not know how to implement data governance, resulting in poor data quality, such as missing data, data islands, data distortion, etc., resulting in the inability to use data resources to obtain more business value. Among them, the main reason for the poor data quality is that during the data exchange process, the data transmission channel is not well established and managed, and problems such as data loss and...

AI Technical Summary

Problems solved by technology

Most traditional enterprises have no awareness of quality management of data resources and do not know how to implement data governance, resulting in poor data quality, such as data missing, data islands, data distortion, etc., resulting in the inability to use data resources to obtain more business value

Among them, the main reason for the poor data quality is that during the data exchange process, the data transmission channel is not well established and managed, and problems such as data loss and information leakage are prone to occur during the data exchange process, which in turn leads to subsequent data loss. Governance operations are time-consuming and laborious, and the governance efficiency is low

However, at present, there is no data governance system platform that focuses on managing the data exchange process, resulting in the failure of data collection and scheduling, which affects the construction quality of enterprise information systems and restricts the development of enterprises

Images