A cross-platform multi-engine supported mass data migration device

Abstract

The invention discloses a cross-platform multi-engine supported mass data migration device. The system comprises an original data storage system, a migration data storage system, a migration flow configuration module, a flow engine execution module and a flow engine scheduling module, wherein the original data storage system is used for storing an original data set to be migrated, and the migration data storage system is used for storing the migrated data set; the migration process configuration module is used for visually configuring the migrated business process logic; the process engine execution module is used for executing an actual migration process; and the flow engine scheduling module is used for regularly scheduling the migrated business flow. The device disclosed by the invention ensures that the data can be migrated by using an engine technology which is most suitable for data characteristics; the processing time delay is reduced to the maximum extent, the processing efficiency is improved, the migration cost is reduced, and the calculation engine can be quickly switched when the data timeliness is changed and the data volume is changed so as to adapt to new data characteristics without the secondary coding development.

Description

technical field [0001] The invention belongs to the technical field of data migration, and specifically relates to a method and system for massive data migration that can run across platforms and support multiple engines. Background technique [0002] The data transfer device refers to the logical and physical aggregation of data from different systems for centralized management, so that users can research and analyze the data and draw valuable conclusions. [0003] The volume of data and the type of data have both increased significantly in the past three decades. Data warehouse technology has grown from scratch. Whether it is the data warehouse methodology proposed by Kimball or Inmon, in order to meet the needs of data storage and data analysis, it needs a A kind of data migration device, this device has also revealed many problems and defectives that need to be improved urgently in the development process of many years. [0004] Traditional data migration devices mainly...

AI Technical Summary

Problems solved by technology

However, in the context of big data with extremely large data volume, these technologies need to be modified according to the requirements of data scale, data storage physical medium, and data processing complexity. running in a computing environment, and may not be able to correlate and process data distributed on different storage media at the same time

Because the computing engine of traditional data transplantation devices mainly relies on relational databases or centralized computing engines, there are limitations in supporting heterogeneous data association processing or distributed engines. Using a distributed computing engine for processing and transplantation requires a lot of coding and commissioning work, which greatly increases the complexity of development, leads to a sharp increase in cost, and causes a great waste of time and money

Images