Big data conversion method and system based on adapter

Abstract

The invention relates to a big data conversion method and system based on an adapter. The big data conversion method based on the adapter comprises the following steps that: receiving a heterogeneous data conversion request, carrying out analysis processing on the heterogeneous data, and configuring the heterogeneous data on the basis of an SQL (Structured Query Language) script; carrying out cache processing on the heterogeneous data subjected to the cache processing, and carrying out operation stage configuration on the configured heterogeneous data; carrying out conversion processing on the heterogeneous data subjected to the cache processing, storing the converted heterogeneous data, and carrying out operation stage operation on the heterogeneous data subjected to the operation stage configuration; and displaying the stored heterogeneous data. By use of the method, a batch data conversion script set is automatically executed, conversion script execution result information is stored, and then, an application analysis report is generated. When the method is adopted, logic errors generated when the heterogeneous data is converted can be reduced, conversion efficiency among the heterogeneous data is improved, and data collection and analysis cost of the heterogeneous data can be lowered.

Description

technical field [0001] The invention relates to the field of data processing, in particular to an adapter-based big data conversion method and system. Background technique [0002] In recent years, with the rapid development and popularization of computer and information technology, the scale of industrial application systems has expanded rapidly, and the data generated by industrial applications has grown explosively. Industry / enterprise big data that can easily reach hundreds of terabytes or even dozens to hundreds of PBs has far exceeded the processing capabilities of existing traditional computing technologies and information systems. Therefore, it is necessary to seek effective big data processing technologies, methods and means It has become an urgent need in the real world. The current total data volume of Baidu has exceeded 1000PB, and the webpage data that needs to be processed every day reaches 10PB~100PB; the cumulative transaction data volume of Taobao is as hig...

AI Technical Summary

Problems solved by technology

However, there are so many heterogeneous data that cannot be seamlessly exchanged with each other, forming data islands, and unable to exert the application value of data

Now, the challenge we face is how to deal with massive amounts of unstructured and uncertain data and extract business value from these data

Images