Mixed type processing system and method oriented to industry big data diversity application

Abstract

The invention discloses a mixed type processing system and method oriented to industry big data diversity application. The system comprises a distributed data collection subsystem used for collecting data from an external system, a storing and parallel calculating subsystem used for storing and calculating the collected data, and an integrated resource and system management platform used for managing the stored and calculated data. The storing and parallel calculating subsystem comprises a big data storing subsystem and a big data processing subsystem, wherein the big data processing subsystem comprises a memory calculating engine which is used for providing distributed memory abstract in a shared-nothing cluster and conducting parallel streamlined and thread lightweight processing on the collected data. The mixed type processing system and method can meet the requirements of industry big data diversity service application, the big data processing performance can be improved by ten or more times through acceleration of the memory calculating engine, and the usability, reliability and expandability of the system can be ensured through the integrated management platform.

Description

Technical field [0001] The present invention relates to the field of big data, in particular, to a hybrid processing system and a processing method for diversified applications of industry big data. Background technique [0002] With the vigorous promotion of informatization construction, the government, finance, energy and other industries have initially realized the integration, sharing and utilization of enterprise-level data sources. However, with the rapid development of sensor technology, network technology, and the digitization and networking of the entire society, human society has entered the era of big data. The number of terminals has increased sharply, the frequency of collection has increased significantly, and the amount of data has changed from TB to PB. With the development, the diversity and richness of data, the time-varying nature and the rapid growth have become increasingly prominent, making the management and utilization of traditional industry data facing t...

AI Technical Summary

Problems solved by technology

However, the current various big data systems are logically independent and lack unified centralized management. If they are deployed in separate clusters, there will be problems of data redundancy and low resource utilization that cannot be ignored; if resource utilization is considered, operating Due to factors such as maintenance costs and data sharing, all frameworks and systems are centrally deployed in a single cluster, and they face problems such as resource sharing, competition, and performance interference between each other.

Images