Big data management platform

Abstract

The invention discloses a big data management platform, which comprises a login interface unit, a large-scale data storage and calculation unit, a data integration unit, a data application developmentunit, a data management unit, a data security unit and a real-time stream calculation unit which are connected with one another, the login interface unit is used for logging in the big data management platform through a user name and a password for use; the large-scale data storage and calculation unit is used for providing a storage scheme from the GB level to the PB level and supporting batch calculation, Spark memory calculation, Kylin multi-dimensional analysis and Impala and streaming calculation schemes; and the data integration unit is used for realizing full-amount offline access andincremental real-time / quasi-real-time access of the relational database and the logs. The method can help an enterprise construct data application, improve development efficiency, help the enterprisesolve the problem of big data landing in a one-stop manner, help the enterprise digitalize and innovate, promote industrial digitalized upgrading, and solve the quality problem in the data productionprocess.

Description

technical field [0001] The invention relates to the field of big data management, in particular to a big data management platform. Background technique [0002] In the development process of traditional software, there are a large number of rich software to ensure that the steps of development, debugging, and release are carried out in an orderly manner. The high quality of the practitioners and the tools for these management processes guarantee the high quality of the online application. However, in the field of big data commercial software, there are few IDEs that are easy to use to help enterprises build data applications, and the efficiency of software development is not high. The main goal of metadata management is to standardize the definition of metadata within an enterprise. However, as data dependence increases year by year, it becomes more and more difficult to track data flow and understand data meaning and blood relationship, and quality problems are prone to e...

AI Technical Summary

Problems solved by technology

However, as data dependence increases year by year, it becomes more and more difficult to track data flow, understand data meaning and blood relationship, and quality problems tend to exist in the data production process

In addition, there are difficulties in the implementation of big data in enterprises, which will hinder the digital innovation of enterprises and is not conducive to the digital upgrading of industries

Images