Big-data management system

Abstract

The invention discloses a big-data management system, which comprises a big-data application side, a data service providing module, a summarization access interface module, a distributed parallel / longitudinal scheduling module, a grid computing query analysis execution engine, an intelligent data mining module and a discrete-form distributed data memory side. The big-data management system has the advantages that a laterally-parallel and longitudinal grid-type distribution structure which is subjected to time division three-dimensional scheduling can be realized, the application of data management is improved, the three-dimensional distribution of the data is realized, and data search and operation efficiency can be improved since the intelligent data mining module is added.

Description

technical field [0001] The invention relates to a big data management system, in particular to a big data management system based on cloud computing technology. Background technique [0002] With the rapid development of applications such as the Internet, mobile Internet, and the Internet of Things, the amount of global data has exploded. According to the Digital Universe research report released by IDC, the total amount of global information will double every two years. In 2011, the total amount of data created and copied worldwide was 1.8ZB. IDC believes that by 2020, the total number of servers owned by all IT departments in the world will increase by 10 times, and the data managed will increase by 50 times. It is estimated that by 2020, the world will have a total of 35ZB of data volume. The rapid growth of data volume indicates that we have now entered an era of big data. However, at present, not only the scale of data is getting bigger and bigger, but also the varie...

AI Technical Summary

Problems solved by technology

However, at present, not only the scale of data is getting bigger and bigger, but also the variety of data types and real-time processing requirements have greatly increased the complexity of big data processing

Big data brings the following technical challenges to traditional data management systems: 1) Traditional data management systems generally can only handle TB-level data volumes. However, big data is often at PB-level or even EB-level, and most parallel databases support limited expansion. Generally, it can be expanded to a scale of hundreds of nodes, and there are no application cases of thousands of nodes. Traditional data management systems cannot handle the high scalability and massive demand of big data; 2) Big data covers various types of data, including structural Structured, semi-structured and unstructured data, the analysis of different types of data is not the same, traditional data management processing is often only for a certain type of data and management is relatively simple; 3) The improvement of the processing capacity of traditional data management systems depends on the CPU / memory / storage / network update and upgrade, and the processing mode of big data is based on a "scale-out" mode, its performance improvement depends on continuously adding low-cost computing and storage nodes to the distributed system; 4) The traditional data management system is processor-centered, but in the big data environment, it is necessary to adopt a data-centered management system to reduce the overhead caused by data movement. The traditional data management system can no longer adapt to the needs of big data. need

[0003] In short, compared with traditional relational databases, big data has the characteristics of huge data volume, complex structure, and many types, which pose new challenges to the storage, processing and analysis of big data. Moreover, the problem of big data has only recently been recognized by people. It is realized that the existing management system cannot realize the effective management of big data well

Images