A system and method for online analysis and processing based on big data

Abstract

The invention discloses an online analysis and processing system and method based on big data. The system can perform fast multi-dimensional query and analysis on data sets of different scales and levels under the Hadoop environment. Query plans selected through query plan evaluation include Hive-supported MDX queries and multi-dimensional queries based on Hbase's precomputing caching mechanism. Realize support for MDX query of Hive data warehouse on scalable cluster nodes, multi-dimensional query optimization based on Hbase pre-computing cache mechanism, meet the low-latency multi-dimensional query requirements of data sets of different scale levels, and solve different OLAP data organization models under the background of a single data source OLAP multidimensional query. Aiming at the performance optimization problem of Hive multi-dimensional query on large-scale data sets, a layer-by-layer dimensionality reduction aggregation algorithm based on Hbase cache is proposed. This algorithm brings MOLAP, which solves the multidimensional query calculation of large-scale data, into the big data OLAP system, which greatly enhances the scalability and efficiency of multidimensional query of data at different scale levels under the background of big data.

Description

technical field [0001] The present invention relates to an analysis and processing method under the environment of Hadoop (a software platform for developing and operating large data processing) of big data, and in particular to an online analysis and processing system and method based on big data. Background technique [0002] In recent years, with the continuous development of OLAP (Online Analytical Processing) technology, OLAP system products emerge in endlessly, but most of them are ROLAP (Relational Database Online Analytical Processing) systems based on relational databases or a single MOLAP (Multidimensional Database Online Analytical Processing) system. )system. Although, the continuous enhancement of single-node memory scalability and column-oriented memory database technology has improved the query performance of the ROLAP system. However, the terabyte to petabyte data scale generated by enterprise-level applications has exceeded the maximum query limit that trad...

AI Technical Summary

Problems solved by technology

[0003] 1. The traditional ROLAP system's MDX (a query language that supports multidimensional object and data definition and operation) query is limited by specific databases, and cannot be completed on scalable cluster nodes. Support Hive (a data warehouse tool based on Hadoop) data warehouse MDX query, the traditional ROLAP system architecture has become a bottleneck factor restricting the performance of multi-dimensional query analysis in terms of scalability

[0004] 2. In MDX query, the traditional ROLAP system lacks certain flexibility

Therefore, it only involves low-latitude small-scale data multi-dimensional query, and the performance of aggregation calculation in memory will be restricted by large-scale, high-latitude data volume query requirements

At the same time, when faced with large-scale data processing, the excessive connection operations of ROLAP restrict the query processing performance

Images