Multi-dimensional data analysis method and system based on Kylin OLAP

Abstract

The invention provides a multi-dimensional data analysis processing method and system based on Kyl in OLAP. The method comprises the steps of collecting a data set from a database regularly and writing the data set into an HDFS file system; obtaining structure information of a data table in the database of a hadoop cluster, and generating a plurality of multi-dimensional cubes according to configuration of the structure information; synchronizing the data set in the HDFS file system through a Kyl in multi-dimensional engine, and constructing and generating a plurality of cube segments according to the data set and a multi-dimensional cube, and storing the cube segments in an Hbase database; and in response to a query statement input by a user, searching the corresponding cubic segments in the Hbase database through the Kyl in multi-dimensional engine, and returning a data result in the cubic segments. Based on a Kyl in OLAP engine and a Saiku tool, a high-customization multi-dimensional data analysis report is realized, manual integration is not needed after data dimensions are configured, the dimensions are freely and flexibly combined according to service requirements, a system automatically executes data scheduling, aggregation, calculation and presentation according to multiple combination modes such as parallelism and level, and multi-form presentation of reports, charts and the like is supported.

Description

technical field [0001] The invention relates to the field of multidimensional data analysis, in particular to a Kylin OLAP-based multidimensional data analysis method and system. Background technique [0002] At present, when each company manages the internal data of the group, it needs to build multiple systems to deal with the data needs of different business fields. Among them, each field application needs to call the data of multiple systems for manual integration, statistics and analysis. The integrated analysis process of existing data is as follows: figure 1 As shown, including: (1) The business party proposes report requirements and statistical dimensions; (2) The business party logs in independently or the system maintainer confirms the feasibility of the report requirements: the number of systems involved, whether the existing data supports it, and whether it can be obtained through query Support and confirm the plan; (3) If the system supports it, the business p...

AI Technical Summary

Problems solved by technology

[0004] (1) The data in each field is managed and maintained by multiple systems separately, resulting in scattered data sources, data fragmentation, high data integration costs, cumbersome and inconsistent data authority settings, etc.;

[0005] (2) The data statistics dimensions of each system, query conditions, and functional solidification, data analysis and reference are difficult to meet the diverse needs of the business, and various needs need to be implemented into system functions, but this will also lead to high system development costs and slow efficiency. The functional structure is cumbersome and the data is redundant;

[0006] (3) Customized data extraction requirements are all triggered by platform operation and maintenance personnel manually executing scripts, which increases manual maintenance costs and increases the risk of inconsistency between data and requirements;

[0007] (4) The comprehensive report requires the business side to repeatedly log in to multiple systems for operation, data recording and integration, which increases labor costs and increases the risk of data errors, affecting work efficiency;

[0008] (5) Changes to changing business rules require modification of the system code, which cannot be adjusted dynamically, and it is prone to missing changes, wrong changes, resumption of work, etc., and the overall configuration is not flexible enough;

[0009] (6) Data viewing is scattered, which is not conducive to monitoring data conditions and fault analysis

Images