Data processing method and data processing system

Abstract

The invention discloses a data processing method and a data processing system. The data processing method comprises the following steps: firstly, storing original data to a theme table, and recording codes and names of dimensions which need to generate a dimension table in the theme table; then, generating the corresponding dimension table according to the codes and the names, recorded in the theme table, of the dimensions; storing corresponding dimension table data to the dimension table, and generating the dimension ID of each dimension; generating a fact table related to the dimension table from the theme table according to the dimension ID of each dimension; storing corresponding fact table data into the fact table; finally, generating an application summary table from the fact table as needed to obtain application summary data, and storing to the application summary table. According to the method and the system, by adding the theme table and the application summary table in a data processing cycle, the data can be repeatedly utilized based on the theme table; through derived dimension-supported calculation, the conversion of a data analysis aperture is realize, and the data processing efficiency and the practicability of processed data are effectively improved.

Description

Technical field [0001] The invention relates to the technical field of data processing in a data warehouse, and in particular to a data processing method and system. Background technique [0002] With the rise of the concept of big data, government departments at all levels are actively using various structured and unstructured data generated in the process of government management based on data warehouse technology. The premise of traditional data warehouse technology is that the data has been clarified. How to use it, but the actual demand is that government departments need to collect the data first, and how to use the data has not yet been fully figured out. [0003] The basic principle of the traditional data warehouse technology is to extract the original data from the data source to the temporary area, after unified cleaning, conversion and processing, update to the dimension table and the fact table, through the data visualization tool based on the fact table and the dimens...

AI Technical Summary

Problems solved by technology

However, a series of challenges are faced in the process of using traditional data warehouse technology to build a government data warehouse. Due to the relatively rough caliber and relatively low standardization of historical data, the data management caliber in different years and different regions is inconsistent, and there is a lack of business Unified conversion rules often require business personnel to confirm the conversion methods one by one for the operation data. This kind of workload and difficulty is unimaginable; Large discrepancies prevent operational data from being directly used for analysis, often requiring technicians to develop complex conversion codes and a large amount of temporary computing work, often resulting in low display efficiency and even data inconsistencies, resulting in inability to quickly respond to needs. The usefulness of the data is greatly reduced

Images