Distributed Internet transaction information storage and processing method

Abstract

The invention provides a distributed Internet transaction information storage and processing method. The method comprises the following steps that: an operation data storage function module extracts structured transaction information data from an external data layer; a distributed data warehouse and a non-relational database read non-structured network log data from a distributed file system respectively; an Internet transaction information data warehouse extracts, transforms and loads the transaction information data and the network log data from the operation data storage function module and the distributed data warehouse, integrates the transaction information data and the network log data into a data tuple, and stores the data tuple with a star model; a data market extracts, transforms and loads market-specified data from the Internet transaction information data warehouse and / or the distributed data warehouse; and a business intelligence system acquires data required by analysis from the data market and / or the non-relational database according to data analysis requirements. The method is specific to the application scene of Internet transaction data analysis and processing.

Description

technical field [0001] The invention relates to a method for storing and processing distributed Internet transaction information. Background technique [0002] The models and results of data analysis and processing can only produce real value when they are used in certain decision-making applications, and a series of theories and methods such as assisting decision-making through a fact-based support system are business intelligence (Business Intelligence, BI for short) . With the rapid development of informatization, more and more attention has been paid to business intelligence, especially the joint online analysis OLAP (that is, online rapid analysis based on database and big data), which is to some extent the main processing method of business intelligence. The core data warehouse architecture design in the BI solution such as figure 1 shown. [0003] However, Internet transaction information data is not suitable for BI solutions, the main reasons are as follows: [0...

AI Technical Summary

Problems solved by technology

[0004] 1. The data sources of Internet e-commerce transactions are diverse, including different e-commerce platforms, various relational databases, data generated by social software, and network log data, etc.; therefore, a single ETL tool cannot handle all the data well

[0005] 2. The amount of data in e-commerce transactions is huge, and traditional centralized relational databases cannot meet the requirements of processing such large-scale data

[0006] 3. Internet transaction monitoring has high real-time requirements for data analysis and processing. Most of the monitoring and early warning needs to be processed in a short time to reflect the value of the data. The traditional offline ETL processing method can no longer meet the demand

[0007] 4. The number of Internet transaction users is huge, while the design of traditional BI display tools is mainly aimed at middle and high-level users of enterprises, and there is a big difference in terms of interaction efficiency and user experience when transplanted to the e-commerce industry

The current traditional business intelligence system is not well suited to the large-scale Internet transaction information, heterogeneous data sources and the real-time demand for data analysis

Images