Real-time and offline business unified processing method and device based on big data technology

Abstract

The invention provides a real-time and offline business unified processing method and device based on a big data technology. The method comprises the following steps: writing business data collected in real time into a KAFKA cluster; processing and integrating the data in the KAFKA cluster based on a real-time service demand and an offline service range to obtain first detail data of the real-time service and second detail data corresponding to the offline service range; and performing analysis processing corresponding to the real-time service on the first detail data, wherein the second detail data is used for performing corresponding analysis processing on the subsequent offline service at a preset moment. According to the method and the device provided by the invention, redundant calculation of repeated access processing of partial data is avoided, and only one set of program needs to be developed to perform unified access processing on the data subjected to real-time analysis mining and offline analysis mining, so that the development time cost of an analysis mining function is reduced, and the form and the structure of the access data of the real-time service and the offline service after processing and integration are unified.

Description

technical field [0001] The present invention relates to the field of big data technology, in particular to a method and device for unified processing of real-time and offline services based on big data technology. Background technique [0002] In all walks of life, attention has begun to be paid to the short-term real-time and long-term uses of business data collected on a daily basis. For shopping websites, the user's click, browse and purchase behavior is crucial to forming an accurate user portrait. For taxi-hailing software, the user's itinerary information is also crucial to predicting the user's demand for taxis in the future. Some data analysis and mining are important for real-time The requirements for real-time performance are very high, such as the hot search list of social software, the hot news that is being viewed, and the number of clicks will be updated in real time, etc., while some data analysis and mining do not have high requirements for real-time performa...

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a method and device for unified processing of real-time and offline services based on big data technology, which is used to solve the problem of separate processing of existing real-time analysis and mining of collected data and offline analysis and mining, resulting in redundant calculation of part of the data, and It is necessary to develop two separate programs to access and process the data of real-time analysis and mining and offline analysis and mining respectively, which increases the development time cost of the analysis and mining function, and the problem of data inconsistency. By writing the real-time collected business data into KAFKA cluster, and then based on real-time business requirements and offline business requirements, the data in the KAFKA cluster is processed and integrated to obtain detailed data, and part of the detailed data is used as the first detailed data for real-time business analysis and mining, and then in Another part of the detailed data is selected from the detailed data as the second detailed data for analysis and mining of offline services, wherein the first detailed data directly enters the analysis and processing link of the corresponding real-time business, and is analyzed and mined to obtain the corresponding real-time business. Analysis results, while the second detailed data is stored as a database detailed wide table, which is waiting to be obtained by offline business requests and then analyzed and mined to obtain the corresponding offline business analysis results

Images