Data processing method and system

Abstract

The invention discloses a method and a system for data processing. The method comprises the following steps: analyzing raw data arriving in real time; extracting at least one field required by data statistics and the value of each field from the analyzed raw data according to configuration information; carrying out data statistics by a preset time window according to the configuration information, at least one field, the value of each field and acquisition time corresponding to the value of each field so as to obtain a key, a key value corresponding to the key and the current time window value corresponding to the key value; carrying out increment updating on the key value corresponding to the key in a key value memory system by the time window according to the key, the key value corresponding to the key and the current time window value corresponding to the key value. Through the adoption of the technical scheme, increment updating can be carried out on the data arriving in real time in a time window sliding form, the defect in the prior art that massive processing needs to be carried out after data collection is finished once so that real-time data cannot be timely processed is overcome, and the processing of the real-time data is extremely convenient.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data processing method and system. Background technique [0002] In many fields such as precise recommendation, real-time monitoring, real-time marketing, data mining, etc., it is necessary to perform real-time statistics on the real-time arrival data according to the time window, such as real-time statistics of clicks, exposures or monitoring indicators. [0003] In the existing technology, data statistics based on time windows are generally based on Hive and Hadoop data warehouses. The more common mode is to first transfer the data offline to a distributed file system (Hadoop Distributed File System; HDFS) of a Hadoop cluster. , the data is partitioned and stored according to Hive's two-dimensional table format and time windows such as days or hours. Then use the Structured Query Language (Structured Query Language; SQL) language Hibernate Query Language (Hib...

AI Technical Summary

Problems solved by technology

[0004] In the process of realizing the present invention, the inventor found that the prior art has at least the following problems: in the data processing of the above-mentioned prior art, MapReduce is a computing method of batch processing, and the data processed by Map is first stored in the disk, and then It is transmitted to the Reduce stage for processing. This calculation mode needs to collect the data first and then process it in full at one time, but cannot process the real-time arriving data, so that incremental updates cannot be achieved. It is very inconvenient to process real-time data and has a large transmission delay

Images