Data processing method and equipment

Abstract

The invention provides a data processing method and equipment. The method comprises the following steps: receiving non-real time update data to be processed and storing the data in a processing database table; determining a time range beginning value of a this-time processing process according to a time range finishing value of a last-time processing process and a predetermined rollback time; determining a time range finishing value of the this-time processing process; scanning the non-real time update data of which the time stamps in the processing database table are between the time range beginning value and the time range finishing value of the this-time processing process, and inquiring the non-real time update data not processed therein; and starting the this-time processing process to process the inquired non-real time update data. The method and the equipment solve the problem of leakage processing of data records in the time range clinical point of two processing processes, and can achieve a better data processing effect and improve the system reliability.

Description

technical field [0001] The present invention relates to the technical field of computer data processing, in particular to a data processing method and equipment. Background technique [0002] According to the timeliness of data processing, online data processing can be divided into two types: real-time update mode and non-real-time update mode. Among them, the data corresponding to the real-time update mode is data that requires high timeliness. It is reflected in that the subsequent user operations must be based on the results obtained from the previous operation, and are related to the sequence of data operations; the data corresponding to the non-real-time update method is data that does not require high timeliness, and the user does not need to immediately after the operation. It is reflected in the data table, it only needs to be reflected within a certain time interval, and the subsequent operations of each user are not necessarily based on the results of the previous ...

AI Technical Summary

Problems solved by technology

[0005] One way is to perform a full table scan on the data table to be processed, find out the records whose processing status is unprocessed and update them, but since each processing process requires a full table scan of the data table to be processed, the Extremely inefficient processing

[0006] Another way is to first determine a time range of this processing process, and process the unprocessed records within this time range in the data table to be processed, and then determine the subsequent time range for processing in turn, compared to the first Although this method improves the processing efficiency, it may cause some records at the critical point of the time range of two adjacent processing processes to be missed.

The reason for missing processing is mainly because there is a certain time difference between the time stamp of the data record and the time point when the data record actually appears on the data table, that is: when the system generates the data record, although it registers a Timestamp, but because the record is actually inserted into the data table after other operations are performed on the record, there is a time difference between the timestamp of the data record and the time point when the data record actually appears on the data table, for example figure 1 As shown, the time stamp generated by a record to be processed is 10:00. After the addition, other operations such as query, addition, and deletion of SQL were performed, and the previous operations were not submitted until 10:01. Processing, that is, to actually write a record into the data table, although this time difference is very short, but for the computer, if the time range of this processing process is until 10:00, because the a record has not yet appeared in the data table Therefore, it is not scanned, and the start time of the time range corresponding to the next processing process is "greater than 10:00", and the timestamp of this record is 10:00, which does not meet the time requirements for the next scan, thus resulting in a record Leaked processing; the same situation may happen to several data records at the critical point in the time range of two adjacent processing processes, resulting in very unsatisfactory data processing results

Images