Batch data processing method supporting breakpoint reget

Abstract

The invention discloses a batch data processing method supporting breakpoint reget, which belongs to a batch data processing method and comprises a breakpoint, a set of architecture and a group of regulations; the breakpoint is a computation node of batch computation; the architecture is a technical architecture and a platform for supporting background batch program running, and comprises a monitoring interface, a master control scheduler, a run logic judgment unit, breakpoint control thread containers, a batch definition maintenance interface; the regulations are used for instructing agreement needs to be obeyed by the developed breakpoint service when the architecture is used in an application system, and include a call interface, a breakpoint configuration file, a batch configuration file, a log regulation and a version management regulation. Compared with the prior art, the batch data processing method supporting breakpoint reget improves the flexibility and operability of the system, improves the tolerance and fault recovery capability of the system on local data flaw and exerts the computation capacity of the hardware and software system to the maximum extent.

Description

technical field [0001] The invention relates to a batch data processing method, in particular to a batch data processing method supporting breakpoint battery life. Background technique [0002] In application systems that process massive amounts of data, background data batch processing is often the core of the system, but it is also a process that consumes a lot of resources and time. The traditional linear batch processing program often faces the following difficulties in actual use: 1. Difficulty in troubleshooting: During the batch calculation process, an error in a certain piece of data often occurs and the calculation process is interrupted. However, it is almost impossible to accurately locate this piece of data in the massive data without a complete data on-site record. Finding errors through recalculation takes a lot of time from the start of operation to where the error occurs. If the error occurs near the end of the batch, it will be a disaster for the system ad...

AI Technical Summary

Problems solved by technology

[0002]In application systems that process massive amounts of data, background data batch processing is often the core of the system, but it is also a process that consumes a lot of resources and time

The traditional linear batch processing program often faces the following difficulties in actual use: 1. Difficulty in troubleshooting: During the batch calculation process, an error in a certain piece of data often occurs and the calculation process is interrupted

However, it is almost impossible to accurately locate this piece of data in the massive data without complete on-site data records

Finding errors through recalculation takes a lot of time from the start of operation to where the error occurs. If this error occurs near the end of the batch, it will be a disaster for the system administrator

2. Difficulty in recalculation: After eliminating errors, recalculation not only takes a lot of time (the time spent in general batches is often measured in hours or even days), but also needs to carefully clean up the site where the previous error occurred, and work Large amount, high risk

3. Difficulty in debugging: If it is necessary to adjust the system and perform trial calculations, even a small adjustment needs to run the entire batch process completely, wasting a lot of time

Images