The invention discloses an incremental data convergence updating method and
system based on CarbonData, and the method comprises the following steps: 1, carrying out the recognition
processing of incremental data, enabling the incremental data to land to a to-be-made
directory, and enabling a
record to be newly added in a
metadata table; Step 2, triggering a trigger preset in the
metadata table after newly adding the
record, and newly adding a scheduling task on a data increment updating convergence module by the trigger; Step 3, the data increment updating convergence module obtains a scheduling task and determines the current task type according to a trigger type task priority principle; Step 4, performing task logic
processing according to the current task type, and moving the incremental file to an ad
directory; 5, executing the updated and converged
sql statement to converge the increment file increment to a
full scale; and 6, after convergence updating is completed, the corresponding
record is recovered to be in an unoccupied state, and the field content information corresponding to the record is modified. According to the method and the device, the problem that a CarbonDatastreaming
processing scheme does not support incremental updating is solved, and the
workload of service transformation is reduced.