Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method for solving historical data change in big data system

A technology of historical data and big data, applied in database indexing, database updating, structured data retrieval, etc., to reduce the amount of calculation and save computing resources

Pending Publication Date: 2021-09-03
众淼创新科技(青岛)股份有限公司
View PDF6 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] The purpose of the present invention is: aiming at the problems described in the background technology, the present invention provides a method to solve the historical data change in the big data system, which solves the problem of when and which part should be recalculated after the historical data changes , so as to save a lot of computing resources under the premise of ensuring the correctness of the business

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for solving historical data change in big data system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0026] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0027] The present invention provides a method for solving the change of historical data in a big data system, which is used to solve the problem that the historical data will change continuously in the scenario of using big data technology. The entire data flow is as figure 1 As shown, the detailed process is described as follows:

[0028] 1. First, all business data is stored in the mysql database. Each business table will have ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method for solving historical data change in a big data system, which comprises the following steps of: setting an update time field for a business table to mark the update time of the field; executing the operation type data storage ODS task once in the early morning every day, checking the updating time field of the service table , and generating a piece of operation type data storage ODS layer data; enabling the zipper table module to check the latest record of all the data of the operation type data storage ODS layer, comparing the latest record with the corresponding record of the last time, and recording the event that the table changes in the day into a change notification table if the latest record changes; and after the zipper table task is finished, executing a task for generating a service daily table, checking the change notification table by the task, executing a corresponding task unit according to the changed table and the changed date, and generating service daily table data. According to the method, the problems of when to recalculate and which part to recalculate after historical data changes can be solved, so that a large number of computing resources are saved.

Description

technical field [0001] The invention belongs to the technical field of big data processing, and in particular relates to a method for solving historical data changes in a big data system. Background technique [0002] As an emerging technology, big data has been widely used in business scenarios such as reporting, analysis, and forecasting. Through big data technology, the data can be processed quickly and can provide various dimensions of support for the business. In most scenarios, for big data technology, it is only necessary to process the input data to obtain the output data, and the subsequent business can use the output data. In general, the historical data is fixed and will not change in the scenario, so the processed output data can be used continuously in the future. A typical scenario is to conduct inductive statistics on daily business data and generate data reports for display. However, if the historical data is allowed to be modified, it will cause the outpu...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/23G06F16/22
CPCG06F16/23G06F16/2282
Inventor 张春剑吴凯潘自滨刘水军
Owner 众淼创新科技(青岛)股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com