A time series data cleaning method and system

Abstract

The present invention relates to a time-series data cleaning method and system, wherein the method includes step 1: collecting a piece of original data, which includes a plurality of original time-series data; step 2: randomly sampling and estimating the original time-series data to obtain a plurality of Estimated data, fill in the gaps and defects generated by random sampling, and obtain multiple pieces of completed estimated data; Step 3: Classify all completed estimated data according to sampling time points, obtain multiple groups of time-classified data, and classify each group of time The data is sorted according to size to obtain multiple sets of sorted arrays; step 4: process each set of sorted arrays to obtain a corresponding average value data, multiple sets of sorted arrays correspond to multiple average values, and multiple average values ​​​​constitute a mean value sequence; step 5: Output the mean value sequence, the mean value sequence is the data from which outliers and high-frequency noise are removed. All-in-one data cleaning to handle missing values, remove wild points and smooth noisy data.

Description

technical field [0001] The invention relates to a time series data cleaning method and system, belonging to the technical field of data processing. Background technique [0002] Data analysis is currently a hot topic in the field of artificial intelligence and database research. The first step in the data analysis process is data preprocessing. Data preprocessing can effectively improve data quality and provide more targeted available data for the data mining core. Not only can save a lot of time and space, but also the mining results can play a better role in decision-making and prediction. [0003] According to statistics, in the entire process of time series data analysis, data preprocessing accounts for 60% of the entire workload, which shows its importance. The reason is that the original time series data often has certain wild points and high-frequency noise, because the data in the real world is often incomplete, noisy and inconsistent, and the real data trend is los...

AI Technical Summary

Problems solved by technology

[0014] The technical problem to be solved by the present invention is to provide a method for removing wild points and high-frequency noise in the original data for the next step, aiming at the shortcomings of the current method that can only process the default value, wild points and noise data separately. For data analysis, time series data cleaning method and system based on Kalman filter and statistical average

Images