Financial stream data-oriented anomaly detection method

Abstract

The invention discloses a financial stream data-oriented anomaly detection method, which comprises the following steps of: (1) acquiring financial data as a data source by kafka, ensuring stream semantics of the data by using the kafka, and transmitting the data to an Flink end; (2) the Flink platform preprocesses the data; and (3) the outlier detection model performs outlier detection on the stream data by adopting a kernel density estimation (KDE)-based method. And (4) dividing the data space into cells with fixed area sizes by an algorithm, assigning weights to the cells according to the distribution condition of the data objects, and processing the influence of the newly arrived data objects and the expired data objects in the sliding window on the KDE value. And (5) cells with small weight net variation are filtered by adopting a weight-based mode, so that the calculation cost is reduced, and the detection speed is improved. The invention provides a set of flow processing system and an anomaly detection method based on kernel density for anomaly detection of financial flow data, and provides a solution for anomaly detection of flow data.

Description

technical field [0001] The invention relates to an abnormal detection method for financial flow data, in particular to a flow data processing system and an outlier detection algorithm, and belongs to the technical fields of data mining and financial information processing. Background technique [0002] With the continuous development of information technology, the transmission and form of data are constantly changing, and the processing technology of stream data is more and more widely used. At the same time, the anomaly detection mechanism in the field of data mining is also of great significance. In the past, data processing needed to store the data in the database and then extract it, which had a huge impact on the timeliness of the processing results. Therefore, using a stream processing platform, data collection, transmission, and processing are carried out in real time and processing results are generated, and abnormal objects can be detected efficiently and reliably f...

AI Technical Summary

Problems solved by technology

[0003] In the field of anomaly detection, the LOF algorithm was applied to stream data in the past, but this has extremely high requirements for computing resources and memory resources. As the data stream continues to accumulate without data expiration, it eventually leads to system crashes.

Although the improved LOF algorithm alleviates the consumption of memory resources, it still requires high calculation time and reduces the detection hit rate

Images