Streaming data anomaly detection method and system based on HTM under sparse coding

Abstract

The invention provides a streaming data anomaly detection method and system based on HTM under sparse coding, and the method comprises the steps: obtaining a data source which comprises data of various domains and application programs generated by a user; inputting the acquired real-time data source into an encoder, and converting the data into a binary vector; performing sparse processing on theconverted data to obtain sparse distributed representative elements of the data; inputting the data into a standard HTM network, and obtaining a predicted value of the input data at the current momentaccording to the input data at the previous moment; comparing the actual input value of the current moment with the predicted value generated at the previous moment to obtain an abnormal score; calculating an abnormal likelihood value by using an HTM model and according to the distribution modeling of the abnormal scores; judging whether the abnormal likelihood value is greater than a preset abnormal threshold value or not, and determining whether to declare abnormality or not. According to the method, the problems that streaming transmission data intrinsically shows concept drift and needs to continuously learn for algorithm improvement are solved, and the security of the streaming application program is improved.

Description

Technical field [0001] The present invention mainly relates to the related technical fields of intrusion detection and data processing, in particular to an HTM-based stream data abnormality detection method under sparse coding. Background technique [0002] The anomaly detection of streaming data in the time series can be traced back to the consideration of the outliers of the two models and their influence in the time series. The use of techniques based on traditional analysis, machine learning and prediction-based anomaly detection models are effective means to combat existing and unknown attacks in the current network. The common practice of statistical-based anomaly detection technology is to treat the intrusion detection problem as a hypothesis testing problem. The key is to select a set of statistical metrics from the attributes describing the behavior and state of the system or network, and establish its normality based on historical data. The range of change is mainly re...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the deficiencies of the current technology, the present invention combines the existing technology and proceeds from practical applications to provide an HTM-based stream data anomaly detection method and system under sparse coding, aiming to solve various methods and traditional methods of existing machine learning. In the method of abnormal detection of real-time streaming data, it is difficult to achieve more accurate results and the training time is long due to the problems of strong real-time data, high data correlation, high repeatability, large dynamic environment changes, and serious concept drift.

Images