Anomaly point detection method based on data structure

Abstract

The invention relates to an anomaly point detection method based on a data structure. The anomaly point detection method comprises the steps that a data set is input; a multidimensional binary tree isconstructed according to the data set, and k neighbors closest to each node in the tree are searched through a binary tree search algorithm; the Euclidean distances between the data points are calculated based on a data structure diagram of constructed data points of the multidimensional binary tree and by combining the neighbor relations of all the nodes in the tree; and in consideration of thesimilarity between the data points and the neighbor relationship of the data points in the tree, anomaly points are automatically determined by sorting the calculated Euclidean distances and setting the threshold value p. According to the anomaly point detection method, the performance of anomaly point detection is improved, and the structural characteristics of the data set is better reflected; in addition, the anomaly point detection method is weakly affected by data distribution and data dimension and is wider in application range during practical application, and the problems that in the prior art, the detection accuracy of special points and the detection performance of high dimensional data are poor are solved.

Description

technical field [0001] The invention belongs to the field of data detection, in particular to a data structure-based abnormal point detection method. Background technique [0002] Outlier detection is the most important task in the process of identifying outliers. Due to the unbalanced distribution of outliers and other reasons, traditional outlier detection methods will lead to inaccurate or even wrong recognition results. Outlier detection technology can effectively improve the performance of outlier detection. Traditional outlier detection technologies are mainly clustering, classification, and pattern recognition. These traditional technologies are to find a common pattern to identify meaningful patterns in data, while outlier detection technology only needs to identify outliers, not Normal points are identified. For example, in a system for detecting life disorders, normal people are regarded as normal points, and patients with disordered vital signs or patients with ...

AI Technical Summary

Problems solved by technology

There are following deficiencies in this method: 1, need to determine threshold value (d in this detection method) min ), and the determination of this value is more difficult, different data sets d min There is a large gap, for a given different d min The obtained anomaly detection results are usually very unstable

However, simply calculating the Euclidean distance cannot effectively and accurately identify all abnormal points, and the calculation process is cumbersome, time-consuming, and computationally complex

According to whether the detection method needs a label, the detection method can be divided into supervised, semi-supervised and unsupervised. Among them, the unsupervised method is the most challenging method because of the lack of effective label training, and coupled with The imbalance of data distribution exacerbates the inaccuracy of detection results

[0007] To sum up, the existing outlier detection algorithms work better under certain conditions or in specific fields, or have a better effect on outlier detection in lower-dimensional spaces. When the dimensionality of the data is higher, the effect of these algorithms is not ideal. The generalization ability of the algorithm is weak, and for high-dimensional spatial data, the computational complexity is high

Images