Large-scale data abnormity detection method

Abstract

The invention relates to a large-scale data abnormity detection method comprising the steps that A. data preprocessing and feature extraction are performed; B. hyperplane calculation based on twin support vector machines is performed, and a hyperplane standard function of partition data space is constructed; C. an isolation tree is formed: the isolation tree is established through the partition criterion of the hyperplane of the twin support vector machines; D. an isolation forest is formed: the step C is repeated, and multiple isolation trees are constructed so as to form the isolation forest; and E. the isolation forest is traversed and the abnormity score is calculated: the isolation forest is traversed through the data under abnormity detection and the abnormity score is calculated to act as the standard for judging the degree of abnormity score, and existence of the abnormal data in the original data is judged according to the standard. The detection data volume can be effectively reduced so that the calculation workload can be reduced, the abnormity detection accuracy can be enhanced without significant increasing of time consumption and the abnormity detection performance for the high dimensional data can be greatly enhanced.

Description

technical field [0001] The invention relates to a data mining method, in particular to a large-scale data anomaly detection method. Background technique [0002] Anomaly detection refers to discovering and looking for data objects that are obviously different from most other data through corresponding technical means. Generally speaking, these data are very small compared to normal data. The object of anomaly detection is called anomaly point, or isolated point, outlier point. Although these data are often hidden among normal data and cannot be found directly, there may be important information hidden behind these data, which has great research value. In 1980, Hawkins first defined an outlier as a value that is significantly different from other values, making people question whether it is produced by a different and unknown mechanism. From then on, outliers are no longer noises in the field of data mining, nor data that need to be discarded in the preprocessing stage. W...

AI Technical Summary

Problems solved by technology

However, since the SCIFOREST algorithm only considers and tests the experimental data, in actual work, the detection performance of the algorithm is not good in the face of unbalanced, mixed, and high-dimensional large-scale data environments, and it is easily affected by complex data.

Images