Distributed outlier detection method and system based on automatic coding machine

Abstract

The invention relates to a distributed outlier detection method and system based on an automatic coding machine. The method includes the steps that a training data set and a testing data set are defined; training data of the training data set are distributed to a plurality of calculation units randomly; the calculation units conduct parallel execution, and each calculation unit solves coding and decoding parameters; the coding and decoding parameters of each calculation unit are summarized to obtain a final coding and decoding parameter, and a self-duplication model is built; the self-duplication model is applied to the testing data set, and concurrent computation is conducted on reconstruction errors of all testing data; the testing data are arranged according to a descending order of the reconstruction errors, and the testing data with the reconstruction errors larger than a predetermined threshold value are outliers. According to the method, the total time required for processing and the number of processed samples are independent, and the total time and the number only depend on the required accuracy of parameter solution. The distributed outlier detection method and system based on the automatic coding machine are very suitable for detecting outliers on large-scale data sets on the basis of MapReduce frameworks, and have good flexibility and good expansibility.

Description

technical field [0001] The invention relates to the field of information security, in particular to a distributed outlier detection method and system based on an automatic encoding machine. Background technique [0002] In data mining and statistics, outliers refer to observations or sample points that deviate significantly from other data. In many data mining applications, outliers are filtered and discarded. However, from the perspective of knowledge discovery, rare events are often more valuable than ordinary events, so outliers are a research field with high application value. For example, in fraud detection, network intrusion detection and other fields. At present, computer-based outlier detection methods at home and abroad can be divided into four categories: statistical distribution-based, depth-based, distance-based and density-based local outlier detection methods. [0003] Auto-encoders are a special type of neural network. An autoencoder consists of encoding an...

AI Technical Summary

Problems solved by technology

The outlier detection algorithm based on the autoencoder usually uses the stochastic gradient descent method to solve the encoding and decoding parameters, the solution speed is slow, the data global dependence is high, and it is difficult to handle large-scale data much larger than the memory.

Images