Method and device for abnormity detection of sparse data

Abstract

The invention discloses a method and device for abnormity detection of sparse data. The method comprises the steps that characteristic processing is conducted to original data of different types, so the original data of the different types is converted into the sparse data with a uniform format; a factorization machine is used for modeling of the sparse data, so a nonlinear manifold model is obtained; according to the nonlinear manifold model, abnormal value scores of data objects are computed; and according to abnormal value scores of the data objects, whether the data objects are abnormal data is judged.

Description

technical field [0001] The invention relates to the technical field of anomaly detection, in particular to a sparse data anomaly detection method and device based on a factorization machine. Background technique [0002] Anomaly detection is mainly based on Kawkins' definition of anomalies: anomalies are data that are far away from other observed data and thus suspected to be generated by different mechanisms. Efficient and accurate detection of anomalies is of great significance to the fields of intrusion detection, fraud detection and fault detection. Anomaly detection has been extensively studied, most of the methods are for traditional non-sparse data. However, in actual scenarios, many data are sparse: (1) Short text data: In recent years, with the development of social media, the analysis and mining of short text data has received more and more attention. limited, it becomes extremely challenging to infer anomalous properties of text. (2) Categorical data with "big ...

AI Technical Summary

Problems solved by technology

First, most of the values ​​in sparse data are 0, and there are only limited non-zero elements in a single data object. It is very challenging to infer the abnormal characteristics of data objects simply using limited non-zero elements, and it is necessary to combine the implicit relationship between attribute values

Many traditional methods, such as distance-based methods and pattern-based methods, cannot obtain the implicit relationship between attribute values.

Second, datasets in reality often contain multiple data types, such as classified data, numerical data, text data, etc., and existing methods generally only detect abnormalities for a certain type of data, which is not universal

Third, the time complexity of many traditional anomaly detection methods (such as distance-based methods) is too high to be applicable to large-scale data

Images