Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

High-dimensional data exception detection system and method

A high-dimensional data and anomaly detection technology, which is applied in the field of high-dimensional data anomaly detection system, can solve problems such as slowing down, increasing data analysis time, and insufficient time to achieve absolute security, accurate anomaly detection, and high accuracy.

Active Publication Date: 2019-06-21
SHANGHAI ADVANCED RES INST CHINESE ACADEMY OF SCI +1
View PDF2 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Statistical methods in mathematics can only show the overall laws of the data, but cannot effectively analyze the contextual connections in the data and the relationship between different dimensions of the data
Moreover, in the case of a sharp increase in the amount of data, the analysis speed of the statistical method will slow down sharply, and there is no application value in the case of analyzing a large amount of data generated by the car
[0005] For the LSTM network, it is very good at analyzing time series data
However, the speed of the LSTM network will be significantly slowed down when analyzing high-dimensional data, and the data generated by the car is unsupervised data. If the unsupervised data generated by the car is converted into supervised data, the analysis time of the LSTM network for the data is still short. will greatly increase
This is obviously not enough for the abnormal detection of the car state when we are doing driverless driving

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • High-dimensional data exception detection system and method
  • High-dimensional data exception detection system and method
  • High-dimensional data exception detection system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0050] see figure 1 , the present invention provides a high-dimensional data anomaly detection method, the original high-dimensional data anomaly detection method includes the following steps:

[0051] 1) Preprocessing the original high-dimensional data to remove the interference value in the original high-dimensional data, and filling the data after removing the interference value;

[0052] 2) Normalize the filled data;

[0053] 3) Dimensionality reduction is performed on the normalized data;

[0054] 4) Reshape the dimensionally reduced data to obtain supervised data;

[0055] 5) Using LSTM network to analyze the supervised data to obtain prediction data;

[0056] 6) Comparing the predicted data with real data to determine whether the original high-dimensional data is abnormal.

[0057] In step 1), see figure 1 In the S1 step, the original high-dimensional data is preprocessed to remove the interference value in the original high-dimensional data, and the data after the...

Embodiment 2

[0089] see figure 2 , the present invention also provides a high-dimensional data anomaly detection system, the high-dimensional data anomaly detection system includes: a preprocessing module 1, the preprocessing module 1 is used to preprocess the original high-dimensional data, to remove the The interference value in the original high-dimensional data, and the data after removing the interference value is filled; the normalization processing module 2, the normalization processing module 2 is connected with the preprocessing module 1, and the normalization processing module 2 is connected with the preprocessing module 1, and the A normalization processing module 2 is used for normalizing the filled data; a dimension reduction processing module 3, the dimension reduction processing module 3 is connected with the normalization processing module 2, and the dimension reduction processing module 3 For reducing the dimensionality of the normalized data; shaping processing module 4,...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a high-dimensional data exception detection system and method, and the method comprises the following steps: carrying out the preprocessing of original high-dimensional data, soas to remove an interference value in the original high-dimensional data, and carrying out the filling of the data after the interference value is removed; performing normalization processing on thefilled data; carrying out dimension reduction on the normalized data; shaping the data after dimension reduction to obtain supervised data; analyzing the supervised data by using an LSTM network to obtain prediction data; and comparing the prediction data with the real data to judge whether the original high-dimensional data is abnormal or not. According to the high-dimensional data exception detection method, rapid and accurate exception detection can be carried out on the high-dimensional data, and when exceptions occur in equipment such as an automobile, the exceptions can be processed immediately, so that absolute safety of automobile driving is ensured.

Description

technical field [0001] The invention belongs to the technical field of big data processing, and in particular relates to a high-dimensional data anomaly detection system and method. Background technique [0002] In modern society, more and more electric vehicles have entered our lives. In the future, unmanned electric vehicles will definitely become the mainstream of automobiles in society. Therefore, how to understand the running status of the car has become a problem that we must care about. In recent years, due to the imperfection of the automatic driving algorithm, many unmanned vehicles have crashed and malfunctioned. Therefore, this objectively requires us to have the ability to use the data on the car body to detect abnormalities in the car. [0003] The data generated by cars has the characteristics of high dimensionality and large quantity. For this, whether it is a mathematical statistical method or a simple LSTM network (long short-term memory network) in deep...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06N3/04G06N3/08
Inventor 汪辉吴迪祝永新田犁黄尊恺
Owner SHANGHAI ADVANCED RES INST CHINESE ACADEMY OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products