A real-time network abnormal behavior detection system and method based on big data

Abstract

The invention provides a real-time network abnormal behavior detecting system based on big data. The real-time network abnormal behavior detecting system comprises a flow collecting layer, a data pipeline layer, a real-time calculation layer, a data storage layer, a data analysis layer and an application layer, wherein the flow collecting layer comprises a collecting device; the data pipeline layer comprises a data pipeline service module adopting a distributed message system; the real-time calculation layer comprises a stream-oriented computation module; the data storage layer comprises a distributed file service module, a distributed database module and an retrieval service module; the data analysis layer comprises a model training module and a real-time detection module; the applicationlayer comprises a visual warning module. The invention also discloses a real-time network abnormal behavior detecting method based on big data. The data collection efficiency is high; the data transmission is stable and reliable; the advanced persistent threat can be efficiently detected and analyzed; the traceability evidence can be realized; the retrieval by analysts is convenient; the model training efficiency is high; the false alarm rate is low.

Description

technical field [0001] The invention belongs to the technical field of network security management, and in particular relates to a real-time network abnormal behavior detection system and method based on big data. Background technique [0002] With the continuous development of network applications, how to find abnormal behaviors from network data packets and give early warnings has become an important research field of current network security management. Most of the existing network traffic anomaly detection platforms rely on collection methods such as SNMP or Netflow. SNMP relies more on the performance of routers, and the collection efficiency is relatively low. However, Netflow only provides flow statistics and does not have information on the data packets themselves. The collected data cannot provide data for the subsequent analysis of the full flow. Secondly, in the process of streaming computing, the existing big data-based network traffic anomaly detection platform...

AI Technical Summary

Problems solved by technology

Most of the existing network traffic anomaly detection platforms rely on collection methods such as SNMP or Netflow. SNMP relies more on the performance of routers, and the collection efficiency is relatively low. However, Netflow only provides flow statistics and does not have information on the data packets themselves. After the collection, the data cannot be provided for the subsequent analysis of the full flow

Secondly, in the process of streaming computing, the existing big data-based network traffic anomaly detection platform uses a streaming computing engine such as spark streaming, which performs small batch processing based on a collection of data slices (RDD) , with suboptimal performance in terms of streaming

In addition, most of the existing network traffic anomaly detection platforms based on big data are based on netflow or ipfix technology, which perform feature matching on network traffic or based on simple statistics, and it is difficult to detect attacks such as advanced persistent threats (APT)

Images