Extraction method of TCP session reassembling and statistic data based on streaming processing

Abstract

The present invention provides an extraction method of TCP session reassembling and statistic data based on streaming processing. The method comprises the following steps of: constructing a data pipeline layer and a real-time calculation layer between a data collection layer and a data storage layer, wherein the data collection layer collects a network data package and sends the network data package to the data pipeline layer for cache, the real-time calculation layer is configured to extract data from the data pipeline layer for processing, and a processing result is stored in the data storage layer; the real-time calculation layer extracts data from the data pipeline to perform deserialization to form an object; and the object is taken as a data element in a streaming calculation process, and TCP session data and statistic data are output through data operation. The extraction method of TCP session reassembling and statistic data based on streaming processing effectively reassemblesthe TCP session of the network, excavates session statistic data according to the extracted session data, provides support for flow information excavation and abnormal behavior analysis, provides efficient and reliable session data and session statistic data calculation service and guarantees efficiency and stability of the system.

Description

technical field [0001] The invention relates to a stream processing TCP session recombination and statistical data extraction method. Background technique [0002] Current session data extraction tools often rely on open source tools such as libnids, netflow, etc., and rely on the computing resources and storage resources of sensors, which can easily reach computing bottlenecks, lack reliability, redundancy and other mechanisms, and have low fault tolerance, hindering the overall performance and reliability of the system. sex, ultimately leading to more waste of physical and human resources. Tools such as Netflow rely on special network equipment and are not universal. Moreover, the lack of flexibility in tools such as libnids is difficult to meet the needs of customized output, which brings more labor costs for subsequent data processing. In the actual production environment, the transmission of data streams often uses the original text to flow in the data pipeline. The pr...

AI Technical Summary

Problems solved by technology

[0002] Current session data extraction tools often rely on open source tools such as libnids, netflow, etc., and rely on the computing resources and storage resources of sensors, which can easily reach computing bottlenecks, lack reliability, redundancy and other mechanisms, and have low fault tolerance, hindering the overall performance and reliability of the system. sex, ultimately leading to more waste of physical and human resources

Tools such as Netflow rely on special network devices and are not universal, and the lack of flexibility in tools such as libnids is difficult to meet the needs of customized output, which brings more labor costs for subsequent data processing

In the actual production environment, the transmission of data streams often uses the original text to flow in the data pipeline. The processing flexibility is low, resulting in a waste of network resources, and it is easy to reach the network bottleneck, which in turn leads to the stagnation of the entire system.

In other statistical data extraction methods, offline calculations are often used. This calculation method will cause high data delay and cannot deliver data in the first time, thus slowing down the speed of the entire system.

Images