Method of tcp session reorganization and statistical data extraction based on stream processing

Abstract

A method for extracting TCP session reorganization and statistical data based on stream processing, comprising the following steps: constructing a data pipeline layer and a real-time computing layer between the data collection layer and the data storage layer, and the data collection layer collects network data packets and sends them to The data pipeline layer is used for caching, and the real-time computing layer is used to extract data from the data pipeline layer for processing, and the processing results are stored in the data storage layer; the real-time computing layer extracts data from the data pipeline and deserializes it into an object; the object will be used as a stream Data elements in the formula calculation process, and output TCP session data and statistical data through data operations. The present invention can effectively reorganize the TCP session of the network, and dig out session statistical data according to the extracted session data, provide support for traffic information mining and abnormal behavior analysis, provide efficient and reliable session data and session statistical data calculation services, and guarantee the system efficiency and stability.

Description

technical field [0001] The invention relates to a stream processing TCP session recombination and statistical data extraction method. Background technique [0002] Current session data extraction tools often rely on open source tools such as libnids, netflow, etc., and rely on the computing resources and storage resources of sensors, which can easily reach computing bottlenecks, lack reliability, redundancy and other mechanisms, and have low fault tolerance, hindering the overall performance and reliability of the system. sex, ultimately leading to more waste of physical and human resources. Tools such as Netflow rely on special network equipment and are not universal. Moreover, the lack of flexibility in tools such as libnids is difficult to meet the needs of customized output, which brings more labor costs for subsequent data processing. In the actual production environment, the transmission of data streams often uses the original text to flow in the data pipeline. The pr...

AI Technical Summary

Problems solved by technology

[0002] Current session data extraction tools often rely on open source tools such as libnids, netflow, etc., and rely on the computing resources and storage resources of sensors, which can easily reach computing bottlenecks, lack reliability, redundancy and other mechanisms, and have low fault tolerance, hindering the overall performance and reliability of the system. sex, ultimately leading to more waste of physical and human resources

Tools such as Netflow rely on special network devices and are not universal, and the lack of flexibility in tools such as libnids is difficult to meet the needs of customized output, which brings more labor costs for subsequent data processing

In the actual production environment, the transmission of data streams often uses the original text to flow in the data pipeline. The processing flexibility is low, resulting in a waste of network resources, and it is easy to reach the network bottleneck, which in turn leads to the stagnation of the entire system.

In other statistical data extraction methods, offline calculations are often used. This calculation method will cause high data delay and cannot deliver data in the first time, thus slowing down the speed of the entire system.

Images