Network traffic measurement method and system, computer equipment, storage medium and application

Abstract

The invention belongs to the technical field of network flow measurement, discloses a network flow measurement method and system, computer equipment, a storage medium and application, and adopts a novel extensible reversible data compression fusion model capable of simultaneously realizing efficient and accurate network flow measurement of a memory. The model can dynamically and adaptively expand and compress the size of the memory occupied by the fusion model according to the distribution condition of network traffic information. The core idea of the invention is that counters with different sizes are utilized to monitor a low-cardinal-number host and a high-cardinal-number host respectively, that is, a small-sized counter is utilized to monitor the low-cardinal-number host, and a counter capable of adaptively expanding the size is utilized to monitor the high-cardinal-number host; and efficient memory utilization during low-cardinal-number host monitoring and high accuracy during high-cardinal-number host monitoring are ensured. Based on the model, the invention designs a rapid and efficient network flow measurement method to accurately analyze the cardinal number of the host, rapidly identify the super host and efficiently reconstruct the abnormal address.

Description

technical field [0001] The invention belongs to the technical field of network flow measurement, and in particular relates to a network flow measurement method, system, computer equipment, storage medium and application. Background technique [0002] At present, with the continuous improvement of network data forwarding speed, an online data processing model with fast data collection speed and high memory usage efficiency is urgently needed. The summary data structure belongs to the data structure based on probability and statistics, which has been widely used in many aspects of network traffic analysis, such as traffic size estimation, host cardinality estimation, and network anomaly detection. They use data-oriented hashing methods to compress security parameters in a compact way, and perform probability estimation on data through specific operations. Utilizing the advantages of the summary data structure in efficiently compressing fusion data and accurately estimating, i...

AI Technical Summary

Problems solved by technology

This characteristic of host cardinality makes it more difficult to monitor high-cardinality hosts and low-cardinality hosts separately than to monitor large and small network flows separately

[0007] (3) Summary data structure Most data compression fusion models cannot store any information about the original host address due to the use of hash technology to fuse data compression

[0008] The difficulty of solving the above problems and defects is: Although there are many methods to improve the memory efficiency of the data compression fusion model, there are still many difficulties in how to ensure the accuracy of network traffic measurement under the premise of memory efficiency

First, the unbalanced nature of network traffic information causes the number of low-cardinality hosts to dominate, while the number of high-cardinality hosts is only a small part

Most host cardinality analysis methods allocate counters of the same size to monitor both types of hosts, which is bound to cause a lot of waste of memory resources

Secondly, how to accurately trace the source of exceptions is also a difficult problem

Images