Massive Network Data Search Method Based on Data Flow Structure

Abstract

The invention discloses a mass network data search method based on a data stream structure. The method comprises a real-time data storage method and a real-time query method. Data in a network arrives in a data stream form, so that it is needed to reduce the network data, capture a specific length of each piece of network message data and store valid information according to a specific data file size. According to the method, a Bloom Filter algorithm is used to establish an index file for a source IP and a host IP in each network message; and during real-time query, the index file is searched for first, a data storage file is searched for after matching succeeds, and detailed information is obtained. Through the method, mass network data can be quickly processed, a small storage space is occupied, and a large amount of network detailed information is saved; by use of the Bloom Filter Hash algorithm to establish indexes for the network message data, grading retrieval of file content is realized, the index structure is simple, and the occupied space is small; by use of three different Hash functions for calculation, the Hash conflict rate is low; and during retrieval, index matching search is quick and accurate, and high time and space efficiency is achieved.

Description

technical field [0001] The invention relates to the field of massive data processing, in particular to a search method for massive network data. Background technique [0002] Data search refers to a technique of extracting required data from a computer file or database according to search requirements. At present, common data search methods for file data include: linear search, binary search, skip search, etc.; linear search is to compare a given keyword value with the records in the file one by one until a matching record is found. . This method is simple and easy to implement, but it is inefficient when querying massive data, and it is difficult to meet the demand. Binary search is to arrange the records in the file according to the size of the key value, use the divide and conquer method to divide the file into two, and compare the given key value with the record at the midpoint. If they match, the search is successful; Otherwise, judge whether the record you want to f...

AI Technical Summary

Problems solved by technology

At present, the query of massive data is generally based on a large database, and the query of massive data is performed through distributed computing technology, table partitioning technology and query decomposition technology. This method is mainly for structured data; for unstructured data, the general Based on the index technology, the data is hashed to establish an index, and the search is performed according to the index to reduce the search workload and the search speed is fast. The current mainstream hash algorithms include MD5 algorithm, Bit-Map algorithm, Bloom Filter algorithm, etc., MD5 algorithm After the data is calculated, a 128-bit information summary is obtained, and the original data is compressed to reduce the storage space of the index structure, but the information summary still occupies a large storage space; the Bit-Map algorithm maps the data to the BitSet Each piece of data corresponds to one bit, and the index structure space is very small, but hash table conflicts are prone to occur, and the probability of error is high for queries; the Bloom Filter algorithm uses an m-bit BitSet, The data is hashed with k hash functions, and the value range of each hash function is 0 to m-1. Each hash function maps the data to a certain bit of the BitSet, so that the k bits of the BitSet Corresponding to one data, not only makes the storage space of the index structure small, but also reduces the conflict rate, and the query efficiency is high

Images