Multi-partition clustering preprocessing method of stream data

Abstract

The invention relates to a multi-partition clustering preprocessing method of stream data. The method includes the following steps that: the trend factors of the stream data are screened, and a correlation degree is calculated; statistical analysis is performed on the stream data, a low-dimensional database partitioning method is adopted for high-correlation factors in a high-dimensional database, and isometric triangle partitioning is performed on low-correlation factors; a distribution-based partitioning method is adopted for a low-dimensional database; a DBSCAN (density-based spatial clustering of applications with noise) algorithm is adopted to perform clustering in each rule partition; and local clustering results are merged. According to the multi-partition clustering preprocessing method of the stream data of the invention, multi-partition improvement is made for the clustering preprocessing of the stream data, and therefore, data distribution is more uniform, clustering results are more accurate, and distributed parallel processing of the data is realized, and the pressure of low data preprocessing efficiency of a large number of data sequences which arrive sequentially, fast and continuously can be alleviated.

Description

technical field [0001] The invention relates to the technical field of DBSCAN clustering algorithm, in particular to a multi-partition clustering preprocessing method for streaming data. Background technique [0002] With the advent of the era of big data, data has gradually been transmitted in the form of data streams. Streaming data has the following four characteristics: 1) The data arrives in real time; 2) The order of data arrival is independent and not controlled by the application system; 3) The data scale is huge and its maximum value cannot be predicted; 4) Once the data is processed, unless it is specially saved, Otherwise, it cannot be retrieved and processed again, or it is expensive to retrieve the data again. The above characteristics of streaming data also bring about the expansion of security audit data. Data mining itself is a common knowledge discovery technology, its purpose is to extract the data information (knowledge) we are interested in from massive...

AI Technical Summary

Problems solved by technology

In the general-purpose density-based clustering algorithm DBSCAN algorithm, there are two weaknesses: one is that the entire database needs to be loaded into the memory during the data clustering process, and when the amount of data is large, the efficiency of the algorithm will drop sharply

Second, when the density of spatial clustering is not uniform, the distance between clusters varies greatly, and the clustering quality is poor.

In the improved grid-based and density-based clustering algorithms, there is still a problem that the processing data volume range is huge, resulting in low clustering efficiency.

Images