Distributed density peak value clustering algorithm based on z value

Abstract

A distributed density peak value clustering algorithm based on a z value is disclosed. The algorithm comprises the following steps of (1) preparing a data set; (2) constructing software and hardware environments; (3) preprocessing data; (4) sampling the data set; (5) determining a cut-off distance parameter value of a density value calculating formula in a density clustering algorithm based on the z value and selecting a subgroup quantile; (6) according to the size of the z value, sending points in the data set to different groups; (7) calculating a density value in the distributed density peak value clustering algorithm based on the z value; (8) calculating a global outlier in the distributed density peak value clustering algorithm based on the z value; and (9) under a Hadoop environment, using the density peak value clustering algorithm based on the z value to carry out large data clustering. A z value characteristic is used, a filtering strategy is adopted during data interaction among subgroups, a lot of ineffective distance calculating and data transmission cost are reduced, and execution efficiency of the algorithm is effectively increased.

Description

technical field [0001] The invention relates to the field of big data processing, relates to a distributed density clustering algorithm, in particular to a distributed density peak clustering algorithm based on z value. Background technique [0002] Cluster analysis is one of the widely studied problems in the fields of data mining and pattern recognition. The density peak clustering algorithm (Density Peaks Clustering, DPC) published in the academic journal "Science" is a typical density-based clustering algorithm. The algorithm clusters the data set according to the property that each cluster has a density maximum point. The algorithm can find clusters of any shape and does not depend on the dimension of the data set; the implementation of the algorithm only needs to calculate two attribute values ​​of each point: (1) the density value ρ (by a certain range (2) Repulsion group value δ (characterized by the minimum value of the distance from a point whose density value is...

AI Technical Summary

Problems solved by technology

In order to improve the efficiency of the algorithm, the paper "EDDPC: An Efficient Distributed Density Central Clustering Algorithm" uses Voronoi segmentation technology to divide the data set into disjoint groups, and then send them to different machines for execution, but the grouping method is insufficient The reason is that the randomness of the seed object is very large, which may cause unbalanced load in the computer cluster

Secondly, when calculating the density value ρ and the repelling group value δ, there are still a large number of redundant copies, which increases the computational overhead

Images