A large-scale data distributed clustering processing method based on mapreduce

Abstract

The invention provides a large-scale data distributed clustering processing method based on MapReduce, which includes sampling large-scale data on the principle of non-repetition of equal scales; inputting sampled data into the MapReduce distributed parallel framework and calculating the local density and sum of the sampled data. Average density; find all sampled data whose local density is greater than the average density as a set of candidate points for the initial cluster center point of each cluster and feed it back to the main node, and select the distance between every two adjacent candidate points to be greater than 2 times the setting All candidate points in the range are used as the initial clustering center points; the MapReduce distributed parallel framework is used to perform parallel clustering tasks, and the average distance between data is calculated for each cluster to update the clustering center point; the error sum of squares criterion function is applied to the child nodes Determine whether to continue iteration; each sub-node clusters large-scale data based on the cluster center point. The invention realizes parallel clustering, reduces the number of clustering iterations, and improves clustering accuracy and parallel clustering efficiency.

Description

technical field [0001] The invention belongs to the technical field of parallel clustering, in particular to a large-scale data distributed clustering processing method based on MapReduce. Background technique [0002] With the rapid development of information technology, the scale of data continues to increase, and the use of parallel mechanisms to effectively mine and analyze large-scale data sets can promote the development and progress of Internet technology. Cluster analysis is an important data processing technology, and one of the important topics in the field of machine learning and artificial intelligence. It is widely used in data mining, information retrieval and other research. The main task is to divide the data set into multiple subsets, so that the similarity between the data objects in the subset is high, and the difference between the data objects in different subsets is relatively large. Due to the increase of data scale, the traditional stand-alone cluste...

AI Technical Summary

Problems solved by technology

The K-Means method combines the Canopy method Canopy-Kmeans, uses the characteristics of Canopy to calculate the similarity of objects, and preprocesses the data. The advantage is that the initial clustering center point can be given to avoid falling into local optimum, but the disadvantage is that the distance between objects can be calculated. The time consumption of the similarity is relatively large

The method based on data density calculation is to calculate the density of all data, and then select the data with the highest density as the cluster center point to avoid the problem of random selection, and it is more accurate, but the traditional calculation cost is also large, and it is easy to cause node clustering. The load is heavy, reducing the overall efficiency of parallel clustering

Images