Approximate data processing method and device, medium and electronic equipment

Abstract

The invention relates to the field of data processing and discloses an approximate data processing method and device, a medium and electronic equipment. The method comprises the steps of obtaining to-be-processed data; obtaining a vector corresponding to the to-be-processed data; performing hash operation on the vector of the to-be-processed data by utilizing each position sensitive hash functionin a position sensitive hash function family to obtain a mapping value corresponding to the vector of the to-be-processed data; repeating the step of constructing coverage groups for a first predetermined number of times to obtain a plurality of coverage groups, wherein the step of constructing the coverage groups comprises the construction of the coverage groups based on mapping value corresponding to the vector of the to-be-processed data and a position sensitive hash function for performing hash operation on the vector of the to-be-processed data; integrating a plurality of coverage groupsto obtain final coverage to which the to-be-processed data belongs, and taking the to-be-processed data belonging to the same final coverage as approximate data. According to the method, the situationthat time consumed for processing a large amount of approximate data is not stable is avoided, and the data processing efficiency is improved on the whole.

Description

technical field [0001] The present disclosure relates to the technical field of data processing, and in particular to an approximate data processing method, device, medium and electronic equipment. Background technique [0002] At present, in data processing, for any piece of data, in order to quickly find data similar to the data, the commonly used scheme is location sensitive hashing (Locality sensitive Hashing, LSH), which maps high-dimensional data to low-dimensional Data, mapping similar data into the same bucket, can make the probability that two data points adjacent to each other in the original data space are still very high in the new data space after mapping, while two data points that are not adjacent The probability of points being adjacent in the new data space after mapping is very small. However, the use of the LSH algorithm involves the given of several hyperparameters. It includes the random numbers in the Hash function, so the effect of mapping different ...

AI Technical Summary

Problems solved by technology

It includes the random numbers in the Hash function, so the effect of mapping different heaps has a great relationship with the given random numbers. When using the mapping results based on the LSH algorithm for subsequent data processing tasks, if a large amount of data needs to be processed , it will have high requirements on the mapping results, which will cause certain instability

On the one hand, if the amount of data in the bucket is too large, the effect of using the LSH algorithm to improve efficiency will be greatly reduced; on the other hand, for the same set of data, the time it takes to execute data processing tasks will be uncertain, depending on the amount of data in the heap. size effect

Images