Large-scale image data similarity searching method based on EMD (earth mover's distance)

Abstract

The invention discloses a large-scale image data similarity searching method based on an EMD (earth mover's distance). The method comprises the following steps that an image data mapping function f used for mapping to a one-dimension real number key value space Omega(phi) is designed; an operation MR1 is started, and a load of each key value in the Omega(phi) is estimated; the operation MR2 is started, the cutting is carried out on the Omega(phi) through a Map task on the basis of the estimated key value load, and data corresponding to the cutting region are sent to a Reduce task in a segmented way; image data received by each Reduce task is mapped to the key values in the Omega(phi) on the basis of f, and an index structure oriented to the EMD is built on the basis of the key values; the similarity searching based on the EMD is executed on the basis of the index structure; execution results of each Reduce task based on EMD similarity searching in the MR2 are subjected to union set taking and output. The large-scale image data similarity searching method has the advantages that the network transmission data quantity is lower, the calculation load distribution is more balanced, the similarity searching efficiency is higher, and the big data set analysis and processing expandability is better.

Description

Technical field [0001] The invention relates to a similarity search technology for computer image data, in particular to a large-scale image data similarity search method based on EMD distance. Background technique [0002] With the popularization of digital devices such as portable computers, smart phones, and digital cameras, multimedia data represented by images has increased and exploded. All this indicates that the era of image big data has arrived. At present, academia, industry and even government agencies have begun to pay close attention to the analysis and processing of image big data. [0003] To find important value image information from large-scale image data sets, the traditional text-based image retrieval method (Text-Based Image Retrieval, referred to as TBIR) obviously cannot meet the demand. Because TBIR technology relies on manual annotation of image content, when the number of images increases sharply, it brings two serious problems: first, the workload of ma...

AI Technical Summary

Problems solved by technology

[0003] To find valuable image information from large-scale image datasets, the traditional text-based image retrieval method (Text-Based Image Retrieval, TBIR for short) obviously cannot meet the demand

Because TBIR technology relies on manual annotation of image content, when the number of images increases sharply, it brings two serious problems: first, the workload of manual annotation is too large, and the cost of annotation is too high; second, the subjective If it is too strong, it will directly affect the reliability of image retrieval results.

In view of the high computational complexity of the EMD distance, it is obviously biased to estimate the calculation cost of the computing node by the amount of data (rather than the actual number of EMD distance calculations), which is not conducive to balancing the computing load of each computing node and directly reduces the overall Query Processing Performance in Distributed Systems

On the other hand, when the size of the image data set surges, the filtering performance of the distributed index in Melody-Join for irrelevant calculations is still insufficient.

The above two aspects directly lead to the fact that the scalable performance of Melody-Join in processing large-scale image datasets cannot meet the needs of practical applications.

Images