Text deduplication method and device and electronic equipment

Abstract

The embodiment of the invention discloses a text duplicate removal method and device and electronic equipment. A specific embodiment of the method comprises the steps of obtaining a to-be-deduplicated webpage set; for each webpage to be subjected to duplicate removal in the webpage set to be subjected to duplicate removal, extracting webpage features from webpage data of the webpage to be subjected to duplicate removal, and performing duplicate removal on the webpage to be subjected to duplicate removal on the basis of a vector space hash algorithm and a minimum common substring matching algorithm by utilizing a webpage title and a webpage body of the webpage to be subjected to duplicate removal; determining whether a webpage similar to the to-be-deduplicated webpage exists in a candidate webpage set, and if yes, setting a similar flag bit of the to-be-deduplicated webpage; grouping the to-be-deduplicated web pages in the to-be-deduplicated web page set by utilizing the similar flag bits; and based on the webpage features, selecting a target webpage from each group of to-be-deduplicated webpages, and deleting other webpages except the target webpage to obtain a deduplicated webpage set. According to the embodiment, while the webpage text deduplication effect is improved, the deduplication efficiency is improved, and the memory is saved.

Description

technical field [0001] The embodiments of the present disclosure relate to the field of computer technologies, and in particular to a text deduplication method, device and electronic equipment. Background technique [0002] There are usually a large number of mutual reprints in some Internet websites. For example, local government websites reprint news from central government affairs websites, which will lead to repeated retrieval of captured data in search engines, resulting in a bad user experience. [0003] Existing text deduplication methods are based on the vector space hash (sim-hash) algorithm, or the least common substring matching (Jaccard) algorithm. However, the plagiarism check based on the Jaccard algorithm involves a lot of calculations, the speed is relatively slow, and it requires computer memory. If the text of the webpage that needs to be checked is too long, it will cause memory overflow. The duplication check based on the sim-hash algorithm is more suita...

AI Technical Summary

Problems solved by technology

However, the duplicate checking based on the Jaccard algorithm involves a lot of calculations, the speed is relatively slow, and it requires computer memory. If the text of the web page that needs to be checked is too long, it will cause memory overflow

The duplication check based on the sim-hash algorithm is more suitable for the similarity comparison of long texts, and the deduplication effect of short texts will be inaccurate

Images