Similar web page duplicate-removing system based on parallel programming mode

Abstract

The invention provides a similar web page duplicate-removing system based on a parallel programming mode, comprises a web page content pre-processing module, a web page eigenvector extracting module, a web page feature fingerprint calculation module, a web page fingerprint on-line duplicate-removing module, a web page fingerprint distributed batch duplicate-removing module and a computing platform based on specific distribution. The system can complete links of carrying out unified conversion of text content encoding, standardization of document structure, web page noise content abortion, thematic content analysis and identification of web pages, lexical segmentation of continuous text content, and the like on the web pages obtained by crawling of web crawlers, thereby forming eigenvectors which can present the web pages. Relative algorithms can be used to obtain web page fingerprints which present web page characteristics aiming at the vector. The system provided by the invention accurately and fast detects fully complete repetition or approximate repetition of the web page contents caused by site mirroring, web document transshipment, and the like on the condition of massive amount of data of Internet and completes corresponding repetition-removing works, thereby enhancing the storage efficiency of search engines and bringing better use experience for the search engines.

Description

technical field [0001] The invention belongs to computer Internet information retrieval and analysis technology, and in particular relates to a similar web page deduplication system based on a parallel programming mode. The analysis technology extracts the feature vector of web pages, uses the parallel mode-based web page fingerprint deduplication algorithm, completes the deduplication function of similar web pages in a distributed system environment, and improves the efficiency of the search engine index module and retrieval module. Background technique [0002] With the unprecedented development of Internet technology and scale in recent years, more and more traditional resources are migrating to the Internet. Search engines have become the main tool for information retrieval for today's users because of their powerful and convenient retrieval functions. However, due to the large scale of the Internet and geographical access restrictions, many websites use server mirroring...

AI Technical Summary

Problems solved by technology

This method also has shortcomings. For example, different webpages on the same topic may have a large overlap rate in their keyword sets, but the content of the webpages is not repeated, and a large number of misjudgments may occur in the judgment of such webpages. , that is, non-similar pages are considered similar, so the accuracy is not high

Images