Similar web page duplicate-removing system based on parallel programming mode

Abstract

The invention provides a similar web page duplicate-removing system based on a parallel programming mode, comprises a web page content pre-processing module, a web page eigenvector extracting module,a web page feature fingerprint calculation module, a web page fingerprint on-line duplicate-removing module, a web page fingerprint distributed batch duplicate-removing module and a computing platformbased on specific distribution. The system can complete links of carrying out unified conversion of text content encoding, standardization of document structure, web page noise content abortion, thematic content analysis and identification of web pages, lexical segmentation of continuous text content, and the like on the web pages obtained by crawling of web crawlers, thereby forming eigenvectorswhich can present the web pages. Relative algorithms can be used to obtain web page fingerprints which present web page characteristics aiming at the vector. The system provided by the invention accurately and fast detects fully complete repetition or approximate repetition of the web page contents caused by site mirroring, web document transshipment, and the like on the condition of massive amount of data of Internet and completes corresponding repetition-removing works, thereby enhancing the storage efficiency of search engines and bringing better use experience for the search engines.

Description

technical field [0001] The invention belongs to computer Internet information retrieval and analysis technology, and specifically relates to a similar webpage deduplication system based on parallel programming mode. The system is an improvement to the existing similar webpage deduplication system, combining the existing webpage structure and theme Analysis technology, extracting feature vectors of webpages, using webpage fingerprint deduplication algorithm based on parallel mode, completing deduplication function of similar webpages in distributed system environment, improving the efficiency of search engine indexing module and retrieval module. Background technique [0002] With the unprecedented development of Internet technology and scale in recent years, more and more traditional resources are migrating to the Internet. Search engines have become the main tool for information retrieval by users today because of their powerful and convenient retrieval functions. However, ...

AI Technical Summary

Problems solved by technology

This method also has shortcomings. For example, different webpages on the same topic may have a large overlap rate in their keyword sets, but the content of the webpages is not repeated, and a large number of misjudgments may occur in the judgment of such webpages. , that is, non-similar pages are considered similar, so the accuracy is not high

Images