Chinese web page text deduplication system and method

Abstract

The invention discloses a Chinese web page text deduplication system and a Chinese web page text deduplication method. The deduplication system comprises an index server and a search server, wherein the index server comprises a web page text preprocessing module, a combined characteristic sentence extraction module and a digital signature calculation module; and the search server comprises a web page text capture module and a Hash query module. The deduplication method comprises the following steps of: normalizing a web page text; extracting a combined characteristic sentence of the text; calculating a digital signature of the combined characteristic sentence; and comparing the digital signature with the existing digital signature in a Hash table, and judging whether the digital signature is duplicated or not. By the deduplication system and the deduplication method, a search engine can quickly and accurately determine and remove a large number of Chinese web pages with duplicated contents in the Internet; and when the search engine captures a new web page, the digital signature of the web page is calculated and compared with the digital signature of the web page, which has been stored by the search engine, whether the web page is duplicated or not is judged, and the web page is not stored if the web page is duplicated, so that the waste of a storage space is avoided, and the search accuracy of the search engine is improved simultaneously.

Description

technical field [0001] The invention relates to a Chinese web page text deduplication system, and the invention also relates to a method for removing duplicate Chinese web pages by using the deduplication system. Background technique [0002] At present, the amount of information on the Internet is increasing at an explosive rate, and users must rely on search engines to find the information they want in the massive amount of information on the Internet. Full-text search engines, such as Google, Baidu, etc., are search engines in the true sense. They usually send out "spider" (spider) programs regularly to grab web pages on the Internet according to certain rules and save them in the local storage system. After inputting query keywords on the search interface of the search engine, the search engine searches the local web database for records matching the query conditions, and returns the search results to the user according to certain sorting rules. [0003] However, since ...

AI Technical Summary

Problems solved by technology

Since the search engine will download and save the web pages captured by the spider program to the local storage system, and the capacity of the storage system is limited, a large number of repeated web pages will waste the resources of the search engine and occupy the storage space of other valuable web pages. In addition, the retrieval efficiency of search engines will also be reduced due to the increase in the amount of data in the local webpage database, which not only wastes the user's time, but also affects the user's search experience

[0005] Deduplication of webpages, that is, removing duplicate webpages on the Internet, is an effective way to solve the above problems. However, due to the extremely large number of webpages collected by search engines, which are more than tens of millions of pages, the number of webpages collected by large search engines like Google There are even billions of web pages, and the existence forms of web pages are also very complex and diverse. If you directly compare a web page newly captured by the spider program with the massive web pages already included in the search engine system one by one, the calculation complexity will be very high. High, for example, assuming that the number of documents included in the search engine is n, and the average length of the documents is m, if the complexity of similarity calculation is T, T is a function of m, that is, T=T(m), and the comparison of documents is complicated degree is 0(n^2), then the combined complexity is 0(n^2×T(m)), such a complexity is obviously unacceptable for a system like a search engine that needs to process massive amounts of data

Images