Chinese web page repeated document detection and filtration method based on full stop characteristic word string

Abstract

The invention discloses a Chinese web page repeated document detection and filtration method based on full stop characteristic word string, which comprises the following steps: extracting the full stop characteristic word string of the web page to be detected; using the full stop characteristic word string to filtrate template information for the web page to be detected to retain and extract text contents of the web page subjects; calculating similarity of the text contents of the web page subjects, and judging repetition relationship and inclusion relationship between the web pages; and clustering the web pages having the repetition relationship and inclusion relationship. According to the invention, aiming at Chinese web page, particularly Chinese news web page, firstly effective detection characteristics are searched, which can effectively detect the effective text portions on the web pages and filtrate noise which is irrelevant with the text content of the subject, such as advertisement on the webpage; and on the basis, the problems of similarity measurement and document repetition detection are solved, and finally the parallel processing problem during large-scale repeated document detection is solved.

Description

technical field [0001] The invention relates to a document detection method, in particular to a method for detecting and filtering duplicate documents in Chinese webpages based on period feature strings. Background technique [0002] There are a large number of nearly repeated webpages in the Internet (according to statistics, the repetition rate of Chinese webpages reaches 29%), which brings many problems to search engines and greatly increases the overhead and burden of webpage crawling, index establishment, and space storage. And significantly affect the experience of search engine users, reducing user satisfaction. [0003] It is relatively easy to detect two identical web pages, but in actual situations, there are almost no identical web pages. The first major difficulty in duplicate webpage detection is that many websites, especially news websites, will reproduce the same report or article. Therefore, the subject content in these webpages is exactly the same, but some...

AI Technical Summary

Problems solved by technology

The second difficulty in duplicate web page detection is that, since it involves comparison among millions of documents, and because each document has a relatively large length, it is necessary to complete the interaction between a large number of documents within a certain time limit. Comparison is a very time-consuming calculation process. If the calculation time is too long, it cannot meet the actual needs of search engines to regularly crawl and update search pages as soon as possible.

[0004] Existing duplicate web page detection methods are difficult to achieve ideal processing results in both detection accuracy and computing performance

The Shingling detection method has a fast processing speed and poor detection accuracy; the Random Projection method also has great advantages in performance, but it has not improved much in accuracy; the Imatch method improves the detection accuracy by strengthening the semantic information of the document, but its The amount of calculation is too large, and the processing speed is difficult to meet the needs of real-time document detection and processing of search engines; SpotSigs uses stop words in English as a feature to distinguish the subject content of web pages from irrelevant content, but Chinese lacks the stop word feature similar to English

[0005] In addition, although the methods proposed above can be used for reference to solve the duplicate detection problem of Chinese webpages to a certain extent, due to the significant differences in grammar and semantics between Chinese and English, the Chinese processing methods are quite different from the English processing methods. are different, especially in the processing of news webpages, therefore, none of these existing methods can be applied to duplicate document detection for Chinese webpages

Images