Repeated text judgment method and apparatus

Abstract

The invention discloses a repeated text judgment method and apparatus. The method comprises the steps of judging whether summary information corresponding to a first text and a second text is repeated or not; if the summary information is not repeated, extracting feature contents of the first text and the second text; identifying whether the feature contents corresponding to the first text and the second text are repeated or not; and if the feature contents are repeated, judging that the first text and the second text are repeated. According to the method and the apparatus provided by embodiments of the invention, the judgment of repeated texts can be finished by using the method for texts with the same summary information; for texts with different summary information, the feature contents are extracted and the feature contents of the texts are judged, so that the judgment of the repeated texts is finished; and the calculation amount required for the judgment process of the repeated texts is relatively small, the judgment efficiency is relatively high, and an algorithm is flexible to use.

Description

technical field [0001] The invention relates to information processing technology, in particular to a method for judging repeated texts and a device for judging repeated texts. Background technique [0002] In the process of text processing, the text deduplication method is often used to remove repeated information in the text. At present, when determining whether two texts are duplicated, the similarity of the two texts is usually calculated first, and then the calculated similarity is judged. If the calculated similarity is less than the similarity threshold, the two texts are determined to be duplicated. [0003] Commonly used text similarity algorithms include cosine similarity algorithm and text hash algorithm. Among them, the cosine similarity algorithm converts text into vectors, calculates the cosine value of the angle between the vectors, and pre-smalls the similarity of the two texts The higher the text hash algorithm is, the text is mapped to the corresponding ha...

AI Technical Summary

Problems solved by technology

[0004] However, both of the above algorithms have drawbacks

Specifically, text conversion and vector angle calculation in the cosine similarity algorithm require a large amount of calculation, which makes the calculation efficiency of the cosine similarity algorithm low, so the judgment efficiency of using the cosine similarity algorithm to judge repeated text is low; the text The conditions for judging the similarity in the hash algorithm are very strict, and the algorithm lacks flexibility in use

Images