Method for filtering same or similar files

Abstract

The invention discloses a method for filtering the same or similar files. The method comprises the following steps of: storing a plurality of files to be filtered into a pat tree data structure file (PT file) in a pat tree data structure; searching all the character string nodes of which continuous character length reaches a low threshold and all the files to which the character string nodes belong in the PT file; searching files in which continuous character contents are same, and the same continuous character length reaches a high threshold in the files; searching all the character string nodes of which continuous character length reaches a low threshold and all the files to which the character string nodes belong in the PT file; searching files in which continuous character contents are same, and the ratio of the same continuous character length to the total length of the prior file content reaches a ratio threshold; and marking the files as files having the same content or high degree of approximation. The method can filter the files having the same content or high degree of approximation and solves the problem repeating a great number of same or similar files.

Description

technical field [0001] The invention relates to a file filtering method, in particular to a method for filtering the same or similar files among multiple files by using a computer and performing cluster classification. Background technique [0002] With the rapid development of computers and the Internet, the amount of information to be processed has increased rapidly. Users often use computers to search for required data or information in huge file files or on the Internet; the search engine (searching engine) in the Internet is A tool that helps web users quickly search for data in the vast Internet. [0003] Generally speaking, search engines will present all search results that match the search keywords to users, even web pages with the same content will be presented without any filtering, although a small number of search engines will search The results are filtered, but web pages with high approximation are still easy to appear again and again. For users, time will be...

AI Technical Summary

Problems solved by technology

[0003] Generally speaking, search engines will present all search results that match the search keywords to users, even web pages with the same content will be presented without any filtering, although a small number of search engines will search The results are filtered, but web pages with high approximation are still easy to appear again and again. For users, time will be wasted in reading repeated content, and the convenience of data search will be reduced.

Images