Field-oriented method and system for collecting invisible web resources

Abstract

The invention provides a field-oriented method for collecting invisible web resources. The method includes the steps of identifying and designating pages related to a field from grasped pages, obtaining valid form collection, judging the form type of every form in the valid form collection, constructing valid inquiries according to different form types, and outputting results returned for the valid inquiries to be as the collected invisible web resources, wherein the form types refers to single-initem forms or multi-initem forms. The method can achieve automatic indentifying and classifying of an invisible resource inquiry interface, and simultaneously achieves valid construction of inquiries for a simple inquiry interface and a complex inquiry interface. Accordingly, the collection for the invisible resources is achieved. The method not only can be integrated in an existing search engine in a seamless mode, but also can simultaneously collect the invisible resources directed by the simple inquiry interface and the complex inquiry interface.

Description

technical field [0001] The invention relates to information retrieval, in particular to a method for collecting dark web resources. Background technique [0002] With the rapid development of Internet technology, many different types of databases have appeared on the Internet. The information stored in them is huge in quantity and high in quality, forming a huge online information resource library. The information stored in the background database has standardized and unified storage, a good data structure, and high data quality, but most of these databases are hidden in the query interface—behind the form, users can only enter a series of keywords through the query interface The background database information can only be obtained after the query is submitted. However, the current web crawlers do not have the ability to automatically fill in the query interface, so this information cannot be directly obtained by the web crawler through the page hyperlink relationship, so t...

AI Technical Summary

Problems solved by technology

This method adopts domain-oriented thinking. Although it can achieve high resource coverage, it still has many defects for a search engine: (1) It does not consider the difference in website design in the same field, and it is difficult to solve the problem of simple query interface and Semantic mapping between intermediary forms, especially dark web resources hidden behind forms that cannot effectively capture single input items; (2) Maintaining intermediary forms and preparing input data for them is heavy work, resulting in poor scalability of this method

This method is more suitable for simple query interfaces, but it is difficult to collect resources for complex query interfaces. The reason is that complex query interfaces contain multiple input items, and it is difficult to automatically determine the value range or accepted data type of each input item, and it is difficult to establish Automatic matching between queries and entries

Images