Semi-automatic vertical crawler generation tool and method

Abstract

The invention discloses a semi-automatic vertical crawler generation tool and method. The semi-automatic vertical crawler generation method comprises the steps that a user can select to newly establish a template or adopt the lex-yacc technology to open a template, conduct lexical and grammatical analysis on a template file, maintain a symbol table and construct a grammatical analysis tree; a tree template structure is constructed according to the content of the template by searching for the rules of the lexical and grammatical analysis and storing and processing data in the whole analysis process; according to the content to be extracted, template nodes are increased, modified or deleted in the tree template structure, wherein node information contains the skip relation, an XPath expression and the data storage modes; the template is saved. The semi-automatic vertical crawler generation tool comprises an automatic crawler generation tool module and a crawler module. According to the semi-automatic vertical crawler generation tool and method, automatic generation of the template content replaces manual configuration of the template, so that the template is configured more conveniently and quickly, and the workload of related staff can be greatly saved.

Description

technical field [0001] The invention belongs to the technical field of search engines, in particular to a semi-automatic vertical crawler generation tool and method. Background technique [0002] With the development of search engine technology, vertical search applications serving specific fields began to emerge. Vertical crawlers can selectively access target links of Internet pages according to specific targets to obtain page information. It does not pursue large and wide coverage, but only focuses on a certain field or industry, and selects the next page to be crawled from the URL queue according to the search engine indexing strategy. Although vertical crawlers have advantages such as higher accuracy compared with batch crawlers and incremental crawlers, vertical crawlers still need to manually intervene in the configuration of crawler templates, which is also determined by the characteristics of vertical crawlers. [0003] The efficiency of manual template configurat...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the embodiments of the present invention is to provide a semi-automatic vertical crawler generation tool and method, aiming to solve the existing vertical crawlers that require manual intervention in the configuration of crawler templates, which are prone to grammatical errors, and cannot visually check the template content and work. The problem of inefficiency

Images