Construction method for anti-mimic death crawler system

A technology of a crawler system and a construction method is applied in the construction field of a network data acquisition system to achieve the effects of preventing suspended animation and reducing development costs.

Inactive Publication Date: 2009-08-12
BEIJING UNIV OF POSTS & TELECOMM
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

At present, there is no systematic and effective method for constructing an anti-fake crawler system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Construction method for anti-mimic death crawler system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] Specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0032] figure 1 is a flowchart of a method according to one embodiment of the present invention. The process starts at step 101 . Then in step 102, a web page is requested from the server. It should be noted that the initial hyperlink should be a web page rich in hyperlinks, such as the home page of a website, etc. This is only an optimal example, and the difference in the initial hyperlink does not constitute a limitation to the present invention.

[0033] To request a web page from the server, one implementation is through the HTTP protocol GET method, that is, by sending GET request information to the server, hoping to obtain the web page specified by the URL. The above is an embodiment of requesting a webpage from a server, and other different implementation examples do not constitute limitations to the present invention.

[0034] Afte...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for establishing an anti-halt creeper system. The method comprises the following steps: (1) detecting and processing requested web pages; (2) detecting and processing network response; (3) detecting and processing memory space; and (4) repeatedly executing the step (1), the step (2) and the step (3) until all the hyperlinks of the web pages are processed. The method can effectively prevent the generation of the halt state of the creeper system, obviously reduce the waiting time of the creeper system and improve the creeping efficiency of the creeper system, provide a general framework for the establishment of the creeper system with robustness, and effectively reduce the development cost of the system.

Description

technical field [0001] The invention relates to a construction method of a network data collection system, in particular to a construction method of an anti-fake crawler system. Background technique [0002] Human beings have entered the information age, and the information explosion, more and more overwhelming information makes people breathless. In this situation, in order to extract useful information quickly and improve the efficiency of work and study, search engines have been proposed and implemented. As the basis of search engines and the only source of data processed by search engines, the status and importance of crawler systems are gradually highlighted. Unlike other search engine components, crawlers are closely related to network and storage, which causes the external environment to have a profound impact on the robustness of crawlers. The current general search engine crawler system has poor robustness and cannot adapt to the diversity of the network environme...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 杨溥郭军徐蔚然
Owner BEIJING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap