Automatic optimized crawler grab method

Abstract

The invention discloses a grasping means for automatically optimizing a creeper. The prior creeper grasping web page means uses the same frequency to grasp issued peak information and issued trough information, thereby influencing the timeliness for grasping the information, reducing the efficiency of a system and increasing the pointless resource consumption. In order to solve the problem, the invention includes the following steps: firstly, the information is cramped out of an information page grasped from the Internet, if the cramping-out is successful, then the frequency for cramping out the information page again is quickened; or else, the frequency for cramping out the information page again is slowered; secondly, the step one is repeated when the amended frequency is reached. The invention is applicable for the various prior search engines.

Description

technical field [0001] The invention relates to a method for grabbing information by a web crawler, in particular to a method for a search engine to grab information by using crawler technology and automatically optimize the grabbing frequency. Background technique [0002] Search engine is a technology that is widely used on the Internet today. People only need to input some keywords of the information they are looking for to find a large amount of information related to this keyword through search engines, such as Google and Baidu. [0003] There are various sources of information for search engines, some of which pay advertising fees to the search engine operator from the advertiser who initiates the advertisement in the form of bidding advertisement, and then the search engine operator publishes the brief information and information of the advertisement in its own search engine. The link of the advertisement, and more non-advertising information, such as news and academi...

AI Technical Summary

Problems solved by technology

[0005] However, the above method of using crawlers to grab online information is still an information grabbing method based on an ideal state. In practical applications, the crawler's crawling efficiency cannot be maximized, because the release of new information often has It is extremely time-sensitive, and the release is more concentrated in one time period, and it is relatively quiet at other times. For example, the annual peak sales of train tickets, air tickets and long-distance bus tickets are winter and summer vacations and Golden Week. The peak is a period of time before and after the graduates of colleges and universities leave school every year, etc.

If we use the same frequency to capture the above information during the peak release period and the information during the low release period, obviously the highest efficiency of information capture cannot be achieved, because the frequency of information capture during the peak release period is relatively low, which affects all The timeliness of capturing information, the frequency of information capturing is relatively high for the low release period, which reduces system performance and increases unnecessary resource consumption

Until now, there is no way to effectively solve this problem

Images