Method and device for grabbing timeliness seed page

Abstract

The invention discloses a method and device for grabbing a timeliness seed page. The method comprises the steps that for the timeliness seed page, a frequency regulation factor corresponding to grabbing conducted on the seed page every time within the current preset time period is obtained; a frequency regulation coefficient is calculated based on the frequency regulation factors; the grabbing frequency of the historical preset time period corresponding to the current preset time period is determined; whether the current preset time period belongs to the switching time between holidays and working days or not is judged, and if yes, the holiday factor is determined according to the switching situation of the holidays and working days; the grabbing frequency within the current preset time period is dynamically regulated according to the grabbing frequency, the frequency regulation coefficient and the holiday factor. According to the method and device for grabbing the timeliness seed page, the holiday factor and the grabbing frequency can be dynamically regulated, unnecessary grabbing rate of the seed page is reduced, and new links can be found timely and will not be lost.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a method and device for capturing time-sensitive seed pages. Background technique [0002] The Internet is always generating new content, such as news, various popular discussions, and so on. These new contents are scattered in different corners of the Internet. In order to retrieve them in time, search engines need to find and grab them from the vast Internet in time. Fortunately, links to time-sensitive content almost always appear on a specific type of page, which is called a time-sensitive seed page (hub page for short), such as http: / / news.sina.com.cn / . So in theory, you only need to find these hub pages, and then check their changes in time to find all the time-sensitive links. [0003] The content of the hub page is constantly changing, and new links are likely to disappear after a period of time. Like the layout of the forum, the scrolling is very fast, and the ne...

AI Technical Summary

Problems solved by technology

Failure to spot their changes in time can result in new links being lost

But every time you check a page, you need to initiate a crawl. If the search engine keeps checking this kind of page, it will generate a lot of crawls for the website to which it belongs.

This large amount of crawling may not be able to discover new links in many cases, resulting in a large number of meaningless crawls, and it is also a very impolite behavior, which may even trigger the ban of search engine crawlers on the website, resulting in the crawler being unable to link for a period of time. visit website

Due to the obvious difference in the number of people accessing the Internet during holidays and working days, the amount of information generated on the Internet is also significantly different, so if the search engine keeps the same frequency of crawling during holidays and working days, some new links will not be crawled or cause meaningless crawling. Pick

Images