Crawler identification method and device for website server

Abstract

The invention belongs to the field of the Internet, and particularly relates to a crawler identification method and device for a website server. The method comprises the following processes that: on the basis of the IP (Internet Protocol) address of a user, marking an access user; dividing user access requests, classifying the user access requests with the same request resource, request domain name, subdomain name and access path in the user access request into the same type of access requests; on the basis of resource type of each class of access requests, dividing a core resource access request; on the basis of the request time continuity, the request content continuity, the access duration and the access frequency of the core resource access request of the access user, carrying out crawler access judgment; and processing the user who is judged as crawler access. By use of the method, through various means, the crawler behavior of the user is effectively identified, implementation cost is low, a coupling degree with website businesses is low, and transportability is high. The method is independent of website business service logic, is easy in updating and upgrading and is low in operation and maintenance cost and technical risks.

Description

Technical field [0001] The invention relates to the Internet field, in particular to a method and device for identifying a crawler of a website server. Background technique [0002] Internet applications are currently prosperous. Web crawlers will occupy the valuable bandwidth and computing resources of the crawled website servers. In addition, with the rise of big data technology, website data and content resources have increasingly become the core assets of website service providers. How to effectively identify the data crawling behavior of crawler robot programs from a large number of ordinary user access requests has become one of the main technical problems that major websites urgently need to solve. [0003] At present, there are anti-crawler work, and it is often that each website developer fights separately and builds the anti-crawler into the business server according to their own business characteristics. However, on the one hand, it increases the technical and business...

AI Technical Summary

Problems solved by technology

However, on the one hand, this increases the technical and business costs of each website. On the other hand, such technical methods are often too highly coupled with the specific business of the website and are not portable.

[0004] In addition, even if the anti-crawler logic is coupled with the website business server logic, it will cause trouble in improving and upgrading the anti-crawler technology, and cannot effectively deal with new crawler technologies and situations

If the server logic is redeployed due to the anti-crawling logic upgrade, it will increase the website operation and maintenance cost and increase the technical risk

Images