Management method, device and system of website access request

Abstract

The invention discloses a management method, device and system of a website access request. The method comprises the following steps: intercepting a website access request initiated from terminal equipment; if the website access request is not initiated through a browser, identifying the website access request as a website crawler request, and refusing the website access request; if the website access request is initiated through the browser, analyzing and judging whether the website access request is a suspect network crawler request according to the relevant access parameters of all website access requests initiated from the terminal equipment in a preset duration or not; if the website access request is the suspect network crawler request, generating a two-dimensional code used for identity verification, and returning the two-dimensional code to the terminal equipment; and if the terminal equipment passes the identity verification, releasing the website access request, identifying the website access request as the website crawler request if the terminal equipment does not pass the identity verification, and refusing the website access request. The management method, device and system can effectively prevent network crawlers from crawling website data through ways that the network crawler pretends to be a browser and utilizes the IP (Internet Protocol) address agent or IP address rotation.

Description

technical field [0001] The invention belongs to the technical field of computers and networks, and in particular relates to a method and device for managing website access requests. Background technique [0002] A web crawler is a program or script that automatically grabs information on the World Wide Web according to certain rules. With the development of search engine technology, web crawlers have become a very popular web search technology. Except for Google, Yahoo, Microsoft, and Baidu, which specialize in search, almost every large portal website has its own search engine. A variety of crawler systems used by unknown companies to crawl website data. For a content-driven website, it is inevitable to be patronized by web crawlers. [0003] Some intelligent search engine crawlers crawl website data at a reasonable frequency and consume less website resources. But many bad web crawlers often send hundreds or even thousands of requests to repeatedly crawl website data. T...

AI Technical Summary

Problems solved by technology

However, the above-mentioned anti-crawler measures are not perfect. Web crawler designers can often avoid existing anti-crawler measures by adopting some targeted strategies, such as: using proxy technology to simulate browser visits to reduce visit frequency, And by constantly changing the IP address (Internet Protocol Address, Internet Protocol address) to crawl the website page data, etc.

Therefore, existing technologies cannot effectively manage website access requests based on web crawler technology

Images