Self-involved efficient proxy resource supply system and method

Abstract

The embodiment of the invention discloses a self-involved efficient proxy resource supply system and method. The system comprises a proxy supply terminal, a service terminal and a proxy resource server, wherein the proxy supply terminal constructs a multi-proxy engine in which a plurality preset proxies coexist and a corresponding local strategy base; the service terminal obtains the use behaviordata of the proxies, packs the use behavior data and uploads the use behavior data to the proxy resource server; the proxy resource server constructs a proxy base, a proxy strategy base and a proxy behavior base, provides proxy resource information for the proxy supply terminal, generates an corresponding efficient proxy strategy by performing machine learning according to the use behavior data and updating the efficient proxy strategy to the proxy strategy base. In the self-involved efficient proxy resource supply system and method disclosed by the embodiment of the invention, the multi-proxyengine is used for ensuring a plurality of proxy services can be provided quickly and efficiently after the IP of a single proxy resource is changed, and collecting the use behavior of the proxy to perform the machine learning to generate the efficient proxy strategy, the problem of low proxy utilization is solved, and thus reducing the proxy cost.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a self-evolving and efficient proxy resource supply system and method. Background technique [0002] The information on the Internet is growing massively every day, and there is huge potential commercial value in a large amount of information. Web crawlers are becoming more and more valuable. However, the current target website has many restrictions on the number of visits and methods of crawlers, especially for the access IP There are strict constraints, such as certain pages or services can only be accessed several times with the same IP within a period of time, which is far from meeting the growing needs of crawler business. [0003] At present, the existing crawler technology mainly adopts the following methods to obtain agents: [0004] 1. The network sniffs out a large number of agents and records them in the agent management center: the sniffing efficiency is low, th...

AI Technical Summary

Problems solved by technology

[0004] 1. The network sniffs out a large number of agents and records them in the agent management center: the sniffing efficiency is low, the cost is high, the reliability is low, and there are legal risks;

[0005] 2. Using agents provided by online agent service providers: the intelligent linkage between business and agents cannot be realized;

[0006] 3. Internet broiler: illegal and without business cooperation, intelligent linkage cannot be realized, and the agent is uncontrollable (it will be gone after shutdown);

[0007] 4. Collection of free internet proxies published on the Internet: the availability rate is low and the proxies are uncontrollable;

[0009] Even if the agent is obtained by the above five methods, due to the many restrictions on the machine where the agent is deployed, it is unstable, which increases the input cost of the crawler

Due to the high cost of the proxy, there needs to be a way to use it efficiently; in addition, the existing proxy is only used for one purpose, such as HTTP proxy, TCP proxy or other business proxies, which cannot be shared and cannot effectively improve proxy utilization.

Images