Self-evolving efficient agent resource provisioning system and method

Abstract

The embodiment of the present invention discloses a self-evolving and efficient agent resource supply system and method. The system includes an agent supply end, a business end, and an agent resource server end. The agent engine and the corresponding local policy library; the business end obtains the use behavior data of the agent, manages the use behavior data and uploads it to the agent resource server; the agent resource server builds the agent library, the agent policy library and the agent behavior library to provide The provider provides agent resource information, performs machine learning based on usage behavior data, generates corresponding efficient agent strategies and updates them to the agent strategy library. The embodiment of the present invention uses a multi-agent engine to ensure that a single agent resource can quickly and efficiently provide multiple agent services after changing the IP, and collect agent usage behaviors for machine learning to generate efficient agent strategies, which solves the problem of low agent utilization. problems, thereby reducing agency costs.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a self-evolving and efficient proxy resource supply system and method. Background technique [0002] The information on the Internet is growing massively every day, and there is huge potential commercial value in a large amount of information. Web crawlers are becoming more and more valuable. However, the current target website has many restrictions on the number of visits and methods of crawlers, especially for the access IP There are strict constraints, such as certain pages or services can only be accessed several times with the same IP within a period of time, which is far from meeting the growing needs of crawler business. [0003] At present, the existing crawler technology mainly adopts the following methods to obtain agents: [0004] 1. The network sniffs out a large number of agents and records them in the agent management center: the sniffing efficiency is low, th...

AI Technical Summary

Problems solved by technology

[0004] 1. The network sniffs out a large number of agents and records them in the agent management center: the sniffing efficiency is low, the cost is high, the reliability is low, and there are legal risks;

[0005] 2. Using agents provided by online agent service providers: the intelligent linkage between business and agents cannot be realized;

[0006] 3. Internet broiler: illegal and without business cooperation, intelligent linkage cannot be realized, and the agent is uncontrollable (it will be gone after shutdown);

[0007] 4. Collection of free internet proxies published on the Internet: the availability rate is low and the proxies are uncontrollable;

[0009] Even if the agent is obtained by the above five methods, due to the many restrictions on the machine where the agent is deployed, it is unstable, which increases the input cost of the crawler

Due to the high cost of the proxy, there needs to be a way to use it efficiently; in addition, the existing proxy is only used for one purpose, such as HTTP proxy, TCP proxy or other business proxies, which cannot be shared and cannot effectively improve proxy utilization.

Images