Semi-automatic anti-crawling system based on behavior characteristics

Abstract

The invention belongs to the field of network data security, and relates to a semi-automatic anti-crawling system based on behavior characteristics, which comprises an ETL processing unit, a behavioranalysis and management unit, a log analysis engine and a request protection processing unit, wherein the ELT processing unit is used for carrying out ETL (Extract Transform and Load) processing on request information of a request initiated by a user side to obtain UID (Uniform Identifier) and URI (Uniform Resource Identifier) data; the behavior analysis and management unit performs deduplicationprocessing on the received URI data by taking the received URI as a main key aggregation to generate a behavior set and statistical indexes related to a service scene;, takes the behavior set as a main key aggregation, and generates an analysis view to judge whether the behavior set is threatened or not; manages the threat behavior set,and if it is judged that the behavior set is the threat behavior set, records and tracks the threat behavior set, automatically generates a disposal strategy parameter according to the behavior set access frequency, the URI data type and the URI data quantity, and pushs the disposal strategy parameter to a database for crawler real-time monitoring.

Description

technical field [0001] The invention belongs to the field of network data security, and relates to a crawler automatic identification and disposal system based on user behavior analysis. Background technique [0002] The existing anti-crawling system based on back-end data generally has two directions: one is the crawler identification and interception method based on artificial strategy and character feature matching, and the other is the crawler identification and interception combined with supervised and unsupervised machine learning method. The former method requires the cooperation of business experts and strategy experts to extract and apply crawler identification features. It has a long cycle from identification to interception, high labor costs, and low versatility. The second type introduces machine learning methods to identify reptiles. The advantage is that the rule discovery is automatically learned from the data samples by the algorithm. The disadvantage is tha...

AI Technical Summary

Problems solved by technology

The former method requires the cooperation of business experts and strategy experts to extract and apply crawler identification features. It has a long cycle from identification to interception, high labor costs, and low versatility.

The second type introduces machine learning methods to identify crawlers. The advantage is that the rule discovery is automatically learned from the data samples by the algorithm. The disadvantage is that a large amount of data annotation is required, and the cost of generalization and update is high.

Also, if the behavior of the crawler changes, it will be difficult to catch the new crawler if the machine learning pre-designed characteristics cannot characterize it

In addition to the above shortcomings, the common problem of existing anti-climbing systems is low generalization ability

Artificial rules, character features, and machine learning are often summarized in specific business scenarios, and it is difficult to quickly access and take effect when facing new scenarios

Images