Anti-crawler data processing method, device, system and storage medium

A data processing device and data processing technology, applied in the field of data processing, can solve problems such as reducing the sense of operation of non-reptile users, and achieve the effect of suppressing real data

Inactive Publication Date: 2020-04-10
BEIJING GRIDSUM TECH CO LTD
View PDF0 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the method of setting IP access frequency will reduce the sense of operation of non-crawler users with high access frequency.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Anti-crawler data processing method, device, system and storage medium
  • Anti-crawler data processing method, device, system and storage medium
  • Anti-crawler data processing method, device, system and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0045] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0046] Usually, a web page has source code and display characters, what the user sees is the display character, and what is recorded in the background of the web page is the source code. For example, on a shopping website, the displayed characters seen by the user are "refrigerator", "washing machine" and the corresponding prices, and when the webpage is programmed, the source code is written to enable the webpage to display the abo...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides an anti-crawler data processing method, a device, a system and a storage medium. The method comprises the steps of firstly obtaining a to-be-replaced display character and a source code corresponding to the to-be-replaced display character; determining a target character code corresponding to the to-be-replaced display character based on a pre-configured corresponding relationship between common characters and character codes; and replacing the source code corresponding to the to-be-replaced display character with the target character code. Thus, when crawling the sourcecode, the crawler crawls the replaced target character code, and the crawler party does not know the corresponding relationship between the pre-configured common character and the character code, sothat the content of the real display character cannot be analyzed on the basis of the currently crawled target character code, and anti-crawling is realized. And the anti-crawling method not only caninhibit the crawler from acquiring the real data of the website, but also does not reduce the operation feeling of the non-crawler user.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to an anti-reptile data processing method, device, system and storage medium. Background technique [0002] The web crawler can save the visited pages and write the web index to achieve the purpose of obtaining website content and website index. However, the process of web crawlers accessing a website consumes system resources of the website, such as the number of website connections, network bandwidth resources, and the load of background servers. [0003] In addition, with the rapid development of the Internet, network information security issues have become increasingly prominent. For the purpose of protecting website data security, an anti-crawler mechanism is usually set up on a website to prevent crawlers from obtaining its website data. [0004] The commonly used anti-crawler mechanism is to restrict crawlers by setting the frequency of IP access. For exampl...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F21/14G06F8/51G06F40/126
CPCG06F8/51G06F21/14
Inventor 李可欣
Owner BEIJING GRIDSUM TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap