A method and device for extracting deduplication feature values ​​of web crawlers

A technology of web crawler and extraction method, which is applied in the field of extraction of web crawler deduplication eigenvalues, which can solve problems such as waste of network resources and system memory, failure of deduplication, and impact on data analysis effects, etc., to achieve improved accuracy and high computing efficiency , the effect of weight removal is obvious

Active Publication Date: 2018-02-09
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] For the above situation, the failure to deduplicate different URLs of the same page not only wastes network resources and system memory, but also in the field of data mining, treating the same page as different pages will seriously affect the final data analysis effect

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and device for extracting deduplication feature values ​​of web crawlers
  • A method and device for extracting deduplication feature values ​​of web crawlers
  • A method and device for extracting deduplication feature values ​​of web crawlers

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0023] The technical solutions of the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0024] Based on the characteristics of the URL string of the target website, the invention extracts a substring that can uniquely identify the webpage from the URL link pushed by the crawler as a webpage feature string, and then converts the format of the feature string to obtain a deduplication feature value. In the case of multiple URL links pointing to the same page, after deduplication of the characteristic value of this method, the page is downloaded only once, and repeated downloads will not occur.

[0025] figure 1 A block flow diagram showing a method for extracting a web crawler deduplication feature value according to an embodiment of the present invention, including:

[0026] S101, generating a regular expression according to the URL domain name and the URL seed feature of the target website;

[0027] S...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for extracting a web crawler reduplication-removing characteristic value. The method comprises the following steps of: step 1, generating a regular expression according to the uniform resource locator URL realm name and URL seed characteristics of a target website; step 2, capturing a character string matched with the regular expression from the URL address of the target website; step 3, carrying out predetermined format conversion on the character string to obtain the reduplication-removing characteristic value of the URL address. The invention further provides a device for extracting a web crawler reduplication-removing characteristic value. The method and the device disclosed by the invention are suitable for realizing reduplication-removing for the case that different URL links point to the same page.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a method and a device for extracting a deduplication feature value of a web crawler. Background technique [0002] The number of links in the Internet is huge, and the number of URL (Uniform Resource Locator, Uniform Resource Locator) links is tens of billions. Web crawlers (Web Spider) filter and deduplicate duplicate URLs when crawling web pages, which can prevent duplication of the same web pages. Download, which not only avoids the waste of network IO resources and the slowdown of crawler crawling speed, but also avoids the waste of resources caused by data redundancy and repeated processing. In the crawler deduplication process, the extraction of the URL deduplication feature value (the string that uniquely identifies the webpage) is an important part of filtering and deduplication. Whether the deduplication feature value is correct or not directly affects the crawling ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/9566
Inventor 黎小为周东
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap