Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Crawler scheduling method, device, electronic device and storage medium

A scheduling method and technology of a scheduling device, applied in the computer field, can solve problems such as low crawling efficiency and achieve the effect of improving crawling efficiency

Active Publication Date: 2021-01-26
BEIJING QIANXIN TECH
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Existing crawler technology usually uses manual rules to formulate the weight of crawling sites, so as to allocate corresponding crawler resources, and crawls in a regular polling manner, which is low in crawling efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Crawler scheduling method, device, electronic device and storage medium
  • Crawler scheduling method, device, electronic device and storage medium
  • Crawler scheduling method, device, electronic device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0034] In order to make the purpose, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of the present invention.

[0035] see figure 1 , figure 1 A schematic flow diagram of the crawler scheduling method provided in the first embodiment of the present invention, the method can be applied to electronic equipment, and the electronic equipment includes: mobile phones, tablet computers (Portable Android Device, PAD), notebook computers and personal digital a...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a crawler scheduling method, which is applied in the technical field of computers, and includes: obtaining data parameters of webpages to be crawled, and calculating statistics of the data parameters according to time series, and based on the statistics, the statistics include the number of times , mean, variance, covariance, and autoregressive coefficients. Through the logistic regression algorithm and the FTRL algorithm, determine the scheduling time for crawling the data parameters of the web page next time, and update the scheduling task queue according to the scheduling time. The invention also discloses a crawler scheduling device, electronic equipment and a storage medium, which can improve crawling efficiency of the crawler.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a crawler scheduling method, device, electronic equipment and storage medium. Background technique [0002] With the explosive growth of Internet information, the traditional way of collecting data by web crawlers has gradually shown its disadvantages. The existing crawler technology usually uses manual rules to formulate the weight of crawling sites, so as to allocate corresponding crawler resources, and crawls by periodic polling, which is low in crawling efficiency. Contents of the invention [0003] The main purpose of the present invention is to provide a crawler scheduling method, device, electronic equipment and storage medium, which can improve crawling efficiency of crawlers. [0004] In order to achieve the above object, the first aspect of the embodiment of the present invention provides a crawler scheduling method, including: [0005] Obtain the data par...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/951
Inventor 陈劲
Owner BEIJING QIANXIN TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com