Multi-task processing system and method based on crawler technology

A crawler technology and processing system technology, applied in the field of electric vehicles, can solve the problems of difficult multi-task processing at the same time, unable to effectively guarantee the crawler effect, etc., to achieve the effect of improving the crawler speed, preventing download failure, and improving the crawler effect.

Pending Publication Date: 2020-11-03
芯薇(上海)智能科技有限公司
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The purpose of the present invention is to solve the problem that the existing web crawler technology is difficult to process multiple tasks at the same time and cannot effectively guarantee the crawler effect

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0030] A kind of multi-task processing system based on crawler technology of the present embodiment, comprises

[0031] A crawler engine, which is used to control the processing flow of the system;

[0032] A scheduler, which is used to receive commands from the crawler engine and feed back execution results to the crawler engine;

[0033] A downloader, which is used to receive commands from the scheduler and grab page data;

[0034] Pipeline component, which is used to receive and process the page data captured by the downloader;

[0035] The scheduler is respectively connected to the crawler engine, the downloader and the pipeline, and the downloader is connected to the pipeline component.

[0036] It also includes an intermediate component and a user operation module, the intermediate component includes a download intermediate component and a user intermediate component, and the download intermediate component is respectively connected to the crawler engine and the downlo...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The multi-task processing system based on the crawler technology comprises: a crawler engine, wherein the crawler engine is used for controlling the processing flow of the system; the scheduler whichis used for receiving a command of the crawler engine and feeding back an execution result to the crawler engine; the downloader which is used for receiving a command of the scheduler and capturing page data; the pipeline assembly which is used for receiving the page data captured by the downloader and processing the page data. The scheduler is in communication connection with the crawler engine,the downloader and the pipeline, and the downloader is in communication connection with the pipeline assembly. Compared with a traditional crawler technology, the crawler speed is increased through the mechanism, downloading failure caused by the fact that access is never forbidden by the crawler technology discovered by a website can be prevented, and the crawler effect is improved by 300%.

Description

technical field [0001] The invention belongs to the technical field of electric vehicles, and specifically relates to a multi-task processing system and method based on crawler technology. Background technique [0002] With the development of the Internet, the way people obtain information is gradually replaced by the Internet. In the early days of Internet development, people mainly obtained the information they needed by browsing portal websites, but with the rapid development of the Web, it became more and more difficult to find the information they needed in this way. At present, most people obtain useful information through search engines. Therefore, the development of search engine technology will directly affect the speed and quality of people's access to the information they need. [0003] In 1994, the world's first web search tool, Web Crawler, came out. At present, the more popular search engines include Baidu, Google, Yahoo, Info seek, Inktomi, Teoma, Live Search...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/951G06F16/955
CPCG06F16/951G06F16/9566
Inventor 钟静蔡斌江帅田元元
Owner 芯薇(上海)智能科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap