Mobile terminal web crawler system

A web crawler, mobile terminal technology, applied in the field of mobile terminal web crawler system, can solve the problems of low coverage, difficult to support query, etc., to achieve the effect of strong function, good testability and maintainability, and simplified complexity

Pending Publication Date: 2019-12-13
河北上通云天网络科技有限公司
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, this anti-crawler method based on user access behavior and system search results contain a lot of irrelevant information, the coverage rate is low, and it is difficult to support queries based on semantic information.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mobile terminal web crawler system
  • Mobile terminal web crawler system
  • Mobile terminal web crawler system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0030] A mobile web crawler system, such as Figure 1 to Figure 6 shown, including the following steps:

[0031] Step 1, starting from the URLs of one or several initial web pages, obtaining the URLs on the initial web pages;

[0032] Step 2, in the process of crawling the webpage, continuously extract new URLs from the current page and put them...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a web crawler system, in particular to a mobile terminal web crawler system, which comprises the following steps of: 1, starting from URLs (Uniform Resource Locator) of one ormore initial web pages, obtaining the URLs on the initial web pages; 2, continuously extracting new URLs (Uniform Resource Locators) from the current page and putting the new URLs into a queue in a webpage crawling process until a certain stop condition of the system is met; 3, filtering links irrelevant to the theme according to a certain webpage analysis algorithm, reserving useful links, and putting the useful links into a URL queue to be grabbed; 4, selecting a webpage URL to be captured in the next step from the queue according to a certain search strategy, and repeating the process until a certain condition of the system is met; 5, storing all the web pages captured by the crawler by the system. According to the technical scheme, the defects that in the prior art, a search result contains a large amount of irrelevant information, the coverage rate is low, and query proposed according to semantic information is difficult to support can be effectively overcome.

Description

technical field [0001] The invention relates to a web crawler system, in particular to a mobile terminal web crawler system. Background technique [0002] A mobile web crawler is a program or script that automatically grabs information on the World Wide Web according to certain rules. The mobile web crawler has a wide range of applications, and can be used on mobile phones, mobile phones, and tablets. The systems include Huawei Hongmeng, Android, ios, and Windows on the mobile side. With the rapid development of the network, the World Wide Web has become the carrier of a large amount of information, how to effectively extract and use this information has become a huge challenge. As a tool to assist people in retrieving information, it becomes the entrance and guide for users to access the World Wide Web. However, these general search engines also have certain limitations, such as: first, users in different fields and backgrounds often have different retrieval purposes and ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/9535G06F16/951G06F16/955G06F16/903
CPCG06F16/9535G06F16/951G06F16/955G06F16/90344
Inventor 张鹏
Owner 河北上通云天网络科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap