Webpage data crawling method and device and webpage login method and device

A technology of web page data and login method, applied in the network field, can solve the problem of easy failure of web page data, and achieve the effect of avoiding multiple repeated logins

Pending Publication Date: 2020-04-07
BEIJING GRIDSUM TECH CO LTD
View PDF6 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In view of the above problems, the present invention provides a method and device for crawling webpage data, the main purpose of which is to realize the crawling function of webpage data when crawling a website with a user account verification mechanism, and solve the problem of webpage data in such websites. There is an extremely easy failure problem in the process of data crawling

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Webpage data crawling method and device and webpage login method and device
  • Webpage data crawling method and device and webpage login method and device
  • Webpage data crawling method and device and webpage login method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0077] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present invention will be more thoroughly understood, and will fully convey the scope of the present invention to those skilled in the art.

[0078] In order to solve the problem that access requests are restricted or prohibited due to multiple logins in the process of logging in to a website that is set with account name and password verification, the embodiment of the present invention provides a web page login method, such as figure 1 As shown, the method includes:

[0079] 101. When a webpage login request is received, obtain identity credential information.

[0...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a webpage data crawling method and device, a webpage login method and device, relates to the technical field of networks, and mainly aims to solve the problem that crawling ofwebpage data in websites is extremely easy to fail. The method comprises the steps that when a crawling request is received, identity certificate information is obtained, and the identity certificateinformation is generated according to a user name and a password when a user logs in a webpage; logging in the webpage through the identity certificate information; and after logging in the webpage, crawling webpage data in the webpage. The invention is suitable for crawling the data in the website through the crawler.

Description

technical field [0001] The present invention relates to the field of network technologies, and in particular, to a method and device for crawling web page data, and a method and device for logging in to a web page. Background technique [0002] With the gradual increase of screen names, the number of visits to different websites in the network also increases gradually. Usually, in order to obtain the data in the website more comprehensively, many users like to crawl the data in the website through web crawlers. Among them, crawlers are often referred to as web crawlers, web spiders or web robots. It is a program or script that automatically grabs information from the World Wide Web according to certain rules. [0003] At present, many websites are equipped with user identity verification mechanisms, which require users to log in through account numbers and passwords. When crawlers crawl web page data in such websites, crawlers generally need account names and passwords to ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/951H04L29/06
CPCH04L63/083H04L63/0876
Inventor 满悦
Owner BEIJING GRIDSUM TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap