Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Webpage information processing method and system, electronic equipment and storage medium

An information processing method and webpage information technology, applied in the field of systems, electronic devices and storage media, and webpage information processing methods, can solve the problems that analysis modules and processing modules cannot be flexibly configured and modified, so as to save analysis and processing resources and improve The effect of processing efficiency

Active Publication Date: 2019-04-09
江苏运满满信息科技有限公司
View PDF9 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] In view of this, the present application provides a webpage information processing method, system, electronic device and storage medium to solve the problem that the parsing module and processing module cannot be flexibly configured in the prior art, and the source code must be modified every time a new crawling requirement is added. question

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Webpage information processing method and system, electronic equipment and storage medium
  • Webpage information processing method and system, electronic equipment and storage medium
  • Webpage information processing method and system, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The same reference numerals denote the same or similar structures in the drawings, and thus their repeated descriptions will be omitted.

[0038] The present application provides a method for processing webpage information that can dynamically modify configuration, and is flexible in parsing and processing, and saves resources, as well as a corresponding system, device, and storage medium. The following uses the Java language as an example to illustrate the web page information processing method of the present application. Those skilled in the art can also u...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a webpage information processing method and system, electronic equipment and a storage medium, relates to the field of data processing, and is used for crawling and processing webpage resources. The webpage information processing method comprises the steps: acquiring a URL address to be crawled, and crawling webpage information according to the URL address; Calling an analysis module bound with the URL address in a database, analyzing the crawled webpage information, and extracting the data content of the webpage information; And calling a processing module bound with the URL address in a database, and processing the analyzed data content. According to the invention, the URL address is bound with the corresponding analysis module and processing module in a mutual mapping manner; when a crawling task of a certain URL address is executed, the URL address is transmitted in a process, so that a corresponding analysis module and a corresponding processing module are flexibly called from a database according to the URL address to process crawled webpage information; And the URL address and the corresponding analysis module and processing module can dynamically modify the configuration.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular, relates to a webpage information processing method, system, electronic device and storage medium. Background technique [0002] Crawler technology is a technology that automatically crawls specific web resources according to certain rules. Since the web pages of each website are different, custom development for specific web resources is required when crawling different web resources. [0003] Existing crawler technology system reference figure 1 As shown, the scheduling center controls multiple crawling lines, and each crawling line corresponds to grabbing a webpage resource. Each crawling line includes three modules, namely the webpage download module, the webpage analysis module and the data processing module. The crawling lines are independent of each other. When adding a web page resource acquisition requirement, it is necessary to add a program for web...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/955
Inventor 曹功源
Owner 江苏运满满信息科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com