Chinese literature data automatic acquisition method based on web crawler technology

A web crawler and document data technology, applied in the field of web crawlers, can solve the problems of inability to obtain data, achieve high crawling quality, solve data duplication and data missing problems, and facilitate data filtering

Pending Publication Date: 2020-07-03
BEIJING NORMAL UNIVERSITY
View PDF3 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Therefore, big data also puts forward higher requirements on people's ability to obtain data. Traditional t...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Chinese literature data automatic acquisition method based on web crawler technology
  • Chinese literature data automatic acquisition method based on web crawler technology

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049]In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings and embodiments. It should be noted that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0050] A method for automatically acquiring Chinese literature data based on web crawler technology, mainly including the following modules: a web page analysis module, a web page operation module, an element identification module, and a data storage module.

[0051] The webpage analysis module aims at performing webpage structure analysis and automatic operation on a given webpage. Such as figure 1 As shown, the basic process is: (1) for each program script, a network request will be created and sent to the browser driver; (2) the browser driver includes a web server to receiv...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a Chinese literature data automatic acquisition method based on a web crawler technology, which constructs a set of network framework for automatically acquiring Chinese literature data by calling a Selenium library and other related modules in Python in combination with structural characteristics of a target webpage. According to the method, starting from webpage structureanalysis, the parameterized expression of the required text is obtained by analyzing the Xpath path expression in the webpage, and efficient, accurate and automatic crawling of data is achieved through a large number of experimental debugging. The method has important significance for establishing a Chinese scientific literature database and promoting science development.

Description

technical field [0001] The invention is an automatic acquisition method for Chinese document data based on the web crawler technology, and belongs to the technical field of web crawlers. Background technique [0002] In 2014, my country included "big data" in the government work report for the first time, ushering in the "first year of big data in China" in the true sense. As a product of the highly developed era of information technology, compared with traditional data, big data often has the characteristics of large data scale, fast circulation speed, multiple data types, and low value density. [0003] Therefore, big data also puts forward higher requirements on people's ability to obtain data. Traditional tools and methods have been unable to allow humans to obtain the data needed for research within an affordable time frame. [0004] In order to solve the technical problems brought about by the explosive growth of data, web crawler technology came into being. We can f...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/951G06F9/455G06F40/143G06F16/81G06F16/16
CPCG06F16/951G06F9/45529G06F16/81G06F16/164
Inventor 赵子鸣李本继陈清华李小萌
Owner BEIJING NORMAL UNIVERSITY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap