Webpage data analysis method and device and computer readable storage medium

Abstract

The invention relates to the technical field of data acquisition, and discloses a webpage data analysis method, which comprises the following steps: when data capture is carried out, aiming at a network page where data to be captured is located, analyzing and acquiring the number of pages where data to be captured is located; according to the number of pages where the analyzed to-be-captured datais located, data capture is conducted in a data capture mode matched with the number of the pages, and captured page data are obtained; and carrying out data processing on the captured page data, generating required structured data and storing the structured data. The invention further provides a webpage data analysis device and a computer readable storage medium. According to the method, a webpage data analysis technology that webpage capturing primary analysis and structured data format conversion are executed separately is realized, and excessive dependence of webpage data analysis on a source website structure is reduced; in addition, webpage capturing primary analysis and structured information extraction format conversion are separated, independent operation is achieved, and the beneficial effect of high cohesion and low coupling is achieved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a web page data parsing method, device and computer-readable storage medium. Background technique [0002] With the continuous advancement of Internet information technology and the diversification of Internet applications, network technology has increasingly profoundly changed people's work, study and lifestyle, and even affected the entire process of society. The rapid development of the Internet has accelerated the arrival of the era of big data. Global companies are full of enthusiasm for big data, and big data analysis and processing have emerged as the times require. The big data processing process mainly includes data acquisition, data storage integration, data preprocessing, data mining analysis, data presentation and application, etc. When traditional industries develop big data, the first thing they face is how to obtain Internet data that is not based ...

AI Technical Summary

Problems solved by technology

The first method cannot deal with issues such as sequence and data association between web pages well during centralized parsing; the second method relies too much on the structure and format of the source website, and if there is an exception, the web page must be re-crawled. Coupling is too high

In addition, the above two analysis methods of webpage data are lacking in data analysis engine, configurable data extraction and format conversion.

Images