Real-time information crawler method and device based on intelligent page analysis, and equipment

Abstract

The invention relates to a real-time information crawler method and device based on intelligent page analysis and computer equipment. The method comprises the following steps: analyzing a URL of a detail page according to XPath configuration of each website, and logging a queue to be crawled into a table of a database through an SQL query component; crawling a URL of the detail page by utilizing a crawler assembly line; performing page rendering through a preset automatic page analysis algorithm or by calling a headless browser; extracting an HTML document of the detail page to obtain a text title, a source and release time of an article of the detail page; calculating MD5 of the detail page article and writing the MD5 into a database, where the text titles, the sources and the release time of the detail page articles and the MD5 of the detail page articles form crawler moment page snapshots; and polling the crawled detail pages according to a preset timed task and a crawler moment page snapshot to obtain a changed article list. By adopting the method, the crawler accuracy can be improved.

Description

technical field [0001] The present application relates to the technical field of data processing, in particular to a real-time information crawler method, device, computer equipment and storage medium based on intelligent page analysis. Background technique [0002] The official media releases real-time information on the webpage. It needs to use crawler technology to extract the article title, text, release time, source / author and other structured information on the webpage, and provide it to end users through its own APP. The development of the crawler system will face many problems. The label structure of the webpage lacks a unified standard, which leads to the need to write a crawler parsing code for each label style of the webpage to extract structured information, which requires a large amount of development work and is difficult to maintain. When the original When the structure of the web page is revised, it will lead to crawler failures; due to regulatory requirement...

AI Technical Summary

Problems solved by technology

The development of the crawler system will face many problems. The label structure of the webpage lacks a unified standard, which leads to the need to write a crawler parsing code for each label style of the webpage to extract structured information, which requires a large amount of development work and is difficult to maintain. When the original When the structure of the web page is revised, it will lead to crawler failures; due to regulatory requirements, it must be traceable, that is, structural information such as title / release time / source / content must be highly consistent with the original, when the title or text of the original web page occurs When changes are made, it is necessary to be able to reflect such changes in a timely manner

[0003] However, the current crawler technology has flaws: one is that information such as title / source / publishing time cannot be obtained; the other is that it cannot correctly parse the text of SMS articles, but instead crawls to company profiles or navigation pages, navigation lists and other content ; Can not achieve high accuracy (neither can contain page advertisements, navigation, and other spam information, nor can some paragraphs be missed); For web pages that dynamically obtain background data and render through asynchronous interfaces, they cannot be automated according to html text Analysis, unable to cope with changes in the content of the original page (such as retraction, content revision, etc.), low accuracy

Images