Resource link acquisition method and device, electronic equipment and storage medium

Abstract

The invention provides a resource link acquisition method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a to-be-processed webpage corresponding to an access link; searching for events existing in all document nodes in the to-be-processed webpage, and storing the events existing in the document nodes into a to-be-processed queue; simulating and triggering events in the to-be-processed queue in a multi-thread manner by using a headless browser; intercepting a resource request generated by the event in the to-be-processed queue in the triggering process, and obtaining a resource link in the resource request. In the implementation process, the events existing in the webpage are stored in the to-be-processed queue, and then the events in the to-be-processed queue are triggered and intercepted, so that the situations that the webpage jumps for multiple times, the page is rendered again, a new page pops up for multiple times and the like are effectively avoided; and meanwhile, excessive consumption of computing resources and bandwidth resources caused by the conditions is avoided, so that the computing resources and the bandwidth resources are effectively saved.

Description

technical field [0001] The present application relates to the technical fields of network security and network communication, and specifically relates to a resource link acquisition method, device, electronic equipment and storage medium. Background technique [0002] At present, in the process of using crawlers to grab resource links in public web pages, multiple page jumps, page re-rendering, and multiple pop-ups of new pages often occur, which cause the browser to run many unnecessary processes resources or thread resources, and at the same time, multiple loading and jump requests to obtain web pages will also cause a waste of bandwidth resources. Therefore, there is a problem of excessive consumption of computing resources and bandwidth resources in the process of using existing crawlers to grab resource links in public web pages. Contents of the invention [0003] The purpose of the embodiments of the present application is to provide a resource link acquisition meth...

AI Technical Summary

Problems solved by technology

[0003] The purpose of the embodiments of the present application is to provide a resource link acquisition method, device, electronic device, and storage medium, which are used to improve the problem of excessive consumption of computing resources and bandwidth resources in the process of grabbing resource links from public web pages

Images