Webpage crawling method and device based on multiple threads, storage medium and equipment

Abstract

The embodiment of the invention discloses a webpage crawling method and device based on multiple threads, a storage medium and equipment. The method comprises the steps of obtaining a plurality of to-be-crawled keywords from a keyword dictionary; determining a link address corresponding to the keyword to be crawled according to the keyword to be crawled; adding a link address corresponding to thekeyword to be crawled into a queue; then, obtaining a link address corresponding to the keyword to be crawled from the queue as a link address to be crawled through each thread crawler of the multi-thread crawler in a multi-thread crawler mode; and according to the to-be-crawled link address, performing webpage crawling and content analysis through each thread crawler to obtain effective information corresponding to the to-be-crawled link address, and updating the effective information of the keyword dictionary according to the effective information corresponding to the to-be-crawled link address. According to the method, manual search is not needed, the situation that the website providing the information frequently accesses the professional website in the peak period of user access is avoided, and the keywords in the webpage of the website providing the information can be updated in time.

Description

technical field [0001] The present invention relates to the technical field of computer networks, in particular to a method, device, storage medium and equipment for crawling webpages based on multithreading. Background technique [0002] With the rapid development of the Internet, more and more people are looking for the information they need on the computer network. The information they find includes many keywords. For example, keywords include professional terms. You need to read the explanation of the keywords to understand the whole information. Keyword explanations can be manually searched through search engines, resulting in poor user experience, especially when there are multiple keywords in one piece of information, switching back and forth between the search engine and the webpage displaying the information is not conducive to user reference, and it is relatively Time-consuming; the explanation of keywords can also be provided by the information-providing website,...

AI Technical Summary

Problems solved by technology

Keyword explanations can be manually searched through search engines, resulting in poor user experience, especially when there are multiple keywords in one piece of information, switching back and forth between the search engine and the webpage displaying the information is not conducive to user reference, and it is relatively Time-consuming; the explanation of keywords can also be provided by information-providing websites, but the explanations of keywords are maintained by professional websites. Display the explanation of the obtained keywords to the user, causing the websites that provide information to frequently visit professional websites, and it is not conducive to the timely update of the explanation of keywords

Images