Domain name data mining method and device and Redis server

Abstract

The embodiment of the invention discloses a domain name data mining method and device and a Redis server. When the method is applied to any host, the method comprises the steps of creating connectionwith the Redis server, operating a plurality of crawler threads and controlling each crawler thread to sequentially obtain a crawling task from a task distribution module and execute the crawling task; analyzing response data returned after the target crawler thread executes the target crawling task, obtaining corresponding target domain name information, sending the corresponding target domain name information to a domain name deduplication module, and judging whether the target domain name information is repeated with a crawled domain name or not; if so, abandoning the target domain name information; and if not, storing the target domain name information to a local storage module. The domain name data collected repeatedly is deleted before storage, so that the problem of distributed crawler deduplication is effectively solved, and the subsequent situations of task accumulation and data redundancy are avoided.

Description

technical field [0001] The present application relates to the technical field of network security, in particular to a domain name data mining method, device and Redis server. Background technique [0002] At present, the mining of domain name data is mainly realized by means of web crawlers. Web crawling refers to writing crawler scripts to obtain domain name data. The basic workflow includes: first select some URLs (Uniform Resource Locators, Uniform Resource Locators) as seed URLs and put them in the queue to be crawled, and then write crawler scripts to target Take the seed URL in the queue, access the website by simulating manual browsing, store and analyze the crawled web page HTML (Hyper Text Markup Language, Hypertext Markup Language) data, and use the parsed new link as the next layer of crawling Fetched torrent URL. [0003] In order to improve crawling efficiency, the same crawler program is usually executed on multiple hosts for distributed crawling of domain na...

AI Technical Summary

Problems solved by technology

[0004] In order to solve the problem that each host first saves all the domain name data collected locally, and subsequently adopts the parsing program to de-duplicate the domain name data collected by all hosts, which will lead to the accumulation of tasks when there are many crawling tasks, resulting in data For the problem of redundancy, this application discloses a domain name data mining method, device and Redis server through the following embodiments

Images