Scrappy-based distributed dark-net resource mining system and method

Abstract

The invention relates to the field of data mining and discloses a scrappy-based distributed dark-net resource mining system and a scrappy-based distributed dark-net resource mining method. The scrappy-based distributed dark-net resource mining system and the scrappy-based distributed dark-net resource mining method are used for improving the efficiency, scope and flexibility of dark-net resource mining. The system comprises a center node control module and a slave-node crawling module, wherein the center node control module includes a crawler seed task queue, a task preprocessing module, a dark-net task queue and an open-net task queue; the slave-node crawling module includes a dark-net crawling module, an open-net crawling module and a crawler manager; the dark net crawls more dark-net domain names from a dark-net page and an open-net page through the dark-net crawling module and the open-net crawling module on the basis of artificial provision and the dark-net domain name crawled through the open net so as to implement the effects that a lot of dark-net domain names are acquired and dark-net web pages are stored. The scrappy-based distributed dark-net resource mining system and the scrappy-based distributed dark-net resource mining method are suitable for mining of dark-net resources.

Description

technical field [0001] The invention relates to the field of data mining, in particular to a scrapy-based distributed dark network resource mining system and method. Background technique [0002] The dark web refers to a network that can only be accessed through special software or using non-standard communication protocols and ports. Tor is currently the most mainstream anonymous communication system on the dark web. Due to the complete anonymity of the dark web, it has given birth to a large number of illegal transactions. Therefore, the research is of great significance to the mining of dark web resources. Traditional search engines and crawler technology can crawl only a small part of web information provided on the Internet, that is, information on the Mingnet. The mining of dark web resources cannot be realized. Most of the existing research is on the non-surface web content on the Internet that cannot be indexed by standard search engines, that is, the deep web, no...

AI Technical Summary

Problems solved by technology

Therefore, it is likely to cause stand-alone memory exhaustion

Using scrapy to crawl on a single machine, memory will be the bottleneck

Images