Internet data collection method and system based on man-in-the-middle

Abstract

The invention provides an Internet data collection method and system based on a man-in-the-middle. The method comprises the steps that a man-in-the-middle proxy certificate is installed to webpage information collection equipment, a man-in-the-middle of the webpage information collection equipment is established, and when the webpage information collection equipment accesses webpage information inthe Internet, the man-in-the-middle proxy certificate acts on all network traffic of the webpage information collection equipment; the man-in-the-middle acquires an collection task accommodating a URL regular expression of a to-be-acquired webpage, captures traffic conforming to the URL regular expression in all network traffic as intermediate traffic, injects the collection task into an HTML page of the intermediate traffic to obtain a to-be-analyzed page, and stores the to-be-analyzed page into a first database; and the analysis module distributes the to-be-analyzed page to the analyzer instance for analysis according to the URL information of the to-be-analyzed page in the first database, acquires a webpage collection result accommodating the structured data from the analyzer instanceand stores the webpage collection result into the second database. According to the invention, data collection of all applications which provide information by integrating browser kernel functions canbe supported.

Description

technical field [0001] The present invention relates to the field of web reptiles, in particular to a data acquisition method and system based on "man-in-the-middle attack", which can continuously inject different task codes into different application programs by using a man-in-the-middle agent to modify traffic data attacks to complete different pages. requests and obtain relevant data. Background technique [0002] A web crawler is a program that can effectively use various existing resources to automatically grab a large amount of web page information on the Internet, and is sometimes called a "web spider (Spider)". However, with the popularization of the mobile Internet, more traffic is directly distributed through various terminal applications, and WEB access is not provided or some data will be limited in WEB access, which brings great difficulties to data collection. . [0003] In the crawling process of the web crawler, it is necessary to obtain the request URL, se...

AI Technical Summary

Problems solved by technology

However, with the popularization of the mobile Internet, more traffic is directly distributed through various terminal applications, and WEB access is not provided or some data will be limited in WEB access, which brings great difficulties to data collection.

[0003] In the crawling process of the web crawler, it is necessary to obtain the request URL, send a WEB request to download the page, parse the structured data from the web page, filter the duplicate data, and process the seed task. There are 5 links in total. Each link consumes different resources and each Problems in each link will affect the efficiency and stability of the entire crawler system

In addition, with the rapid development of Internet technology, more and more information cannot be obtained through traditional WEB channels. A large amount of information is disseminated through specific applications, such as mobile information applications, etc., and among them A large amount of data is asynchronously requested and encrypted data using HTTPS. At present, there is a lack of a general data acquisition system compatible with various applications and various types of data.

Images