Agent-Based Intrusive Social Data Collection Method

Abstract

The invention discloses an agent-based intrusive social data collection method, which relates to the field of information collection. The invention includes a WeChat client, a proxy server, a program server and a WeChat server; a packet capture tool is used to obtain the data packet returned by the server to the client, inject JS and return it to the client, and automatically execute the JS code when the client loads the page, so that The browser establishes a connection with the program, and the program sends instructions to the browser to control the entire acquisition process. The invention loads more data through the pull-down operation, grabs the article link, and then executes the detailed link to obtain the article content, the number of likes, the number of readings and comments, etc., the article collection data of the official account is comprehensive, the operation is simple, and the data collection efficiency is improved.

Description

technical field [0001] The invention belongs to the field of information collection, in particular to an agent-based intrusive social data collection method. Background technique [0002] With the rapid development of the Internet, the network has become the most important means for people to obtain information, and with the continuous increase in the amount of data, how to effectively obtain and use these data has become a critical step. Information collection technology can more accurately obtain the specific data that users want, and the collection of a large amount of information data also provides a stage for the rise of anti-crawler technology, making data collection more and more difficult. As WeChat is the mainstream social software, WeChat official account articles have also become an important source of information collection. There are three existing WeChat public account article collection portals: (1) Sogou WeChat, (2) the interface provided by the material man...

AI Technical Summary

Problems solved by technology

Although there are also descriptions on the Internet about the collection of WeChat official account articles, most of them are incomplete, or just a simple overview. For related patents, some implementations are more complicated, or the data is obtained through the interaction between the browser and the Internet.

[0003] Today’s existing WeChat official account article collection technology uses Sogou WeChat as the entrance. The disadvantages of this method are: (1) anti-crawler restrictions, which require the assistance of an ip proxy and coding platform (2) the collected article links are not permanent (3) The number of likes, readings and comments of the article cannot be collected (4) The number of articles collected is limited to the last 10; there is an interface provided by the material management of the WeChat public platform For data, the disadvantages of this method are: (1) The login is more troublesome and requires the user to log in and scan the code to confirm; (2) Anti-crawler restrictions, frequent operations, and direct ban; (3) The obtained article link still cannot be liked counts, views, and comments

Images