An agent-based intrusive social data acquisition method

Abstract

The invention discloses an agent-based intrusive social data acquisition method, and relates to the field of information acquisition. The system comprises a WeChat client, a proxy server, a program server and a WeChat server, a packet capturing tool is used for obtaining a data packet returned by a server side to a client side, a JS is injected and returned to the client side, a JS code is automatically executed when the client side loads a page, so that the browser is connected with a program, and the program sends an instruction to the browser to control the whole collection process. According to the method, more data are loaded through the pull-down operation, the article links are captured, then the detail links are executed to obtain article content, the like number, the reading number, comments and the like, the public number article collection data are comprehensive, operation is easy, and the data collection efficiency is improved.

Description

Technical field [0001] The invention belongs to the field of information collection, and particularly relates to an agent-based invasive social data collection method. Background technique [0002] With the rapid development of the Internet, the Internet has become the most important means for people to obtain information, and with the continuous increase in the amount of data, how to effectively obtain and use these data has become a critical step. Information collection technology can more accurately obtain the specific data users want, and the collection of a large amount of information data also provides a stage for the rise of anti-crawler technology, making data collection more and more difficult. As the mainstream social software, WeChat public account articles have also become an important source of information collection. There are three existing WeChat public account article collection entrances: (1) Sogou WeChat, (2) WeChat public platform material management interfac...

AI Technical Summary

Problems solved by technology

Although there are also descriptions on the Internet about the collection of WeChat official account articles, most of them are incomplete, or just a simple overview. For related patents, some implementations are more complicated, or the data is obtained through the interaction between the browser and the Internet.

[0003] Today’s existing WeChat official account article collection technology uses Sogou WeChat as the entrance. The disadvantages of this method are: (1) anti-crawler restrictions, which require the assistance of an ip proxy and coding platform (2) the collected article links are not permanent (3) The number of likes, readings and comments of the article cannot be collected (4) The number of articles collected is limited to the last 10; there is an interface provided by the material management of the WeChat public platform For data, the disadvantages of this method are: (1) The login is more troublesome and requires the user to log in and scan the code to confirm; (2) Anti-crawler restrictions, frequent operations, and direct ban; (3) The obtained article link still cannot be liked counts, views, and comments

Images