Social platform-oriented data acquisition method and system

Abstract

The invention discloses a social platform-oriented data acquisition method. The method comprises the following steps of: 1) acquiring Web page data of related pages on a social platform on a host by simulating a browser access request sent by a user; 2) connecting the real machine or the simulator to a host, simulating an APP access request sent by a user, setting a network agent on the host, intercepting the APP access request to obtain a corresponding HTTP / HTTPS request, and obtaining APP page data of related pages on a social platform; and 3) respectively carrying out page analysis and data alignment fusion on the Web page data and the APP page data, and outputting and storing the Web page data and the APP page data into a database. The invention discloses a social platform-oriented data acquisition system. The social platform-oriented data acquisition system comprises a Web data acquisition module, an APP data acquisition module and a double-terminal data alignment and fusion module. The method and the system provided by the invention are simultaneously suitable for social platform data acquisition of a Web end and an APP end, and full social platform data can be obtained.

Description

technical field [0001] The invention relates to the technical field of network data collection. More specifically, the present invention relates to a social platform-oriented data collection method and system. Background technique [0002] With the development of Internet technology, the Internet has become a new public opinion field, and new media terminals and self-media platforms emerge in endlessly. Website and APP development technologies are also diverse. The page development technologies and programming frameworks used by each platform are varied, and there is no uniform standard. Emerging media usually do not have collection interfaces that are directly exposed to the outside world. For individual platforms that have data interfaces, it is usually difficult to apply for interface permissions or the number of daily interface requests is limited. On the other hand, in terms of data loading, new media platforms generally adopt a dynamic asynchronous loading method to ...

AI Technical Summary

Problems solved by technology

The existing collection methods are oriented to web-side data collection, and cannot support requests for app-side data acquisition and web page asynchronous data loading, and are difficult to be compatible with changeable media

[0004] When the inventor researched the data collection technology of the social platform, he found that the existing technology is usually analyzed for a specific terminal platform to form a data traffic acquisition technology for a specific platform. Can use the web or app for data collection, but cannot obtain the full amount of platform data

Images