A method and system for real-time collection of public opinion data based on distribution

Abstract

The invention discloses a public sentiment data real-time collecting method and system based on distribution. The method comprises the following steps that firstly, a public sentiment data website class library is established, and crawling data items are classified and defined; secondly, a data acquisition website list is transmitted to a data collection server, the server allocates corresponding crawlers to circularly crawl data in a dormant mode; thirdly, crawled source webpage data is subjected to label analysis, the position of a target data item is positioned, and the target data item is obtained; fourthly, an obtained result data item is encapsulated into a uniform format of a corresponding class; fifthly, the encapsulated data is stored into a corresponding database; sixthly, a monitoring log file is generated. According to the public sentiment data real-time collecting method and system based on distribution, the architecture is advanced, a factory mode is used as a main design mode of the system, new examples can be quickly generated, system core functions such as browser assess, log generating, data encapsulation, agent setting and queue setting are encapsulated, the expandability and transportability of the system are enhanced, and the reusability of codes and the maintainability of the system are improved.

Description

technical field [0001] The present invention relates to a method and system for distributed and highly concurrent collection of Internet public opinion data, in particular to a method and system for realizing high-efficiency and real-time collection of target data, and especially to a method and system for real-time collection of public opinion data based on distribution. Background technique [0002] Public opinion refers to the social and political attitudes of the people towards social managers in a certain social space around the occurrence, development and changes of intermediary social events. At present, the Internet has become one of the main carriers to reflect public opinion, and plays an important role in the dissemination of public opinion. The outbreak of public opinion crisis makes more and more people pay attention to the emergence and development of public opinion. Party and government organs at all levels must keep abreast of information, strengthen monitor...

AI Technical Summary

Problems solved by technology

However, these general search engines also have certain limitations. 1. The results returned by the general search engines include a large number of web pages that users do not care about.

General search engines are often powerless to data with dense information content and a certain structure, and cannot discover and obtain key information well

2. The simple Internet-based data acquisition system has a single acquisition method and cannot perform multiple tasks at the same time, which leads to low data acquisition efficiency and cannot meet the real-time nature of the data

3. In other current public opinion analysis and detection systems, most of the public opinion data adopts an offline processing mechanism, which inevitably leads to a certain time delay in the data in terms of structure

Images