Distributed online real-time processing method and system for multi-source and heterogeneous flow-state big data

Abstract

The invention provides a distributed online real-time processing method and system for multi-source and heterogeneous flow-state big data. The method specifically comprises the steps of crawling webpage data of each source by utilizing a distributed crawler duplicate removal algorithm; pre-processing the crawled page, constructing a corresponding tree by utilizing a visual page segmentation algorithm, pruning noise nodes according to a visual rule, classifying multi-layer pages, determining predicates under different types of pages according to different characteristics, and inferring data recording block nodes and data attribute nodes through the rule; Distributing the preprocessed data source by using a distributed message system, providing a data stream, and describing the state of a data node in the data stream to form state information; Selective storage operation is carried out on data streams by utilizing a Hadoop distributed file system based on K- And detecting the processed data by using a means text clustering method, determining a text similar to a predetermined sensitive information text, and screening out the sensitive information.

Description

Technical field [0001] The present disclosure relates to a method and system for distributed online real-time processing of multi-source, heterogeneous flow state big data. Background technique [0002] The statements in this section merely provide background information related to the present disclosure, and do not necessarily constitute prior art. [0003] The network technology revolution marked by the formation of the Internet has pushed human society into the era of information networking, forming a brand-new social life space-the network environment, which reflects all aspects of social life in real time. In the era of rapid development of the mobile network and the Internet, the high expansion of information has made the current security situation more complicated, and cyber warfare has become an important issue in the field of non-traditional social security. [0004] Because social networking sites such as forums, microblogs, blogs, private spaces, and Renren.com carry a la...

AI Technical Summary

Problems solved by technology

Whether it is the discovery of sensitive topics, the mining of criminal organization relationships, or the dissemination of rumors, current research relies on the accumulation and analysis of a certain amount of data, which belongs to post-event research and judgment. It plays a role in supporting and assisting decision-making in criminal activities and public opinion guidance. However, it is difficult to achieve real-time monitoring and early warning of social security.

Images