Topic detecting device and topic detecting method based on distributed multistage cluster

Abstract

The invention discloses a topic detecting device and a topic detecting method based on distributed multistage cluster. The topic detecting device mainly comprises a news acquiring module, a news classifying module, a topic detecting module, a topic integrating module and a topic displaying module. The topic detecting method is characterized by comprising the steps of A, acquiring news; B, classifying the newly acquired news; C, performing multistage cluster for various channels concurrently; and D, computing hotness of all topics, and screening hot topics of a total system and hot topics in each channel. By the topic detecting device and the topic detecting method, a sharp contradiction between a detecting effect and time cost in topic detection under the condition of quick updating of a large number of files in an internet environment can be solved.

Description

technical field [0001] The invention relates to network information analysis, text classification and text clustering technologies in text information processing, in particular to a topic detection device and method based on distributed multi-level clustering. Background technique [0002] With the rapid development of the Internet, the information on the Internet has become more and more diverse and abundant. At the same time, the social influence of Internet public opinion has continued to increase. Many hot social events are disclosed and disseminated in the Internet at the first time. Therefore, its important value is becoming more and more obvious. In the Internet environment, there are a large number of documents in the form of natural language, including news, blogs, forum posts, and emerging microblogs, etc. These documents provide the most basic data source for discovering hot topics. [0003] The Topic Detection and Tracking project (TDT, Topic Detection and Track...

AI Technical Summary

Problems solved by technology

[0004] However, since these methods process each document in a serial manner, once faced with a large amount of data, it is difficult to guarantee a practical processing if the clustering algorithm is not selected with a less complex but less effective algorithm. speed

Moreover, since no measures to effectively reduce noise have been taken in advance, the detection effect needs to be improved.

In short, since a large number of new documents are generated in the network environment all the time, how to analyze them effectively and quickly to detect hot topics poses a great challenge to the existing technology

Images