Large scale text data external clustering method and system

Abstract

Disclosed are an external clustering method and a system for large-scale text data, applied to the information technical field; the designed clustering system comprises a candidate analyzer, a relationship generator, a relationship selection component and a clustering component; each sample point serves as a cluster; a candidate related candidate point cluster is selected for each sample through search technology; the relationship between the sample and the candidate relationship sample is calculated by the relationship generator and outputted to the external storage space by an increasing or decreasing order. The method mainly comprises the following steps: to pre-process the input text set and generate the inverted index and the eigenvector of the text set; to use the search technology to retrieve the candidate relationship set of each file; to calculate the relations of the files with candidate relationship through a relation calculation method; to orderly output the calculated results which are greater than a certain threshold value; then according to the orderly outputted results, the clustering algorithm repeatedly and iteratively combines a text pair with a first direct relationship, so as to eventually achieve the text set clustering output. The device has novel and scientific design, occupies small space in the clustering process and uses large-capacity external memories to make differentiated treatment to the treatment process.

Description

Technical field: [0001] The invention relates to a method and implementation of large-scale text data clustering using external memory in the field of information technology, an external document clustering method based on retrieval technology, and a large-scale method that overcomes the shortcomings of existing methods in processing problem scale and time. External clustering method and system for text data. Background technique: [0002] In the past 10 years, information-based organizations or knowledge-based enterprises have been full of vitality, creating and disseminating knowledge has become a key element to test the core competence of enterprises, and the ability to create and apply knowledge has become an uncompromising force for the core competitiveness of enterprises. support. Knowledge is not only stored in the brains of employees, but also rooted in various documents accumulated by the enterprise for a long time, as well as in the data of other application syste...

AI Technical Summary

Problems solved by technology

However, in order to ensure the performance of the system, the selected representative points cannot be too small, so in essence, the above method still does not solve the incomputable problem caused by the space complexity of the processing scale

Images