Data processing method and system and related device

Abstract

The embodiment of the invention discloses a data processing method, a data processing device and a related device, for increasing the parameter solving speed and the parameter solving accuracy of an hLDA (hierarchical latent Dirichlet allocation) model. The method disclosed by the embodiment of the invention comprises the following steps of: sending the global initial global statistical information to various slave nodes, merging the local statistical information received from the slave nodes to obtain the new global statistical information, calculating the probability distribution between documents and themes and the probability distribution between themes and words according to the new global statistical information if the Gibbs sampling performed by the slave nodes is over, establishing and maximizing a likelihood function of a text set according to the calculated probability distributions to obtain new hLDA hyper-parameters, and calculating and outputting the probability distribution between the documents and the themes and the probability distribution between the themes and the words according to the new hLDA hyper-parameters if the iteration for solving the hLDA hyper-parameters is converged.

Description

technical field [0001] The present invention relates to the technical field of information retrieval, in particular to a data processing method, system and related devices. Background technique [0002] Information Retrieval (Information Retrieval) refers to the process and technology of organizing and storing information in a certain way, and finding relevant information according to the needs of information users. In the narrow sense, information retrieval only refers to the process of finding out the required information from the information collection, which is equivalent to what people call information query. Today, with the rapid development of the Internet, the information on the Internet is increasing exponentially. Facing such a massive amount of information resources, how to efficiently and quickly obtain the information they need is becoming more and more important to people. In order to improve the quality and efficiency of user information retrieval, a powerful...

AI Technical Summary

Problems solved by technology

In order to improve the quality and efficiency of user information retrieval, a powerful information retrieval tool - search engine can be used, but while search engine brings great convenience to people, it also exposes the search technology with keywords as the basic index unit. There are many shortcomings: on the one hand, no matter what keywords the user submits, too many results will be returned, and the information that the user really needs often only accounts for a small part, and the user has to spend a considerable amount of time manually filtering these results ; On the other hand, due to synonyms and synonyms, many texts related to the search topic do not exactly match the keywords entered by the user, so that the search engine cannot find these texts

[0004] However, in the prior art, since the hyperparameters of the hLDA model are regarded as invariants, the maximum approximate solution cannot be reached during the solution process, and the finally obtained parameters hLDA model hyperparameters have low accuracy and slow solution speed

Images