News sentence clustering method, device and storage medium based on semantic similarity

Abstract

The invention provides a method for clustering news sentences based on semantic similarity, the method comprising the following steps: preprocessing the news sentences of the corpus to extract available words; using the available words to train the continuous bag-of-words model, Obtain the initial word vector of each available word; Utilize the initial sentence vector of each news sentence and the initial word vector of the left and right adjacent available words of a certain available word in this news sentence to iteratively train the described continuous word bag model, obtain the The current word vector of each available word in the news sentence and the final sentence vector of the news sentence; the average value of the word vectors of all available words of each news sentence, the one-hot vector of high-frequency words and the final sentence vector are merged, The semantic vector of the news sentence is obtained; the distance between the semantic vectors is calculated to obtain the semantic similarity between different news sentences, and the news sentences of the corpus are clustered accordingly. The invention also provides an electronic device and a computer-readable storage medium.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a news sentence clustering method, device and storage medium based on semantic similarity. Background technique [0002] With the development of Internet technology, the amount of information owned by humans has shown explosive growth, and the amount of news text data has also increased rapidly. In the face of a huge amount of news corpus, clustering sentences can facilitate the inductive analysis of similar news and realize the comprehensive utilization of news data. [0003] The current commonly used news clustering methods often ignore the importance of word elements when calculating and comparing sentence similarities. The existence of a large number of polysemous words makes it easy for these clustering methods to gather different types of information together. There are also some news clustering methods that take into account the role of semantic information behind the s...

AI Technical Summary

Problems solved by technology

[0003] The current commonly used news clustering methods often ignore the importance of word elements when calculating and comparing sentence similarity. The existence of a large number of polysemous words makes it easy for these clustering methods to gather different types of information together.

There are also some news clustering methods that take into account the role of semantic information behind the surface text information of keywords, and use the knowledge platform with rich semantic concepts as an intermediate reference space to calculate the similarity of news sentences, but it is difficult to overcome the gap between clustering quality and efficiency. contradiction

Images