Neural network and tag library-based statement similarity algorithm

Abstract

The invention discloses a neural network and tag library-based statement similarity algorithm in the information retrieval field, which is characterized by comprising the following steps: (1) loading a semantic dictionary and a synonym lexicon with a neural network respectively; (2) inputting a complete statement to be analyzed; (3) analyzing the integral syntactic structure of the statement by using a dependency grammar analyzer, then layering the statement, and acquiring an effective component sequence of the statement; (4) determining a corresponding header field of the statement in an exUCL tag library according to the layering and the effective component sequence thereof; and (5) judging whether the statement has similar word pairs, if so, calculating the similarity of the statement, otherwise, re-inputting a new statement to be analyzed, and performing the similarity calculation again. The algorithm combines the advantages of dependency-based statement similarity algorithm and edit distance algorithm so that the calculation precision is greatly improved.

Description

technical field [0001] The invention relates to a sentence similarity algorithm, in particular to a sentence similarity algorithm based on a neural network and a tag library. Background technique [0002] In recent years, due to the continuous emergence of new network applications, especially the introduction and deepening of the Internet concept, network traffic and behavior have undergone great changes, shaking the traditional theoretical basis of the Internet, that is, the traffic model has changed from Poisson distribution to self-similar properties. The lack of accurate understanding and precise description of network traffic distribution, traffic characteristics, transmission efficiency, user and network behavior, etc., seriously affects the effective use of network resources and the development of the network itself, thus making the network controllable And manageability is getting worse and worse, there is a sharp contradiction between the quality of service provide...

AI Technical Summary

Problems solved by technology

Among them, the method based on the same vocabulary has obvious limitations: it can do nothing for the replacement of synonyms; the method of using semantic dictionaries can solve the problem of synonyms replacement well, but the method of simply using semantic dictionaries has no effect. Considering the internal structure of the sentence and the interaction between words, the accuracy rate is not high; the method of calculating the edit distance is usually used in the field of fast fuzzy matching of sentences, but the editing operation specified by it is not flexible enough, and it does not consider the same words. The method based on statistics needs to construct a large amount of training corpus, the workload is very huge, and there is also the problem of data sparseness

Images