Cross-modal retrieval method for querying video from complex text based on semantic tree enhancement

Abstract

The invention discloses a cross-modal retrieval method for querying a video from a complex text based on semantic tree enhancement. For complex text query statements, words of complex text query statements are converted into leaf node representations, the relationship between child nodes is mined, the two child nodes with the highest dependency are combined, a semantic tree structure of the querystatements is constructed in a recursion mode, and query representations based on semantic tree enhancement are obtained. For coding of candidate videos, video preliminary features are obtained through a CNN, time dependence and semantic correlation between the videos are captured through a GRU and a self-attention mechanism module, and robust video feature representation is obtained. The complextext query representation and the video feature representation are mapped into a public space, and a matching relationship between the complex text query representation and the video feature representation is is automatically learned, thereby realizing cross-modal retrieval from complex text query to video. Information components in the complex text query statements can be explained, the user intention can be better understood, and the retrieval performance is improved to a great extent.

Description

technical field [0001] The invention relates to the field of cross-modal retrieval from text query to video, in particular to a cross-modal retrieval method from complex text query to video based on semantic tree enhancement. Background technique [0002] With the exponential growth of user-generated videos on the Internet, uploading videos in daily life and searching for videos of interest have become indispensable activities in people's daily life. The cross-modal retrieval method from text query to video is one of the techniques to obtain interesting videos. Early cross-modal retrieval methods from text query to video are mainly based on text keywords, and have been extensively researched and developed. But such methods only allow the user to input several keywords as queries. With the further improvement of people's demand for Internet video search capabilities, keyword-based queries are difficult to fully express users' search intentions, thereby affecting search expe...

AI Technical Summary

Problems solved by technology

However, this kind of method has the following shortcomings: First, it is usually not very effective for complex text queries, because it is usually difficult to fully describe the semantic content of complex text queries through a number of visual concepts, resulting in information loss, and the semantic content of complex text queries Not just aggregations for extracting concepts

2. How to effectively train a concept classifier and select related concepts is also a very challenging problem

Although this type of direction can better handle longer text queries than concept-based methods, this type of method has the following shortcomings: First, the user's text query is represented by a word vector, which cannot effectively understand the user's intent. Therefore, the video retrieval effect is not good for complex text queries

2. This type of method lacks the interpretability of the sub-retrieval process

Images