Video semantic representation method and system based on multi-mode fusion mechanism and medium

Abstract

The invention discloses a video semantic representation method and system based on a multi-mode fusion mechanism and a medium. Feature extraction: extracting visual features, voice features, motion features, text features and domain features of a video; Feature fusion: performing feature fusion on the extracted visual, voice, motion and text features and domain features through the constructed multi-level hidden Dirichlet distribution topic model; And feature mapping: mapping the fused features to a high-level semantic space to obtain a fused feature representation sequence. The model utilizesthe unique advantages of the theme model in the semantic analysis field, and the video representation mode obtained through model training on the basis of the model has ideal distinction in the semantic space.

Description

technical field [0001] The present disclosure relates to a video semantic representation method, system and medium based on a multimodal fusion mechanism. Background technique [0002] With the explosive growth of data volume in the Internet era, the arrival of the era of media big data has been accelerated. Among them, video, as an important carrier of multimedia information, is closely related to people's life. The evolution of massive data not only requires a great change in the way data is processed, but also poses great challenges to the storage, processing and application of video. An urgent problem to be solved is how to organize and manage data effectively. With the continuous generation of data, due to the limitation of hardware conditions, the data can only be stored in segments or time-sharing, which will inevitably cause different degrees of information loss. Therefore, providing a simple and efficient data representation method for video is meaningful for vid...

AI Technical Summary

Problems solved by technology

The diversity of video content and the difference and ambiguity of understanding video content make it difficult to extract features for representing video data, which in turn makes video understanding based on semantic information more challenging.

[0004] Traditional data representation methods, such as vision-based video feature learning methods, can obtain concise representations of videos. However, in order to reasonably construct good features, certain experience and professional domain characteristics are required.

The application of deep learning methods has made remarkable progress in vision tasks, but there are still problems such as "semantic gap" and "multimodal heterogeneous gap"

Images