First-view video behavior prediction system and method

Abstract

The invention relates to a first-view video behavior prediction system, which is used for carrying out behavior prediction according to an existing video, and comprises a visual feature extraction module used for extracting visual features in the existing video; an intuition-based prediction module, which is used for carrying out intuition-based prediction according to the visual features extracted by the visual feature extraction module to obtain a first prediction result; an analysis and reasoning-based prediction module, which is used for performing analysis and reasoning-based prediction according to the visual features extracted by the visual feature extraction module to obtain a second prediction result; and an adaptive fusion module, which is used for organically fusing the first prediction result and the second prediction result by adopting an attention mechanism to obtain a final behavior prediction result. According to the method, prediction based on intuition and predictionbased on analytical reasoning are organically combined, the visual gap problem is effectively relieved, the next action can be predicted more directly and more accurately from the human psychologicalperspective, the accuracy is higher, and more comprehensive support is provided for actual engineering application.

Description

technical field [0001] The present invention relates to the field of video, in particular to the field of video behavior prediction, and more specifically, to a first-view video behavior prediction system and method. Background technique [0002] Human beings perceive and interact with the outside world through a self-centered perspective (that is, the first perspective). With the development of smart wearable devices, recording video data from the first perspective has more and more application scenarios, such as virtual reality, human-computer interaction, and so on. Since the first-person view can better help the intelligent system understand human intentions and purposes, it is particularly important to analyze the behavior in the first-person video, including behavior recognition (recognition of completed behaviors) and behavior prediction (recognition of behaviors that have not yet occurred). behavior to predict). Although the technology of behavior recognition is re...

AI Technical Summary

Problems solved by technology

Most of the existing methods do not effectively use information other than visual modalities (such as text modalities), and it is difficult to alleviate the "visual gap" problem

In addition, because it directly involves the understanding of human intentions, first-person behavior prediction needs to learn from human psychology or cognitive science. [4] It is also not considered by the existing technology to explore from different angles, which causes a large deviation between the predicted behavior and the actual behavior, and the prediction accuracy is not high.

Images