Space-time attention based video classification method

Abstract

The invention relates to a space-time attention based video classification method, which comprises the steps of extracting frames and optical flows for training video and video to be predicted, and stacking a plurality of optical flows into a multi-channel image; building a space-time attention model, wherein the space-time attention model comprises a space-domain attention network, a time-domain attention network and a connection network; training the three components of the space-time attention model in a joint manner so as to enable the effects of the space-domain attention and the time-domain attention to be simultaneously improved and obtain a space-time attention model capable of accurately modeling the space-domain saliency and the time-domain saliency and being applicable to video classification; extracting the space-domain saliency and the time-domain saliency for the frames and optical flows of the video to be predicted by using the space-time attention model obtained by learning, performing prediction, and integrating prediction scores of the frames and the optical flows to obtain a final semantic category of the video to be predicted. According to the space-time attention based video classification method, modeling can be performing on the space-domain attention and the time-domain attention simultaneously, and the cooperative performance can be sufficiently utilized through joint training, thereby learning more accurate space-domain saliency and time-domain saliency, and thus improving the accuracy of video classification.

Description

technical field [0001] The invention relates to the technical field of video classification, in particular to a video classification method based on spatio-temporal attention. Background technique [0002] With the widespread popularization and rapid development of social media and self-media, the number of videos on the Internet is showing a trend of rapid growth. Research shows that more than 300 hours of videos were uploaded to YouTube every minute in 2016. The 2016 video traffic statistics and forecast report of CISCO in the United States further pointed out that global video traffic will account for 82% of Internet traffic in 2020. At that time, it will take a user five million years to watch all the videos transmitted on the Internet within one month. video. Video and other media data have become the main body of big data. How to accurately analyze and identify video content is of great significance for meeting users' information acquisition needs. [0003] Video cl...

AI Technical Summary

Problems solved by technology

However, existing deep video classification methods cannot simultaneously model spatial and temporal saliency in videos, ignore the connection between these two saliencies, and thus cannot make full use of both saliency to learn more effective video features , which limits the effect of video classification

Images