Time sequence action detection method and device, storage medium and terminal

Abstract

The invention discloses a time sequence action detection method and device, a storage medium and a terminal. The method comprises the following steps: acquiring an action video; inputting the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on training of a plurality of local-global fusion features, and the plurality of local-global fusion features are constructed according to a local self-attention module and a global self-attention module; and outputting the time sequence action information in the action video. According to the method, the self-attention module is adopted, information of long-time-sequence features extracted by a convolutional layer in a video can be reserved, global features can be captured on the long-time-sequence features, information loss is reduced, meanwhile, multiple local-global fusion features are constructed through the local self-attention module and the global self-attention module, and the fusion accuracy of the local-global fusion features is improved. Context information of a time sequence video can be fused and utilized, so that the utilization rate and the calculation efficiency of a GPU in a network training process based on a plurality of local-global fusion features are greatly improved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a time sequence action detection method, device, storage medium and terminal. Background technique [0002] With massive increases in online and personal media archives, people are generating, storing and consuming vast amounts of video. Under this trend, the development of efficient algorithms to intelligently parse video data is encouraged. One of the fundamental challenges for the success of these improvements is the temporal and spatial aspects of action detection in videos, ie, temporal action detection. The current mainstream temporal action detection network is mainly composed of a CNN network layer and a fully connected layer. The CNN network layer is mainly responsible for feature extraction, and the fully connected layer is mainly responsible for behavior classification. In general, the input shape size and output shape size of the CNN network layer and the ...

AI Technical Summary

Problems solved by technology

Although the self-attention module can focus on features in different ranges, extreme scale (duration) changes cause the network model to have a high probability of misalignment between the receptive domain range and the corresponding behavioral time boundary range, that is, the content and time within the model's perception range The content within the bounds is biased

In order to reduce deviation, the current mainstream multi-stage temporal action detection network first selects high-confidence behavior time boundaries (start time point and end time point, usually in seconds) through the nomination network, and then according to the selected behavior Time boundary, the content within the boundary range is intercepted at the video level, and then input to the detection network for boundary refinement and behavior classification, but this method requires a large amount of memory access, data exchange between video memory and memory, and even hard disk reading and writing, resulting in GPU utilization drops, calculation efficiency is not high

Images