Fight recognition method and device based on dual-channel cross-attention mechanism

Abstract

The invention discloses a fight recognition method and device based on a dual-channel cross-attention mechanism. The method first collects and generates video data sets, sets two frame rates of fast and slow to obtain different frame image sequences, and sends them to the fast and slow channels respectively after preprocessing. The fast and slow channel adopts the Transformer encoder based on the separate spatiotemporal self-attention mechanism to extract the spatiotemporal encoding features of the image sequence; then, through the cross-attention module, the CLS token of one channel and the patch token information of the other channel are fused to realize the fusion of dual-channel spatiotemporal encoding features; finally , the fused spatio-temporal coding features are passed through the multi-layer perception machine head for fighting behavior recognition. The invention can effectively extract the spatio-temporal features of the video through the dual-channel Transformer model and the cross-attention module, improve the accuracy of fighting behavior recognition, and is suitable for indoor and outdoor monitoring systems.

Description

technical field [0001] The invention relates to the field of intelligent video monitoring and deep learning, in particular to a transformer fight recognition method and device based on a dual-channel cross-attention mechanism. Background technique [0002] Nowadays, surveillance video has been widely used in social public places and plays an extremely important role in maintaining social public safety. Effective identification of abnormal behaviors and events in surveillance video can better play the role of surveillance video. Fighting is a common behavior in the video that disturbs public order. In severe cases, it is suspected of gathering crowds to fight and picking quarrels and provoking trouble, which affects social stability. It is very important to maintain social stability to detect fighting behaviors in a timely manner through intelligent means in massive videos. [0003] Most of the existing video-based fighting behavior recognition methods are implemented by con...

AI Technical Summary

Problems solved by technology

[0006] The disadvantage of the network based on optical flow image input is that optical flow extraction takes a long time, consumes a lot of resources, and the entire network cannot be end-to-end.

The network based on RGB images as input does not set different number of image frames for input according to the different characteristics of the video in the time and space domains. If the number of input frames is large, the resource consumption will be large, and the algorithm training and testing time will be longer. If it passes If the number of sparse image frames is sampled, time-domain features cannot be extracted well, because behaviors and actions vary greatly between frames, and sparse sampling will affect spatial feature extraction

[0007] Based on the convolutional neural network to capture short-distance spatiotemporal information through the convolution kernel, it cannot model dependencies beyond the range of the receptive field

Although the deepening of the network can expand the receptive field and solve this problem to a certain extent, the problem still exists. There are ways to consider integrating local information into the attention mechanism, but the fusion method of local information and global information is relatively simple, which makes the feature fusion not comprehensive enough. Not strong enough

Images