A two-stream network action recognition method based on spatio-temporal saliency action attention

Abstract

A dual-stream network behavior recognition method based on spatiotemporal saliency behavior attention belongs to the field of machine vision. The method employs a spatio-temporal two-stream network-based network architecture called the spatiotemporal saliency behavioral attention network ST‑SAMANet. Aiming at the problem of large memory consumption and excessive redundant information caused by direct input of RGB frames and optical flow frames in the traditional dual-stream network, the present invention introduces a key frame mechanism to obtain the maximum difference between frames and significantly reduce the memory consumption of the temporal network. In addition, in the network, there are a large number of feature redundancy and background disturbance on the frame, which greatly affects the performance of the network. The present invention introduces Mask R-CNN technology into the network, pays close attention to the human bodies and objects in each behavior category, and performs feature extraction on the salient regions on each frame. Finally, the two-way LSTM and C3D network are used to encode space-time respectively to obtain perfect space-time information, which improves the robustness of the behavior recognition model.

Description

technical field [0001] The invention belongs to the field of machine vision, in particular to a dual-stream network behavior recognition method based on spatio-temporal salient behavior attention. Background technique [0002] With the extensive research of machine vision in theory and practice, behavior recognition has gradually become an important branch of it. Due to the diversity of objective environments and the subjective complexity of human behavior, there are still many problems to be solved in human behavior recognition. At present, behavior recognition is mainly divided into two types of methods based on static pictures and based on videos. For a long time before video research became popular, most behavior recognition research was based on images, but the information provided by images was very limited, and it was not easy to capture effective information for recognition. In comparison, for behavior recognition on videos, researchers can extract sufficient 'acti...

AI Technical Summary

Problems solved by technology

However, the traditional two-stream network still faces the following problems: (1) How to make full use of the temporal semantic information of consecutive frames under the premise of effectively controlling the model complexity

(2) The network directly extracts the features on each frame, and it is difficult to guarantee the activity and availability of effective information in the network

Images