Super-joint and multi-modal network and behavior identification method thereof

Abstract

The invention relates to the technical field of neural networks, in particular to a hyper-articular and multi-modal network and a behavior recognition method thereof, and the method comprises the steps: collecting a human body depth map, carrying out the feature extraction of the depth map through a DMMS flow, and calculating a depth data prediction score; the method comprises the following steps: collecting a human skeleton sequence, respectively extracting original joint and hyper-joint data, constructing skeleton information by combining hyper-joints and common joints, sending the skeleton information into a structured spatial-temporal feature learning model to obtain static and dynamic joint data streams and static and dynamic hyper-joint data streams, and carrying out adaptive weight fusion on the data streams of the original joints and the hyper-joints to obtain a structured spatial-temporal feature learning model; respectively obtaining a joint data prediction score and a super joint prediction score; and adding the classification prediction scores of the DMMS stream and the skeleton stream to generate a final prediction score. According to the method, rich texture information of a human body part in space is learned from a depth map, and rich spatial-temporal characteristics in motion posture changes are learned from a skeleton sequence.

Description

technical field [0001] The invention relates to the technical field of neural networks, in particular to a hyper-joint and multi-modal network and a method for recognizing it in action. Background technique [0002] Human action recognition has become a hot topic in the field of machine learning and computer vision because of its broad application prospects, and has very important theoretical research value. Different from the recognition of static images, behavior recognition should comprehensively consider continuously changing images, and construct the spatiotemporal feature representation of human behavior from a series of motion sequences composed of static human pose images to realize the classification and recognition of actions. At present, human behavior recognition has been widely used in assisting human-computer interaction, sports analysis, intelligent monitoring and virtual reality and other fields. [0003] At present, behavior recognition algorithms are mainl...

AI Technical Summary

Problems solved by technology

Although the skeleton joints are specially designed to capture the spatial structure information of the human body, the information carried by the joint points can only represent the position but not the dependency with the adjacent points.

Previous methods usually use the position of joints as the spatial structure information representation of joints, while ignoring the dependencies between joints

Images