Behavior recognition method based on multi-modal data fusion

Abstract

The invention discloses a behavior recognition method based on multi-modal data fusion. The method is applied to depth video and skeleton sequence data. For depth video data, the method comprises the following steps: firstly, extracting a DMI depth feature map on a video sequence; then, dividing the original depth action sequence into two segments of subsequences with the same length according to the frame number, and respectively extracting sub DMI depth feature maps on the subsequences; and forming a space-time depth motion graph by using the two obtained sub-DMI feature graphs and the DMI image, and extracting the depth features of the motion by using an HOG (Histogram of Oriented Gradient) algorithm. For skeleton sequence data, a space-time diagram convolution feature extractor improved based on a space-time diagram convolution network model is used to directly process a skeleton sequence, and skeleton features of actions are extracted. After action features on two data modalities are obtained, a high-credibility mean value sample fusion algorithm based on CCA improvement is used for fusing the two types of features, and fused features are obtained. And finally, the fusion features are classified by using an SVM. According to the method, an existing behavior recognition algorithm model is improved, the influence of single-mode data and noise samples on an algorithm experiment is overcome, and the recognition accuracy of a behavior recognition algorithm on an existing public data set is improved.

Description

technical field [0001] The present patent application relates to a behavior recognition method based on multimodal data fusion, and belongs to the field of computer vision. Background technique [0002] In recent years, artificial intelligence has developed rapidly, and technologies such as machine vision, pattern recognition, and natural language processing have been widely used in many aspects of social development. The fields involved include intelligent manufacturing, autonomous driving, intelligent robots, etc. Machine vision plays a pivotal role in artificial intelligence. Machine vision is committed to using computers to achieve the role of biological vision, which greatly promotes the development of artificial intelligence. Machine vision is a comprehensive technology, including computer software technology, sensors, optical imaging, image processing, video processing, electric light source lighting, etc. The computer realizes the perception and understanding of t...

AI Technical Summary

Problems solved by technology

However, due to the bottleneck in the research of single-modal human behavior recognition, and the side effects caused by noise samples to the experiment, the accuracy of behavior recognition is high.

Images