Continuous sign language recognition method based on cross-modal data augmentation

Abstract

The invention discloses a continuous sign language recognition method based on cross-modal data augmentation, and the method comprises the steps: carrying out random deletion, insertion and replacement of original video text data, generating a series of pseudo video text data with marks, amplifying a conventional data set, and achieving a purpose of enlarging the data scale. Based on original dataand augmented data, a brand-new multi-objective optimization function is designed, so that the cross-modal distance between a video and a corresponding text is reduced while weak supervision video text alignment learning is carried out, and meanwhile, a network can distinguish the difference between real data and augmented pseudo data. Through cross-modal data augmentation and multi-task learning, the continuous sign language recognition performance is improved.

Description

technical field [0001] The invention relates to the technical field of action recognition in computer vision, in particular to a continuous sign language recognition method based on cross-modal data augmentation. Background technique [0002] Continuous sign language recognition aims to convert an input sign language video into a sequence of sign language words in a consistent order. For the video sign language recognition method, the visual encoder first converts the input video into a high-dimensional feature representation, and then the sequence model learns the mapping from the feature representation to the corresponding text sequence. [0003] 1) Video representation learning. [0004] Discriminative video feature representation plays a very important role in sign language recognition. Early work focused on handcrafted features such as HOG or HOG-3D, motion trajectories, and SIFT. These features are used to describe hand shape, hand orientation and motion state. Wit...

AI Technical Summary

Problems solved by technology

Due to the inconsistency between the CTC objective function and the evaluation standard, the candidate sequence with the highest probability cannot be guaranteed to be optimal under the evaluation index of the word error rate, resulting in the inability to accurately select the optimal candidate sequence, resulting in a decline in system recognition performance

[0009] 2) The continuous sign language recognition algorithm based on deep neural network relies on large-scale labeling of sign language data, and sign language data needs to be labeled by professional sign language practitioners. The labeling process is complicated and expensive, so the scale of existing sign language data sets is limited. Conducive to the training of deep neural networks for sign language recognition, it is urgent to propose data augmentation algorithms and recognition frameworks based on existing data sets

Images