Scene matching fused Chinese sign language translation model construction method and device

Abstract

The invention discloses a scene matching fused Chinese sign language translation model construction method and device. The method comprises the steps of S1, constructing a sign language word recognition model of a mapping relationship between sign language actions and words in different scenes, and constructing a scene matching model of a mapping relationship between different scene vocabularies and sign language actions; s2, respectively training the sign language word recognition model and the scene matching model to obtain a trained sign language word recognition model and a scene matching model; s3, cascading the trained sign language word recognition model and the scene matching model to form a Chinese sign language translation model, acquiring a sign language action data set, and respectively inputting the sign language action data set into the sign language word recognition model and the scene matching model of the Chinese sign language translation model to update the models, and obtaining a final Chinese sign language translation model until the sign language word recognition model and the scene matching model reach dynamic balance. The method and device have the advantages of simple implementation method, high construction efficiency, high accuracy and the like.

Description

technical field [0001] The invention relates to the technical field of Chinese sign language translation, in particular to a method and device for constructing a Chinese sign language translation model integrated with scene matching. Background technique [0002] At present, sign language translation mainly obtains the user's body characteristics through wearable devices or image sensing devices. Ways to analyze the meaning of user sign language. It has been tested that wearable devices have problems such as high cost, poor portability, and uncertain disturbances to user behavior in the process of expressing sign language. At present, images or videos are usually collected by using image sensing devices. [0003] With the development of deep learning, the use of neural networks can mine deeper and more abstract features in images, and establish a more relevant feature-to-sign language text mapping. For example, use AlexNet to extract features from the input image, use the ...

AI Technical Summary

Problems solved by technology

[0004] For the recognition of natural scenes, at present, the scene classification task is usually simply set as a single-label class, and the neural network is used to identify and classify the scene driven by massive label data. However, real scenes usually contain multiple label information, and these labels It may correspond to objects and objects in different scenes, which will lead to misjudgment of the scene classification. If the scene classification model is directly introduced on the basis of the translation model, that is, an additional scene classification model is established, and the classification results of the scene classification model are directly used for classification. Translation, if the accuracy of the scene classification model is not high, it is very easy to cause inaccurate translation due to the wrong classification of the scene classification model

[0005] To sum up, the current research on Chinese sign language translation tasks is still in the initial stage from the extraction of sign language behavior features to the mapping of sign language texts, and no further exploration has been made on the impact of sign language polysemy in different scenarios and contexts. At the same time, the current Chinese sign language translation fails to effectively use the sign language scene to re-optimize the translation results, which also limits the improvement of the accuracy of Chinese sign language translation, and the direct introduction of scene classification has the problem of being affected by the classification accuracy of the scene classification model

Images