A sign language recognition method based on multi-source information fusion

Abstract

The invention discloses a sign language recognition method based on multi-source information fusion, in which a data collection model, a data preprocessing model, a neural network model and a real-time identification model are used. The invention adopts the multi-sensor fusion technology and the depth learning method to recognize the sign language, and sends out the sound corresponding to the signlanguage action through the loudspeaker. The process is as follows: 1. collecting sign language motion information through surface electromyography (sEMG) sensor and inertial measurement unit (IMU),and transmitting data through Bluetooth; 2, cleaning the collected data by feature extraction and denoising algorithm, and processing the data into a data format that can be input into a neural network; 3, constructing a bi-directional double-layer LSTM neural network and training a preservation model; 4, transplantting the model to a mobile phone, cutting the real-time motion data, inputting theprocessed motion data into the model to obtain a label correspond to the sign language motion, and using the open source language library to send out the speech corresponding to the motion label.

Description

technical field [0001] The invention belongs to the technical field of computer science time-frequency signal processing and cyclic neural network model construction, and in particular relates to a sign language recognition method based on multi-source information fusion. Background technique [0002] The deaf-mute is an undisputed vulnerable group that needs social attention. Most of today's deaf-mute communication methods rely on sign language, which really builds a bridge of communication between deaf-mute and deaf-mute, deaf-mute and people who know sign language. However, most ordinary people are not familiar with sign language, which causes most people to be unable to communicate with deaf-mute people normally. discrimination. So our team came up with the idea to help deaf people communicate with the majority who don't understand sign language. [0003] And a current hot technology: wearable technology, has attracted our attention. Wearable technology mainly refers...

AI Technical Summary

Problems solved by technology

[0006] 1) Data gloves are too bulky and poor in portability, and will attract too much attention, which will cause psychological pressure for the aphasia;

[0007] 2) The occlusion between gestures makes it difficult to accurately identify finger changes, and the wearer cannot use it in darker scenes (night, indoor light is insufficient), and it is difficult to accurately detect, track and cut hands, which limits the real-time performance of sign language recognition;

[0008] 3) The distance is too short and the change is too small. Whether it is movable or non-movable recognition, it is difficult to detect accurately, and the portability is very poor;

[0009] 4) It is difficult to ensure the integrity of finger language and sign language recognition, and the combination of coarse-grained and fine-grained sign language recognition cannot be considered

Images