Sentence-level lip language recognition method based on channel attention and time convolutional network

Abstract

The invention discloses a sentence-level lip language recognition method based on channel attention and a time convolutional network, which comprises the following steps: 1, downloading a data set GRID for training a model and a more challenging CMLR, and preprocessing the data set; 2, building a lip language recognition network disclosed by the invention, sending the preprocessed data set into the network for training, and adjusting network parameters to obtain an optimal lip language recognition network model; and 4, performing lip language recognition on the video by using the trained model. According to the method, multi-scale feature extraction of the lip language video in the time domain and the space domain is carried out, and a high-quality lip language recognition feature map is obtained in combination with an attention mechanism, so that the lip language recognition accuracy can be improved, and the corresponding evaluation index on a more challenging CMLR data set is excellent.

Description

technical field [0001] The invention belongs to the technical field of computer machine learning and artificial intelligence, and mainly relates to a deep neural network lip language recognition method. Background technique [0002] Lip reading plays a vital role in human communication and speech comprehension, and studies have shown that humans have poor lip reading ability, while hearing-impaired people can only get less than 30% accuracy. Therefore, good lip language recognition technology can be used to improve hearing aids, improve the acquisition of language information in silent, safe, and noisy environments, etc. Before the advent of deep learning, most of the work in lip reading was based on hand-designed feature learning, which is computationally expensive and low in accuracy. In recent years, with the progress and development of deep learning, lip language recognition methods based on deep learning have received extensive attention. The use of deep neural network...

AI Technical Summary

Problems solved by technology

For CMLR, due to the diversity of Chinese grammatical structure, its sentence structure is not fixed, the length of sentences is different, and the frequency of occurrence of each Chinese character is also different, so the CMLR data set is more challenging.

Images