Chinese lip language recognition method and device based on hybrid convolutional neural network

Abstract

The invention discloses a Chinese lip language recognition method and device based on a hybrid convolutional neural network, and belongs to the field of machine vision and deep learning. The method comprises the following steps: acquiring the facial image information of a speaker through a camera; detecting and cutting a lip image sequence from the face image information by utilizing a face detector; carrying out the lip feature extraction on the lip image sequence by using a hybrid convolutional neural network; inputting the lip features into a Bi-GRU model, obtaining an identification probability result of the phoneme unit; inputting an identification probability result of the phoneme unit into a connection time sequence classifier CTC; obtaining a phoneme unit classification result; processing the classification result of the phoneme units by adopting a decoding method of introducing an attention mechanism. According to the method, the problem that an existing network framework cannot recognize graphic language characters such as Chinese is solved, possibility is provided for application of a lip language recognition technology in an actual scene, and the method can be widely popularized in the field of computer vision.

Description

technical field [0001] The present invention relates to the field of machine vision and deep learning, in particular to a Chinese lip language recognition method and device based on a hybrid convolutional neural network. Background technique [0002] With the development of artificial intelligence technology and the improvement of security awareness, voice interaction and identity recognition have become a widely used technology. However, these technologies still have some disadvantages. For example, voice interaction is easily affected by the environment, and noise interference is prone to occur, resulting in inaccurate voice recognition. Static identification technology is easy to be copied and imitated, resulting in the leakage of personal information and the theft of identity authentication information. In order to improve inaccurate speech recognition and enhance dynamic identity authentication technology, lip language recognition technology has emerged. [0003] Lip ...

AI Technical Summary

Problems solved by technology

However, these mature and excellent network architectures can only recognize English lip language. Due to the differences between graphic language characters such as Chinese and alphabetic languages ​​​​such as English, the above network architecture cannot be applied to Chinese lip language recognition.

Images