Noise-robust audio and video bimodal speech recognition method and system

Abstract

The invention provides a noise-robust audio and video bimodal speech recognition method and a system. The noise-robust audio and video bimodal speech recognition method comprises the steps of: extracting a Mel-frequency cepstrum coefficient of an audio and a first-order second-order dynamic coefficient of the Mel-frequency cepstrum coefficient as audio features, carrying out the video framing, intercepting a fixed lip region through face detection and alignment, and transmitting the fixed lip region to a residual network, so as to obtain video features; and introducing attention mechanisms toalign and correct the feature information of a high-level network of the audio and video to obtain feature representation of the fused audio and video, realizing the early-stage fusion of the features, and realizing the later-stage fusion of the features through the two independent attention mechanisms of audio and video, and then decoding and outputting a recognition result. According to the noise-robust audio and video bimodal speech recognition method, the recognition accuracy rate is obviously improved under the condition of low signal-to-noise ratio noise. As further improvement and optimization, modal attention is added during later-stage fusion of the features, weights are given to the audio and video features in a self-adaptive mode according to the information content of the audioand video modals, then feature fusion is conducted, and the model is more stable.

Description

technical field [0001] The disclosure belongs to the technical field of speech recognition, and relates to a noise-robust audio-video dual-mode speech recognition method and system. Background technique [0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art. [0003] In the process of communication, human beings mobilize the use of multiple senses, not only relying on hearing to understand each other, but also visual feelings can help people communicate better, especially in noisy environments, when we can’t hear clearly When listening to the other party's language, by observing the other party's body language, facial expressions, and lip movements, you can help understand the other party's words. In the technical research of speech recognition, the influence of visual factors has also been considered, in order to improve speech recognition while being close to reality. Spe...

AI Technical Summary

Problems solved by technology

First, due to the diversification of visual information and features, which visual features match the Mel frequency cepstral coefficients of acoustic speech is a difficult point. Second, the fusion of audio and video data streams running at different frame rates is a difficult problem, because The accuracy of lip language recognition is much lower than that of speech recognition in most cases, and improper fusion may even reduce the original results of speech recognition

Images