Unified Chinese-English mixed text generation and speech recognition end-to-end framework

Abstract

The invention provides a universal unified Chinese-English mixed text generation and speech recognition end-to-end framework. The universal unified Chinese-English mixed text generation and speech recognition end-to-end framework comprises an acoustic encoder, a phoneme encoder, a discriminator and a decoder, the phoneme encoder and the discriminator form a generative adversarial network, the phoneme encoder serves as a generator of the generative adversarial network, the discriminator serves as a discriminator of the generative adversarial network, and the acoustic encoder serves as real data input of the generative adversarial network, the generative adversarial network is used for promoting the distribution of phoneme coding representations output by a phoneme encoder to be close to acoustic coding representations output by an acoustic encoder, and the decoder fuses the acoustic coding representations and the phoneme coding representations to obtain decoding representations, and inputs the decoding representation into a softmax function to obtain an output target with the maximum probability.

Description

technical field [0001] This application relates to the field of speech recognition, in particular to an end-to-end framework for unifying Chinese-English mixed text generation and speech recognition. Background technique [0002] The Chinese-English mixed phenomenon refers to the inclusion of both Chinese and English expressions in the speaking process, mainly including two types of inter-sentence conversion and intra-sentence conversion. Among them, the phenomenon of intra-sentence conversion has brought great challenges to speech recognition technology. The main problems are accent problems caused by non-standard pronunciation of speakers; more and more complex modeling units; collaborative pronunciation of different languages; difficulties in data collection; difficulties in data labeling, etc. With the development of deep learning technology, monolingual speech recognition technology has been greatly improved. Especially for the end-to-end speech recognition model, its...

AI Technical Summary

Problems solved by technology

In this way, although the speech recognition model training data can be obtained, the synthetic data does not match the real data. How to use the synthetic data to improve the performance of the recognition system is a challenging problem.

Images