Cold fusing sequence-to-sequence models with language models

Abstract

Described herein are systems and methods for generating natural language sentences with Sequence-to-sequence (Seq2Seq) models with attention. The Seq2Seq models may be implemented in applications, such as machine translation, image captioning, and speech recognition. Performance has further been improved by leveraging unlabeled data, often in the form of a language models. Disclosed herein are Cold Fusion architecture embodiments that leverage a pre-trained language model during training. The Seq2Seq models with Cold Fusion embodiments are able to better utilize language information enjoying faster convergence, better generalization, and almost complete transfer to a new domain while using less labeled training data.

Description

technical field [0001] The present disclosure relates generally to systems and methods for computer learning that can provide improved computer performance, features, and use. Background technique [0002] Sequence-to-sequence (Seq2Seq) models have been used on many sequence labeling problems, including automatic speech recognition, neural machine translation, conversational modeling, and more. These models can transform sequences from the input domain (e.g. audio) into sequences in the labeled domain (e.g. text corresponding to the audio). [0003] Since language models do not require labeled data, they can be trained on billions or even trillions of representations and learn better annotation space models than any Seq2Seq model trained on annotated corpora. Therefore, Seq2Seq models are often combined with language models (LM) to improve generalization. [0004] Algorithms that integrate Seq2Seq models with LMs may be referred to as "fusion" algorithms. The standard way...

AI Technical Summary

Problems solved by technology

While these models have been shown to improve performance over baselines, they have several limitations

[0005] For example, while the deep fusion approach has been shown to improve performance over baselines, it has several limitations

Images