Pre-training and/or transfer learning for sequence taggers

Abstract

Systems and methods for pre-training a sequence tagger with unlabeled data, such as a hidden layered conditional random field model are provided. Additionally, systems and methods for transfer learning are provided. Accordingly, the systems and methods build more accurate, more reliable, and / or more efficient sequence taggers than previously utilized sequence taggers that are not pre-trained with unlabeled data and / or that are not capable of transfer learning / training.

Description

Background technique [0001] Sequential labeling and classification of data (also referred to herein as sequence labeling) has many applications, including those in natural language processing and speech processing. Some example applications include tagging search queries, segmenting ads, and language identification / verification. Several different machine learning techniques have been applied to the problem of labeling sequences, such as conditional random fields (CRF) and neural networks. [0002] Conditional Random Fields (CRFs) are discriminative models that directly estimate the probability of a sequence of states conditioned on the entire sequence of observations, and are also known as information extraction tasks. For example, frames of audio signal data can be converted into features, where a sequence of states is predicted over all frames. Because CRFs can be used for many different tasks, and because they can achieve high accuracy with minimal tuning, conditional ran...

AI Technical Summary

Problems solved by technology

However, CRF fails to exploit unlabeled data

Images