Transfer learning method based on latent semantic analysis

Abstract

The invention provides a transfer learning method based on latent semantic analysis. The method includes the following steps: carrying out stop word removal and stemming on training data; calculating the vocabulary weight in a source domain and the vocabulary weight in a target domain respectively to acquire a vocabulary-text matrix M; carrying out singular value decomposition on the matrix M; mapping vocabulary and text in the matrix M to a lower dimension latent semantic space; removing synonymy noise effect from the source domain; adjusting the matrix M structure; finding vocabulary with a high association degree with target domain text in the source domain as transfer terms; adjusting the matrix M structure again; analyzing target domain vocabulary in the adjusted matrix M structure to acquire new character representation of target domain data; acquiring a final classifier in a training dataset; and classifying a testing dataset S.

Description

technical field [0001] What the present invention relates to is a kind of machine learning method. Background technique [0002] With the development of the Internet, more and more information is stored on the network in the form of text, which becomes the source of information for people. Faced with a huge text library, people urgently need an efficient technical means to organize and classify the data in the text library. Machine learning studies how computers simulate human learning behaviors to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their own performance. But machine learning has a very important assumption, that is, the training data and test data must obey the same distribution. This brings great trouble to practical application. When a new field emerges, the data in the sample space is often less and the features are sparse. At this time, using traditional machine learning to classify the data will prod...

AI Technical Summary

Problems solved by technology

[0003] At present, there are some studies on transfer learning methods, most of which only analyze data from the vocabulary level and do not consider the semantic correlation between the source domain and the target domain. Some "noise" factors are likely to affect the effect of transfer learning.

In addition, some transfer learning methods either require independence assumptions on the training data, or know the prior probability of the data in advance, which is not realistic

Images