Language model training method, device and equipment and computer readable storage medium

Abstract

The invention relates to an artificial intelligence technology, and discloses a language model training method, which comprises the following steps of: respectively carrying out word-level mask, phrase-level mask, entity-level mask and part-of-speech-level mask processing on texts in a training data set to obtain a to-be-used pre-training data set; performing sentence vector representation processing on texts in the to-be-used pre-training data set to obtain a pre-training data set represented by sentence vectors; and inputting the pre-training data set represented by the sentence vector into the language model, carrying out model reasoning iteration training on the language model, and when a preset model training completion condition is satisfied, completing the training of the language model. The invention further relates to a block chain technology, and the training data set is stored in the block chain. The problems that in the prior art, a model obtained through an existing model training mode cannot learn information of a Chinese semantic level and information of a Chinese entity relation, and the sensitivity and accuracy of the model to nouns are low can be solved.

Description

technical field [0001] The present invention relates to the field of artificial intelligence, in particular to a language model training method, device, electronic equipment and computer-readable storage medium. Background technique [0002] The pre-training model is one of the most important development directions in the field of artificial intelligence NLP. In recent years, various large-scale pre-training models have appeared in front of everyone, such as BERT, RoBERTa, XLNET and so on. [0003] Current language models are trained by randomly masking 15% of the basic language units and using other basic units in sentences as input to train a task to predict the masked units. When dealing with the Chinese language, modeling can only be done by predicting Chinese characters, so the model cannot learn the complete semantics of Chinese words and entities. For example, for the training corpus of "Beijing is the capital of China", using the current training method, the model ...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a language model training method, device, electronic equipment and computer-readable storage medium. Entity relationship information, model sensitivity to nouns, and low accuracy

Images