Training method and device for performing operation based on language model

Abstract

The invention discloses a training method and device for performing arithmetic operation based on a language model, and the method comprises the steps: adding a space character in front of each character in an equation text, and generating a quasi equation text; performing word segmentation processing on the quasi equation text to obtain a list corresponding to the quasi equation text, the list comprising a plurality of words; taking a list corresponding to the quasi equation text as a training sample to obtain training sample data; and performing operation training on the training sample data by using a language model to generate an arithmetic model. Therefore, the equation text is preprocessed by adding the space characters and performing word segmentation processing, so that each character in the equation text serves as an independent word to be input into the language model, and the language model can learn each character of the equation text and the digit information of each character; therefore, after the language model is trained by using a small number of training samples, the calculation accuracy of the trained arithmetic model can be effectively improved.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence, in particular to a training method and device for performing calculations based on a language model. Background technique [0002] In the latest research, because the giant language model GPT-3 can perform arithmetic operations directly without training through small-sample learning, it performs well in arithmetic operations tasks. But for smaller language models, such as Bert, GPT-2, etc., the accuracy rate is almost 0 when performing arithmetic operations after learning with small samples, and basically has no practical value. [0003] At present, for smaller language models, most of them use the "pre-training + fine-tuning" method for model training. Under this training method, take GPT-2 as an example. Although the arithmetic model generated after model training can calculate some arithmetic operations, it requires a large amount of training data, such as two-digit plus two-d...

AI Technical Summary

Problems solved by technology

Under this training method, take GPT-2 as an example. Although the arithmetic model generated after model training can calculate some arithmetic operations, it requires a large amount of training data, such as two-digit plus two-digit arithmetic operations, GPT-2 A training set of more than 3200 calculation formulas is required to barely achieve 10% accuracy, and for more complex three-digit addition calculations, a larger training data set is required

It can be seen that the traditional training method is not suitable for the training of smaller language models.

Images