Language model training method and device and computer equipment

Abstract

The invention relates to the field of artificial intelligence, and discloses a language model training method, which comprises the steps of inputting a modified MLM task and a modified NSP task into afirst Bert model for training to obtain a first model parameter corresponding to the first Bert model; applying the first model parameter to a second Bert model, and training the second Bert model through the modified MLM task and the modified NSP task; judging whether an output result of the second Bert model reaches a preset condition or not; and if so, judging that the second Bert model reaches the use standard. A structure and a training method of a Bert model are improved, the application range and accuracy of migrating the pre-trained language model to the specific application scene areexpanded, and the improved Bert model is trained under the current optimized task data, so that the training model can be more suitable for the specific application scene or field, the recognition effect is better, and the specific application scene according to the real-time request is ensured. The method is also suitable for the field of smart government affairs, thereby promoting the construction of smart cities.

Description

technical field [0001] This application relates to the field of artificial intelligence, in particular to a language model training method, device and computer equipment. Background technique [0002] The model trained on general data contains a lot of general language information, such as lexical and syntactic information, but lacks the specific semantic information contained in data in specific fields, so it usually needs to be fine-tuned on downstream data when using it. Optimizing training, the above fine-tuning means fine-tuning, otherwise the effect will be poor, and a certain amount of data needs to be guaranteed when doing fine-tuning, otherwise it will be difficult to migrate from the general field to the vertical field. There are a large number of AI applications in the field of government affairs, but it is difficult to label task data, and it is difficult to ensure that a high-matching training model can be obtained according to the specific application scenario ...

AI Technical Summary

Problems solved by technology

[0002] The model trained on general data contains a lot of general language information, such as lexical and syntactic information, but lacks the specific semantic information contained in data in specific fields, so it usually needs to be fine-tuned on downstream data when using it. Optimize training, the above fine-tuning means fine-tuning, otherwise the effect will be poor, and a certain amount of data needs to be guaranteed when doing fine-tuning, otherwise it will be difficult to migrate from the general field to the vertical field

There are a large number of AI applications in the field of government affairs, but task data labeling is difficult, and it is difficult to ensure that a high-matching training model can be obtained according to the specific application scenarios of real-time requests.

Images