Financial scene-oriented end-to-end natural language processing training framework and method

Abstract

The invention discloses a financial scene-oriented end-to-end natural language processing training framework and method, and relates to the field of natural language processing. The training framework comprises a Google native BERT module, a FinBERT pre-training module, a module for mining new data from external related data based on a similar self-training thought, a module for pre-training on downstream task corpora, a module for fully utilizing untagged corpora by using a semi-supervised learning framework, a knowledge distillation module and an online deployment module. The training method comprises the following steps: step 1, performing FinBERT pre-training; step 2, mining new data from external related data based on a similar self-training thought; step 3, pre-training a downstream task corpus, pre-training the FinBERT again, andcalling an obtained model as TASK FinBERT; step 4, using a semi-supervised learning framework to make full use of the label-free corpus, and on the basis of TASK FinBERT, calling a model obtained by training as UDA FinBERT; and step 5, distillation learning: distilling the learned knowledge and characteristics to the lightweight model.

Description

technical field [0001] The present invention relates to the field of natural language processing, in particular to an end-to-end natural language processing training framework and method for financial scenarios. Background technique [0002] The rapid development of modern information technology has resulted in explosive growth of data and information contained on the Internet. A large amount of data is presented in the form of text, such as the web page data of major Internet, and if big data wants to reflect the value of the data, it is necessary to use Natural Language Processing (Natural Language Processing, NLP) technology to mine the core content of the text. In text mining, different fields will have a variety of natural language processing tasks. For example, natural language processing tasks in the financial field include text classification, sentiment analysis, text clustering, entity recognition, text similarity calculation, information extraction, etc. There are...

AI Technical Summary

Problems solved by technology

The open source BERT model does not perform well in tasks in the financial field, and some tasks are not as good as some lightweight models. To achieve good results, a lot of data needs to be labeled, and the cost of labeling data is very high;

[0007] 2. The BERT12-layer Transformer network has a large number of parameters, the forward calculation is time-consuming, the model pressure test time exceeds the standard, and the online deployment is difficult

[0008] 3. In the financial field, the task is huge. Without a good baseline model, it takes a lot of time to experiment and try, and sometimes detours, wasting a lot of time and experience

Images