Data set acquisition method and device based on artificial intelligence, equipment and medium

Abstract

The invention relates to a data set acquisition method and device based on artificial intelligence, equipment and a medium. The method comprises the steps of obtaining an initial sample set; labelingthe initial sample set by using an initial language model to obtain a model labeling reference index; filtering the initial sample set according to the model annotation reference index to obtain a corrected set; training the initial language model by using the corrected set to obtain a corrected initial language model; when the precision of the corrected initial language model does not reach a preset threshold, expanding the data volume of the corrected set to update the corrected set, continuing to train the initial language model by using the corrected set to obtain a corrected initial language model, and when the precision of the initial language model reaches the preset threshold, obtaining a target language model; and processing to-be-processed service data according to the target language model to obtain a data set. By adopting the method, the data set acquisition efficiency can be improved. In addition, the invention also relates to a block chain technology, and the initial sample set, the corrected set and the data set can be stored in a block chain.

Description

technical field [0001] The present application relates to the technical field of artificial intelligence, in particular to an artificial intelligence-based data set acquisition method, device, computer equipment and storage medium. Background technique [0002] In the development process of artificial intelligence, the industry generally adopts a data-driven approach, so data quality is the top priority. Data with a large quantity, good quality, and complete coverage can help developers develop models with better effects faster, thereby improving customer satisfaction. [0003] In the process of training the model, the data needs to be marked. At present, after the log data is obtained, the computer is used to extract the content and review the data. However, when the log data is processed, the machine cannot know the correct information. Determine the correct data from a large amount of log data, so that the failure of data labeling makes it impossible to obtain the correc...

AI Technical Summary

Problems solved by technology

[0003] In the process of training the model, the data needs to be marked. At present, after the log data is obtained, the computer is used to extract the content and review the data. However, when the log data is processed, the machine cannot know the correct information. Determine the correct data from a large amount of log data, so that the data annotation fails and the correct data set cannot be obtained

Images