Sample data processing method and device, server and storage medium

Abstract

The invention provides a sample data processing method and device, a server and a storage medium. The method comprises the steps that multiple pieces of target sample data are acquired, and the target sample data carry annotation information; a marking information entropy of the target sample data is determined according to marking information carried by the target sample data; and according to the labeling information entropy of the target sample data, first target data of which the labeling quality meets a preset quality requirement is determined from the plurality of target sample data. In the embodiment of the specification, the consistency degree of different labeling sources for labeling the same sample data is quantified by firstly determining the labeling information entropy capable of reflecting the uncertainty of the labeling information of the target sample data; therefore, the target sample data with relatively high labeling quality can be screened out according to the labeling information entropy to be used as the first target data, so that the data with relatively high labeling quality can be efficiently and accurately screened out from the plurality of target sample data, and the data error is reduced.

Description

technical field [0001] This specification belongs to the technical field of the Internet, and in particular relates to a sample data processing method, device, server and storage medium. Background technique [0002] When using sample data for model training, it is usually necessary to label the sample data used first. [0003] For example, the labeler in charge of labeling usually analyzes and judges the attributes of each sample data according to the pre-determined labeling rules, and then sets corresponding labeling information for each sample data according to the judgment results to indicate the attributes of the sample data Features (such as the type or level corresponding to the sample data, etc.), to complete the labeling of the sample data. Then, specific model training can be carried out based on the above-mentioned labeled sample data. [0004] When labeling the sample data according to the labeling rules, the labeler may be subject to personal subjective influe...

AI Technical Summary

Problems solved by technology

However, different labelers often have different understandings and grasps of labeling rules and sample data. As a result, after the same sample data is marked by different labelers, there may be multiple different labeling information, which makes subsequent use There will be confusion or errors in the above-mentioned labeled sample data, which will affect the labeling quality of the sample data

Images