Evaluation method and device for model interpretation tool

Abstract

The embodiment of the invention provides an evaluation method and device for a model interpretation tool. According to one specific embodiment, the method comprises: determining a potential trigger for a pre-trained target model with a backdoor, output of the target model for sample data carrying the backdoor trigger being a target label, and the potential trigger having the function of enabling the target model to output the target label; in response to determining that the potential trigger and the backdoor trigger meet the preset condition, determining that the target model is a model for evaluation; on the basis of the evaluation model and preset evaluation sample data, generating saliency information by using a to-be-evaluated model explanation tool, adding a back door trigger to the evaluation sample data, and the saliency information being used for displaying the influence of each part of the evaluation sample data on the output of the evaluation model; according to the saliency information and the backdoor trigger, determining an evaluation index, and the evaluation index being used for evaluating the model interpretation tool.

Description

technical field [0001] The embodiments of this specification relate to the technical field of machine learning, and in particular to an evaluation method and device for model interpretation tools. Background technique [0002] At present, machine learning is widely used in various fields. In essence, a machine learning model uses a complex function to fit the relationship between data and targets. A machine learning model is very different from some simple rules, which specify the relationship between data and targets, but a machine learning model is a black box with only input and output. People often cannot understand the decision-making process and basis of the machine learning model, so they cannot truly trust the model. In order to explain machine learning models, researchers at home and abroad have conducted a lot of research on the interpretability of machine learning models, and proposed a variety of model interpretation tools for explaining machine learning models....

AI Technical Summary

Problems solved by technology

However, there is currently a lack of practical performance evaluation methods to judge the pros and cons of a model interpretation tool, which poses great difficulties to the interpretability research of machine learning models.

Images