The False Hope of Current Approaches to Explainable AI in Health Care

Raj: I think the problem with explainability and explainable AI is actually exactly that it is a technically squishy definition. There is no real way of verifying that something is explainable. And so, you know, my first problem is that you're demanding this thing that is not well defined. My second problem is when you demand it, you make the model worse according to this metric that I really care about. The third problem is we know explanations make people more likely to follow bad advice. We've known this for a long time in robotic systems.