Explainability refers to a representation of the mechanisms underlying AI systems' operation, whereas interpretability refers to the meaning of AI systems' output in the context of their designed functional purposes. Together, explainability and interpretability assist those operating or overseeing an AI system, as well as users of an AI system, to gain deeper insights into the functionality and trustworthiness of the system, including its outputs. The underlying assumption is that perceptions of negative risk stem from a lack of ability to make sense of, or contextualize, system output appropriately. Explainable and interpretable AI systems offer information that will help end users understand the purposes and potential impact of an AI system.
Risk from lack of explainability may be managed by describing how AI systems function, with descriptions tailored to individual differences such as the user's role, knowledge, and skill level. Explainable systems can be debugged and monitored more easily, and they lend themselves to more thorough documentation, audit, and governance.
Risks to interpretability often can be addressed by communicating a description of why an AI system made a particular prediction or recommendation.
Transparency, explainability, and interpretability are distinct characteristics that support each other:
- Transparency can answer the question of "what happened" in the system
- Explainability can answer the question of "how" a decision was made in the system
- Interpretability can answer the question of "why" a decision was made by the system and its meaning or context to the user