Two things wearing one word
"Explainable AI" has come to mean a specific set of techniques. A heatmap showing which inputs the model attended to. A score attributing the output to each feature. Or, increasingly, the model itself producing a paragraph that describes why it reached its answer. These methods are real and they have their uses. They describe, in one way or another, what a model did.
There is a second thing the word is supposed to mean, especially in regulated work, and it is not the same thing at all. When a regulator asks a bank to explain a decision, they are not asking what the model attended to. They are asking which rule was applied, what that rule requires, where in the regulation it comes from, and what facts triggered it. They want the basis of the decision, traceable to law. The first kind of explanation describes a model. The second kind justifies a decision. A system can have the first in abundance and still fail completely at the second.
Why a narration is not a cause
Ask a model why it decided something and it will tell you, fluently. The problem is that the answer is generated after the fact and is not bound to the actual computation that produced the decision. The model is not introspecting a causal chain. It is producing text that is plausible given its own output, which is a different operation that merely looks like introspection. The stated reason can be wrong while the answer is right. The stated reason can sound airtight while the answer is wrong. Nothing connects the explanation to the cause except that both came from the same model, and that is not a connection a regulator can rely on.
This is the quiet failure of treating model narration as explainability. It produces something in the shape of a justification, with the confidence of one, and none of the guarantee. A justification that might not be the real reason is worse than no justification, because it invites you to stop looking.
What a regulator actually accepts
Now look at what explanation means when a rule made the decision. The rule fired because its conditions were met. The conditions are explicit. They point at specific facts. The rule carries the article it came from. So the account of the decision is complete and it is the actual cause:
{
"decision": "review",
"rule_id": "DP-BREACH-EU-001",
"source": "GDPR Article 33",
"why": "Breach awareness recorded 2026-03-01 09:00; authority notified 2026-03-04 14:00; elapsed 77h exceeds the 72h limit.",
"deterministic": true
}
There is no gap here between the explanation and the reason. The elapsed time exceeding the limit is not a story about why the decision was made. It is why the decision was made. The citation is not decoration added to make a guess look grounded. It is the rule that did the deciding. You can hand this to an auditor and they can check every part of it against the regulation, because every part of it is drawn from the regulation.
Explainability is a property of where the decision came from
This is the heart of it. Explainability is not a feature you bolt onto a decision after you have made it. It is a property of how the decision was made in the first place. A deterministic rule is explainable by construction, because the thing that produced the verdict and the thing that explains it are one and the same. A model's decision is not explainable in this sense at any price, because the explanation is always a separate artifact generated alongside, and a separate artifact can diverge from the truth.
You cannot make a probabilistic decision explainable by attaching a better narration to it. You can only make the decision itself come from something that explains as it decides. That is why the explanation problem and the determinism problem are the same problem. Solve where the decision comes from, and the explanation is already there. Leave the decision in the model and no amount of interpretability tooling will give you an account you can defend.
The point
In consumer AI, explainability is a nice-to-have, a way to build a little trust in a recommendation no one will audit. In regulated work it is the requirement, and the requirement is specific: the decision must trace to a rule, and the rule to its source, with the logic legible to someone who was not in the room. A model narrating its own reasoning does not meet that bar, however convincing the narration. It is a second guess sitting on top of the first one. The decision that can be explained is the decision that was made by something built to be read.