Why explainable AI can still produce wrong decisions without data provenance

A fraud model looks at an insurance claim and returns a score of 0.23, while a SHAP explanation lays out why by citing no prior claims and a modest claim value. [1] A human adjuster reads the explanation, agrees with it, and closes the claim, checking every box that explainable AI asks for. [1] Months later, an auditor might ask if the decision can be reconstructed independently from its own evidence trail. [1]

The auditor checking document creation metadata might find that a corroborating document was generated two days after the claim was filed. [1] Document provenance was never captured anywhere in the intake pipeline as a fact to be verified. [1] The model might also use an asset valuation that is 22 months old and hand it to the model as though it were current. [1]

The scenario demonstrates that focusing solely on model explainability, such as SHAP outputs, is insufficient if the intake pipeline fails to verify document lineage or temporal accuracy. Explainable AI only explains how the model processed the data it was given, not whether that data was reliable or current. When compared to basic data governance frameworks, pure explainability tools fall short because they assume the intake data is flawless. What the sources don't address: How organizations can efficiently automate the verification of document provenance before data ever reaches the AI model.