Preparing for an AI Governance Audit: What Do Examiners Check?

If your organization has deployed machine learning models in anything customer-facing, credit-related, or otherwise consequential, an AI governance audit is probably closer than you think. Regulators, internal audit teams, and third-party assessors are moving from asking "do you have an AI policy?" to "show me the evidence." That shift catches a lot of teams off guard.

This post breaks down what examiners are actually checking for, based on the patterns showing up across financial services, healthcare, insurance, and increasingly any sector using AI in decisions that affect people — hiring, lending, pricing, moderation, and clinical support among them.

The Mental Model: Audits Test for Evidence, Not Intentions

The single biggest misconception teams have going into their first AI audit is thinking it's a conversation about philosophy — "we take fairness seriously," "we have a responsible AI framework." Examiners don't audit intentions. They audit traceable evidence that a control existed, was followed, and produced a specific, reviewable outcome.

If you can't produce a document, log, or approval record for a claim, the examiner treats it as if the control doesn't exist — regardless of whether it actually happened. This reframes the whole preparation exercise: it's not about doing more governance, it's about being able to prove the governance you already do.

1. Model Inventory and Ownership

The first thing most examiners request is a complete inventory of AI/ML systems in use. This sounds basic, but it's where most organizations stumble first, because:

• Shadow AI usage (teams using third-party APIs or open-source models without going through procurement) rarely makes it onto official lists.
• "Model" definitions get contested — does a vendor-embedded recommendation engine count? Does a prompt template count as a model change?
• Ownership is often unclear once a model moves from the data science team into production and gets maintained by engineering.

What examiners want to see:

• A single source of truth listing every model, its purpose, risk tier, owner, and deployment date.
• Clear accountability — a named individual or committee responsible for each model's lifecycle, not just "the AI team."
• Evidence that the inventory is actively maintained, not a one-time spreadsheet from eighteen months ago.

2. Risk Tiering and Proportionality

Not every model needs the same level of scrutiny, and examiners know that. What they're checking is whether your organization has a documented, consistently applied methodology for deciding which models get heavy oversight, and which get lighter treatment.

A common failure mode: risk tiering exists on paper but isn't actually driving different levels of review in practice. If your "high-risk" and "low-risk" models go through identical (or identically absent) validation, examiners will flag the tiering system as cosmetic.

What holds up well:

• Clear, written criteria for risk classification (e.g., materiality of the decision, degree of automation, populations affected, reversibility of harm).
• Evidence that higher-tier models trigger additional requirements: more frequent monitoring, human review thresholds, board-level reporting.

3. Data Lineage and Training Data Provenance

Examiners increasingly ask, "where did this data come from, and were you allowed to use it this way?" This covers:

• Consent and legal basis for data used in training or fine-tuning.
• Whether data used for one purpose (e.g., fraud detection) was repurposed for another (e.g., marketing) without re-assessment.
• Documentation of known data quality issues, missingness, and how they were handled.

Lineage documentation doesn't need to be exhaustive for every dataset, but for anything feeding a high-risk model, examiners expect a reconstructable path from raw source to training set.

4. Testing for Bias and Disparate Impact

This is often the area organizations are least prepared to defend in detail. It's not enough to say "we tested for bias." Examiners want specifics:

• Which protected or sensitive attributes were tested against, and why those were chosen.
• What statistical thresholds were used to define an acceptable disparity (e.g., four-fifths rule, statistical parity, equalized odds), and who approved those thresholds.
• What happened when a disparity was found — was the model adjusted, was a mitigation approved, was the residual risk formally accepted by someone with authority to do so?

The last point matters more than people expect. Finding bias isn't itself a failure in an audit — failing to have a documented decision-making trail for what happened next is.

5. Human Oversight and Override Mechanisms

Regulators are particularly focused on whether "human in the loop" is real or theoretical. Examiners will probe:

• Do reviewers actually have the time, information, and authority to override the model, or is override a formality that rarely happens in practice?
• Is there a log of override rates? A near-zero override rate can itself raise questions — it may suggest reviewers are rubber-stamping outputs rather than genuinely evaluating them.
• Is escalation clearly defined for edge cases the model wasn't designed to handle?

6. Monitoring for Drift and Post-Deployment Performance

A model that was validated at launch but never re-evaluated is one of the most common audit findings. Examiners look for:

• Defined monitoring metrics (accuracy, calibration, fairness metrics) tracked on a set cadence, not just at deployment.
• Documented thresholds that trigger review, retraining, or rollback — and evidence those thresholds have actually been acted on.
• A record of model versions, so examiners can trace which version produced which decision, months or years later.

7. Incident Response and Model-Specific Escalation Paths

Just as with cybersecurity, examiners want to know what happens when something goes wrong — a model produces a harmful, biased, or clearly incorrect output.

• Is there a defined process for reporting AI-related incidents, separate from general IT incident processes?
• Are incidents logged, root-caused, and fed back into model improvement or governance policy updates?
• Is there evidence of at least one real (not hypothetical) incident being handled end-to-end?

8. Explainability and Documentation for Affected Individuals

Particularly in lending, insurance, hiring, and healthcare, examiners check whether people affected by a model's decision can get a meaningful explanation — not necessarily the full technical mechanism, but a clear, accurate account of the key factors involved.

• Are adverse action notices (or equivalents) generated in a way that's traceable to actual model outputs, rather than generic templates disconnected from the real decision logic?
• Is there a process for someone to contest or appeal an automated decision?

9. Third-Party and Vendor Model Governance

If you use vendor models or foundation model APIs, examiners will ask how you govern something you didn't build. This is a growing area of scrutiny because organizations often treat vendor tools as outside the scope of their AI governance program — examiners disagree.

• Do you have documentation of the vendor's own testing, and have you independently validated claims relevant to your use case?
• Is there a contractual and technical plan for what happens if the vendor changes the model without notice?
• Are vendor models included in the same inventory and risk-tiering process as internally built ones?

10. Governance Structure and Accountability

Finally, examiners look at the organizational layer: is there a committee or function with real authority over AI risk decisions, and does it have teeth?

• Meeting minutes, not just charter documents, showing the governance body actually reviewed and made decisions about specific models.
• Evidence that a model was delayed, modified, or rejected based on governance input — a strong signal the process isn't purely ceremonial.
• Clear lines from technical staff to senior leadership on AI risk reporting.

Practical Steps to Prepare

1. Run a mock audit internally first. Pick two or three of your highest-risk models and try to assemble the full evidence trail an examiner would ask for. Gaps will surface fast.
2. Centralize documentation now, not during the audit window. Scrambling to reconstruct decisions from Slack threads and old emails under time pressure is where most audit stress comes from.
3. Interview your own model owners. If they can't clearly explain what the model does, what data it uses, and what oversight exists, examiners won't get a clear answer either.
4. Check that your risk tiering actually changes behavior. If every model gets the same treatment, tighten your methodology before the audit, not during it.
5. Close the loop on past findings. If a previous internal review or audit flagged something, examiners will specifically check whether it was actually remediated.

The Bottom Line

Examiners aren't looking for perfect models — they're looking for a governance program that can demonstrate it knows where its risks are, has proportionate controls in place, and can produce evidence on demand. The organizations that do well in these audits aren't the ones with the most sophisticated AI; they're the ones that have made their governance process legible, current, and genuinely operational rather than a document that exists for exactly this occasion.

Start by asking, for your three riskiest models: could you hand an examiner the full evidence trail today? If the honest answer is no, that's exactly where to start.