Guide
How to audit AI: audit types, a step-by-step process and the evidence auditors want
An AI audit is an independent check of an AI system, or the way an organization manages AI, against a defined set of criteria such as a law, a standard or internal policy. To audit AI, you pick the criteria, scope the systems, collect evidence, test the system and its controls, report findings and re-check after changes. This guide compares the six main kinds of AI audit (NYC bias audits, EU AI Act conformity assessments, ISO/IEC 42001 certification, internal audit, algorithmic audits and SOC 2 with AI in scope), then sets out the process, the evidence auditors ask for and how to choose an auditor.
Reviewed 5 Nov 2026 · next review by 4 Jan 2027 · 20 sources · report an error
What is an AI audit?
An AI audit is a structured review of an AI system, or of an organization’s AI governance, that compares what is actually happening with a stated benchmark and reports the gaps. The UK government’s catalogue of AI assurance techniques separates a bias audit, which assesses a system’s inputs and outputs “to determine if there is unfair bias”, from a compliance audit, which reviews adherence to internal policies or external rules, and from certification and conformity assessment, which attest against a standard or specification [1].
That distinction is the first decision in any AI audit: are you testing the system’s behavior, the controls around it, or both? A bias audit can pass a tool whose governance is weak, and a management-system certificate can be issued to an organization whose individual models have never been tested for bias.
What kinds of AI audit are there?
| Audit | Who performs it | What it checks | Output | Mandatory or voluntary |
|---|---|---|---|---|
| Bias audit (NYC Local Law 144) | An independent auditor, not employed by the employer or vendor and without a financial interest [3] | Selection or scoring rates and impact ratios by sex, race/ethnicity and intersectional categories, using historical data where available [3] | A public summary of results, posted with the tool’s distribution date [3] | Mandatory for employers and employment agencies using automated employment decision tools in NYC, within one year before use [2] |
| EU AI Act conformity assessment | The provider, by internal control (Annex VI); a notified body (Annex VII) for biometric systems where harmonised standards are not fully applied [4] | High-risk requirements, evidenced in Annex IV technical documentation [4][6] | Technical documentation and an EU declaration of conformity [6] | Mandatory for high-risk systems from December 2, 2027 (Annex III) and August 2, 2028 (Annex I) [5] |
| ISO/IEC 42001 certification audit | A certification body; accredited bodies are assessed against ISO/IEC 17021-1 and ISO/IEC 42006 [7][8] | The AI management system: scope, policies, risk assessment, Annex A controls and how they operate [9] | A certificate valid for three years, with annual surveillance audits [9][10] | Voluntary |
| Internal audit of AI | The organization’s internal audit function, often using the IIA’s AI Auditing Framework [11] | AI strategy and governance, AI risks, controls over data, algorithms and cybersecurity [11] | An internal audit report with findings and recommendations | Voluntary as a rule; part of the internal audit plan |
| Algorithmic or technical audit | Internal teams or specialist firms | System behavior: performance testing against benchmarks, bias, robustness, and sometimes formal verification [1] | A technical report, for example an algorithmic audit report [20] | Voluntary unless a law or contract requires it |
| SOC 2 with AI in scope | A CPA firm [14] | Controls against the Trust Services Criteria; AI evidence such as training lineage, inference logs and drift monitoring is added by each auditor, because no AI-specific criteria exist yet [15] | A SOC 2 report, whose system description shows what was in scope [15] | Voluntary; usually requested by customers |
| Model validation (banks) | An independent validation function or outside party [19] | Conceptual soundness, outcomes analysis and ongoing monitoring [19] | Validation report and findings | Supervisory expectation for US banks, not enforceable in itself [19] |
Which AI audit do you need?
| If your reason is… | Start with | Then consider |
|---|---|---|
| You use an automated hiring or promotion tool for NYC roles | An NYC Local Law 144 bias audit [2]; check readiness with our LL144 bias audit checker | A wider bias review beyond the categories the law requires; see our AI bias guide |
| You provide a high-risk AI system in the EU | Conformity assessment under Article 43 [4]; see the EU AI Act hub | ISO/IEC 42001, as a management system to run it |
| Enterprise customers want proof you govern AI | ISO/IEC 42001 certification; see ISO 42001 auditors compared | SOC 2 with AI controls named in the system description [15] |
| Your board wants assurance on AI risk | An internal audit using the IIA’s framework [11] | An outside algorithmic audit of the highest-risk systems |
| You are a bank relying on models | Model validation and MRM review [19]; see our model risk management guide | A separate review of generative AI, which US model guidance leaves out [19] |
How do you audit an AI system? Step by step
The steps below work for an internal audit or for preparing for an external one. They follow the end-to-end internal audit approach proposed by Raji and colleagues in 2020, which runs through the whole development lifecycle and produces documents at each stage [13], and the certification bodies’ Stage 1 (design) and Stage 2 (operation) model [9][10].
- 01
Set the criteria
Name the benchmark: a law (NYC LL144, EU AI Act), a standard (ISO/IEC 42001), a framework (NIST AI RMF) or your own AI policy. An audit without fixed criteria becomes a list of opinions.
- 02
- 03
Confirm independence and competence
For NYC audits, the auditor must not work for you or the vendor, have been involved with the tool or have a financial interest in either [3]. For ISO/IEC 42001, use a body accredited under ISO/IEC 42006, which sets AI-specific competence rules for certification staff [8].
- 04
Collect the evidence
Gather the documents and data in the evidence table below. NYC audits must use historical data from your own use of the tool where it exists, and may not use inferred demographic data [3].
- 05
- 06
Test the system and the controls
Run the technical tests (accuracy, bias by group, robustness) and test that controls operate: monitoring happens, approvals are recorded, incidents are handled. Stage 2 reviews operational evidence such as monitoring, internal audits and management reviews [9].
- 07
- 08
What evidence do AI auditors ask for?
| Evidence | What it shows | Asked for in |
|---|---|---|
| AI inventory and scope statement | Which systems the audit covers, and which it does not | ISO Stage 1 scope review [9]; SOC 2 system description [15] |
| System description and intended purpose | What the system is for, who uses it, how it fits other software | EU AI Act Annex IV, item 1 [6] |
| Dataset documentation | Sources, scope, selection, labeling and cleaning of training, validation and test data | Annex IV, item 2(d) [6] |
| Historical outcomes with demographic categories | Selection or scoring rates by group | NYC bias audits [3] |
| Risk assessment and risk management records | How risks were identified, rated and treated | ISO/IEC 42001 audits [9]; Annex IV, item 5 (risk management system) [6] |
| Test results and metrics | Accuracy, including for affected groups, and why the chosen metrics fit the system | Annex IV, items 2(g), 3 and 4 [6] |
| Human oversight design | Who can intervene, and how | Annex IV, item 2(e) [6] |
| Monitoring and logs | Drift dashboards, per-inference logs, alerts and responses | Practitioner SOC 2 overlays [15]; ISO Stage 2 monitoring evidence [9] |
| Change history | What changed in the system over its life | Annex IV, item 6 [6]; substantial-modification rule [4] |
| Policies, internal audits and management reviews | That the governance system runs, not just exists | ISO Stage 2 [9] |
| Vendor and model-provider evidence | Third-party models’ assurance and data handling | SOC 2 overlays on LLM subprocessors [15]; bank vendor-model validation [19] |
Why two AI audits of the same system can disagree
- 17 vs 1
- instances of potential LL144 non-compliance the State Comptroller found (at least 17) at the same 32 companies where NYC’s own review found 1[17]NY State Comptroller audit, July 2023–June 2025, issued December 2025
- 2
- complaints about automated employment decision tools received by NYC’s enforcement agency in two years[17]July 2023–June 2025
- 75%
- of the Comptroller’s test calls to NYC 311 to file an AI hiring tool complaint (9 of 12) that were not connected to the enforcement agency[18]Comptroller report 2024-N-6, issued December 2025
Change is coming for SOC 2. On October 28, 2026 the AICPA proposed updated Trust Services Criteria with new points of focus on AI governance, model monitoring, prompt manipulation, data poisoning, training data and automated decision-making. Comments close January 15, 2027; if approved, the criteria would apply to engagements starting on or after December 31, 2027 [14]. Our SOC 2 vs ISO 42001 FAQ explains how the two fit together.
How to choose an AI auditor
- Independence. Check the regime’s test. NYC excludes anyone employed by the employer or vendor, involved in the tool or with a financial interest [3]. Ask what else the firm sells you.
- Accreditation, for certificates. An ISO/IEC 42001 certificate from an accredited body means the body was assessed against ISO/IEC 42006 [7]. Canada’s accreditor moved its accredited bodies to the 2025 standard with immediate effect [8]. Check the accreditation body’s directory; our ISO 42006 explainer covers how.
- Competence. The published ISO/IEC 42006 made its competence requirements for certification staff more prescriptive than the draft [8]. For internal and IT auditors, ISACA’s Advanced in AI Audit (AAIA) credential requires an active CISA or a designation such as CIA or CPA with an IT audit focus [12].
- Method you can read. Ask for the metrics, data and criteria up front. BABL AI says it openly discloses its audit criteria and methodology [16]; ORCAA describes inference of protected categories when clients lack demographic data [20].
- Scope in writing. For SOC 2, name the AI evidence categories you expect in the system description [15].
- Fit. Big Four firms and specialist boutiques differ in price, depth and independence trade-offs; see Big 4 vs boutique AI audit firms.
How much does an AI audit cost and how long does it take?
No regulator or standards body publishes AI audit prices, and we found no independent price survey, so we don’t quote a figure. For certification, fees track audit days, and ISO/IEC 42006 now sets how accredited bodies calculate audit time for an AI management system (clause 9.1.4.2 and Annex A) [8]. Timelines we could confirm: BABL AI says NYC bias audits typically take two to three weeks once all documents are submitted [16]; Stage 1 and Stage 2 ISO/IEC 42001 audits each usually take several days, depending on scope [9].
For estimates built from published inputs, use our ISO 42001 certification cost tool, SOC 2 audit cost for AI companies and EU AI Act conformity assessment cost calculators.
Frequently asked questions
+What is an AI audit?
An independent review of an AI system or of how an organization governs AI, measured against criteria such as a law, standard or policy. The UK’s AI assurance catalogue distinguishes bias audits, compliance audits, certification, conformity assessment and performance testing [1].
+How do you audit an AI system?
Set the criteria, scope the systems, confirm the auditor is independent, collect evidence, review design and documentation, test the system and its controls, report findings, then remediate and re-audit when the system changes. Certification audits split this into a design review (Stage 1) and an operating review (Stage 2) [9][10].
+Who can perform an AI audit?
It depends on the audit. NYC bias audits need an independent auditor [3]; EU conformity assessments are done by the provider or, for some biometric systems, a notified body [4]; ISO/IEC 42001 certificates come from certification bodies, which accreditors assess against ISO/IEC 42006 [7]; SOC 2 reports come from CPA firms [14].
+Is an AI audit mandatory?
Only in some cases. NYC requires a bias audit within one year before an automated employment decision tool is used [2]. The EU AI Act requires conformity assessment for high-risk systems from December 2, 2027 for Annex III uses [5]. ISO/IEC 42001 certification, internal audits and SOC 2 are voluntary.
+How often should AI be audited?
At least as often as the rule requires and whenever the system changes materially. NYC bias audits must be no more than a year old when a tool is used [2]; ISO certificates need surveillance audits at least once each calendar year [10]; an EU high-risk system needs a new conformity assessment after a substantial modification [4].
+Is ISO 42001 certification the same as EU AI Act conformity assessment?
No. ISO/IEC 42001 certification audits an organization’s AI management system [9], while EU conformity assessment checks a specific high-risk AI system against the Act’s requirements, using its technical documentation [4][6]. A certificate can support conformity work but does not replace it.
+Does SOC 2 cover AI?
Not specifically yet. The current Trust Services Criteria have no AI-specific criteria, so auditors add their own AI evidence lists [15]. The AICPA proposed AI-related points of focus on October 28, 2026, which would apply to engagements starting on or after December 31, 2027 if approved [14].
+What is ISO/IEC 42006?
A standard published in July 2025 that sets extra requirements for bodies that audit and certify AI management systems under ISO/IEC 42001, extending ISO/IEC 17021-1 [7]. It covers areas such as audit time calculation and competence of certification staff [8].
+What is the IIA AI Auditing Framework?
Guidance from the Institute of Internal Auditors for auditing AI, updated in September 2024 and free to access. It covers AI strategy and governance, AI risks, controls over data, algorithms and cybersecurity, and assurance over AI-enabled processes [11].
Sources
- [1]UK Department for Science, Innovation and Technology, Portfolio of AI assurance techniques (2026-11-02).
- [2]NYC Department of Consumer and Worker Protection, Automated Employment Decision Tools (AEDT).
- [3]NYC Department of Consumer and Worker Protection, Automated Employment Decision Tools: Frequently Asked Questions (2023-06-29).
- [4]European Commission (AI Act Service Desk), AI Act Article 43: Conformity assessment.
- [5]Cuatrecasas, Digital Omnibus on AI has been published (2026-07).
- [6]Future of Life Institute (artificialintelligenceact.eu), AI Act Annex IV: Technical documentation referred to in Article 11(1).
- [7]ISO, ISO/IEC 42006:2025: Requirements for bodies providing audit and certification of artificial intelligence management systems (2025-07).
- [8]Standards Council of Canada, Bulletin 2025-21: Transition to ISO/IEC 42006:2025 (2025).
- [9]Schellman, Guide to ISO 42001 certification.
- [10]Bureau Veritas, The certification process.
- [11]The Institute of Internal Auditors, The IIA’s Artificial Intelligence Auditing Framework (2024-09-13).
- [12]ISACA, Advanced in AI Audit (AAIA) certification.
- [13]arXiv (Raji et al.), Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing (2020-01-03).
- [14]AICPA & CIMA, AICPA proposes update of SOC 2 Trust Services Criteria to enhance the relevance, usability and value of CPAs’ reports (2026-10-28).
- [15]Cloud Security Alliance, The SOC 2 AI Gap: Auditors Improvise Where AICPA Has Not Acted (2026-08-30).
- [16]BABL AI, NYC Bias Audit.
- [17]Office of the New York State Comptroller, Enforcement of Local Law 144: Automated Employment Decision Tools (2025-12-02).
- [18]Office of the New York State Comptroller, Report 2024-N-6: Enforcement of Local Law 144 – Automated Employment Decision Tools (full report, PDF) (2025-12-02).
- [19]Federal Reserve Board, OCC and FDIC, SR 26-2 letter and attached Revised Guidance on Model Risk Management (PDF) (2026-04-17).
- [20]ORCAA, O’Neil Risk Consulting & Algorithmic Auditing: services.
How we keep this page current
Every claim links to its source. We re-check this page at least every 60 days, and sooner when a standard, law or vendor changes; vendor names, acquisitions and shutdowns update automatically from our directory. Vendors cannot pay to be included or ranked.