Independent Bias Audit (NYC LL 144)

Bias audits sit at the intersection of EU AI Act Article 10 (data governance), Article 15 (accuracy, robustness, cybersecurity), and the NYC Local Law 144 on Automated Employment Decision Tools, which has been enforceable since July 2023.

NYC LL144 requires that employers and employment agencies using automated employment decision tools conduct an independent bias audit within one year of using the tool, and re-audit annually. The bias audit results must be published on the employer's website before use.

Similar bias-audit obligations exist or are emerging in Colorado SB 24-205 and in proposed Illinois HB 3773.

Independent bias-audit firms (Babl AI, ORCAA, Holistic AI, Luminos Law) handle the third-party audit deliverable that NYC LL144 specifically requires. Software vendors (Arthur, Fiddler, Credo AI, Holistic AI, FairNow, Saidot) handle the pre-audit calculations, dataset prep, and the ongoing monitoring that lets the next audit go faster.

For NYC LL144 specifically, look for:

• Calculation of the four-fifths rule selection rate analyses across race-ethnicity and sex-gender categories.
• Intersectional analysis across category combinations.
• Audit-report templates that match the LL144 summary-of-results structure.
• Public-posting workflow with version history.

For broader bias work, look for full bias-detection tooling: counterfactual fairness, demographic parity, equalized odds, calibration, and intersectional metrics across the protected-attribute taxonomy defined for the use case.

• [ ] Determine whether the system falls within NYC LL144's "automated employment decision tool" definition.
• [ ] Engage an independent auditor with no financial interest in the tool.
• [ ] Calculate selection rates and impact ratios required by 6 RCNY 5-301.
• [ ] Publish a summary of the most recent bias audit on the public website.
• [ ] Re-audit at least annually.
• [ ] For EU AI Act Article 10, run a broader bias scan beyond the LL144 minimum.
• [ ] Document mitigation when bias is identified.
• [ ] Tie bias-audit findings to the risk management system.

The most common LL144 gap is not knowing the law applies. Tools used to screen, evaluate, or recommend candidates for employment decisions in NYC are in scope, even when the employer is headquartered elsewhere. The tool's user location and the candidate's location both matter.

The second gap is audit independence. Several vendors offer in-house audit reports. NYC's Department of Consumer and Worker Protection has stated those do not satisfy the independence requirement. The auditor must be external.

The third gap is treating the bias audit as a one-time deliverable. The annual re-audit obligation is binding. A 2025 audit does not cover 2026 use of the tool.

• NYC Local Law 144 of 2021 — full enforcement guidance from NYC DCWP.
• 6 RCNY Chapter 5, Subchapter T — the implementing regulation specifying bias-audit math.
• EEOC technical assistance on AI in employment — federal context.
• Colorado SB 24-205 — state-level analogue, effective February 2026.
• EU AI Act Article 10: Data and Data Governance — the EU bias-mitigation obligation.