Guide
What is AI bias? Types, real examples, how it’s measured and how to prevent it
AI bias is when an AI system produces results that are systematically worse, or less accurate, for some groups of people than for others. NIST, the US standards agency, says it comes from three places: the data and algorithms (statistical and computational bias), the people who build and use the system (human bias), and the institutions around it (systemic bias) [1]. This guide explains each type with documented cases, how bias is measured and why the common fairness metrics conflict, what NYC Local Law 144, the EU AI Act, Colorado and US federal agencies require as of November 2026, and an eight-step prevention program.
Reviewed 5 Nov 2026 · next review by 4 Jan 2027 · 36 sources · report an error
What is AI bias?
AI bias (also called algorithmic bias) is a systematic error in an AI system that produces worse outcomes for some people than for others, often along lines such as race, sex, age or disability. “Systematic” is the key word: the error repeats in the same direction, rather than appearing at random.
The most cited reference is NIST Special Publication 1270, “Towards a Standard for Identifying and Managing Bias in Artificial Intelligence”, published in March 2022 [1]. Its main point is that bias is not only a data problem. NIST says AI bias “extends beyond computational algorithms and models”, and that human, systemic and institutional factors are significant sources that are often overlooked [1]. In its announcement, NIST’s lead researcher on AI bias said organizations “often default to overly technical solutions” [2].
The NIST AI Risk Management Framework, released on January 26, 2023 for voluntary use [28], lists “fair, with harmful bias managed” as one of seven characteristics of trustworthy AI [29]. ISO/IEC TR 24027:2021, a technical report on bias in AI systems and AI-aided decision making, describes measurement and assessment methods across the whole lifecycle, from data collection to use [27]. Neither is binding. The laws that do bind are covered below.
What are the types of AI bias?
NIST sorts AI bias into three categories [1]. They overlap, and NIST says the combination of all three can be especially harmful [2].
| Type | What NIST means | Examples NIST gives | Where to look for it |
|---|---|---|---|
| Statistical and computational | Errors that arise when a sample is not representative of the population; systematic rather than random, and possible without any prejudice [1] | Over- and under-fitting; poorly chosen proxies; using past medical costs as a proxy for health need [1] | Training data, labels, features, model choice and evaluation |
| Human | Systematic errors in human thought, often implicit, in how people perceive information or fill in gaps [1] | Anchoring, confirmation bias and automation complacency (over-trusting automated output) [1] | Design decisions, data labeling and how staff act on outputs |
| Systemic | Results of the procedures and practices of institutions, which can happen while people simply follow existing norms [1] | Institutional racism and sexism; infrastructure not built with universal design [1] | The historical decisions the training data records, and who the system was built for |
Other labels map onto these. Sampling and proxy bias are statistical. Historical bias, where data faithfully records past discrimination, is systemic. Deployment bias is often human: NIST describes cities using a housing algorithm as an assessment tool when it was built only for pre-screening [1].
AI bias examples: documented cases
- 34.7% vs 0.8%
- error rates for darker-skinned women vs lighter-skinned men in commercial gender classifiers[3]3 commercial systems, Gender Shades, 2018
- 44.9% vs 23.5%
- Black vs white defendants who did not reoffend but were labeled higher risk by COMPAS[5]ProPublica, 7,000+ people arrested in Broward County 2013–2014, published 2016
- 1.1 billion
- applications Workday said in court filings were rejected using its tools, cited in the Mobley age-bias case[10]Workday’s filings, as reported by Proskauer, June 2025
- 17 vs 1
- instances of potential LL144 non-compliance the State Comptroller found (at least 17) at the same 32 companies where NYC’s own review found 1[17]NY State Comptroller audit, July 2023–June 2025, issued December 2, 2025
| Case | What happened | Bias type | Outcome |
|---|---|---|---|
| Gender Shades, 2018 | Researchers tested three commercial gender classification systems; error rates reached 34.7% for darker-skinned women and at most 0.8% for lighter-skinned men [3] | Statistical (unrepresentative benchmarks and training data) | Published at the first Conference on Fairness, Accountability and Transparency [3] |
| NIST face recognition test, 2019 | NIST tested nearly 200 algorithms from nearly 100 developers; false positive differentials across demographic groups existed in many, but not all [11] | Statistical | Some algorithms showed little or no differential, so results depend on the product [11] |
| US health-risk algorithm, 2019 | An algorithm used health costs as a proxy for health need. Less is spent on Black patients with the same need, so it rated them healthier; the bias more than halved the number of Black patients flagged for extra care [4] | Statistical (proxy) and systemic | Removing cost as the proxy eliminated the racial bias in predicting who needs extra care (Obermeyer et al., Science) [4] |
| COMPAS recidivism scores, 2016 | Black defendants who did not reoffend were labeled higher risk at 44.9%, against 23.5% for white defendants [5] | Statistical and systemic | The developer disputed the analysis [5]; later research showed both sides were measuring different, incompatible kinds of fairness [12][13] |
| Amazon recruiting tool, 2018 | A résumé-scoring model trained on 10 years of mostly male résumés penalized the word “women’s” and downgraded graduates of two all-women’s colleges (Reuters) [6] | Systemic (historical) and statistical | Amazon disbanded the team by early 2017; recruiters had looked at its recommendations but never relied on them alone [6] |
| EEOC v. iTutorGroup, 2023 | The EEOC alleged that application software automatically rejected women aged 55 or older and men aged 60 or older [7] | Human (a rule programmed in on purpose) | $365,000 settlement; the company denied wrongdoing [7] |
| FTC v. Rite Aid, 2023 | The FTC alleged Rite Aid used facial recognition in stores starting in 2012, producing thousands of false positive matches, without testing or measuring false-positive rates [9] | Statistical and human (low-quality images, poor training) | Five-year ban on facial recognition for security or surveillance; safeguards required for any future biometric system [8] |
| Mobley v. Workday, ongoing | A job applicant alleges Workday’s AI screening had a disparate impact by race, age and disability [10] | Alleged systemic and statistical | ADEA claims conditionally certified as a collective action on May 16, 2025; the claims are unproven [10] |
Where does bias enter an AI system?
- Who is in the data. If a group is missing or thin in the training data, the model learns less about it. Gender Shades is the textbook case [3].
- What the label measures. The health algorithm predicted cost, not need, and cost reflected unequal access to care [4]. NIST calls out poorly chosen proxies as a statistical bias [1].
- What history recorded. A model trained on past hiring learns past hiring. Amazon’s tool learned from a decade of mostly male applicants [6].
- How people respond to it. Automation complacency, where people over-trust a system’s output, turns a model’s error into a decision [1]. Rite Aid staff confronted and searched customers after false matches [9].
- Feedback loops. When a system’s outputs become its future inputs, bias can compound. The EU AI Act tells providers of high-risk systems to examine data with this in mind [19].
How is AI bias measured?
Bias is measured by splitting results by group and comparing a chosen statistic. Which statistic you choose is a policy decision, not a technical one, because the main metrics answer different questions.
| Metric | What it compares across groups | Where it shows up | Watch out for |
|---|---|---|---|
| Selection rate and impact ratio (statistical or demographic parity) | The share of each group that gets the favorable outcome, and each group’s rate relative to the highest | NYC LL144 bias audits report selection or scoring rates and impact ratios [16]; the US four-fifths rule [14] | Ignores whether outcomes are correct; small samples swing the ratio [14] |
| Error rate balance (equalized odds) | False positive and false negative rates | ProPublica’s COMPAS analysis [5]; Chouldechova’s “error rate balance” [13] | Usually cannot hold at the same time as predictive parity when base rates differ [13] |
| Predictive parity | Of the people flagged, the share who really are positive | Chouldechova found COMPAS roughly met it at high-risk cutoffs [13] | Can coexist with very different error rates by group [13] |
| Calibration within groups | Whether a score of, say, 70% means a 70% chance in every group | Counter-analyses found COMPAS calibrated within groups [12] | Incompatible with balance for both classes except in narrow special cases [12] |
| Accuracy by group | Overall error rates per group | Gender Shades [3]; NIST face recognition tests [11] | A model can be equally accurate overall and still distribute its errors unequally |
Worked example (illustrative numbers). A screening tool advances 80 of 200 men (a 40% selection rate) and 45 of 150 women (30%). The impact ratio for women is 30 ÷ 40 = 0.75. Under the federal four-fifths rule, a rate below 80% of the highest group’s rate is generally regarded as evidence of adverse impact, although smaller differences can count when they are statistically and practically significant, and large ones may not when samples are small [14]. NYC’s law requires the ratio to be published but sets no pass mark: DCWP says it “does not require any specific actions” based on results [16]. Our NYC LL144 bias audit checker runs this by category, and the LL144 bias audit FAQ covers the data rules.
What does the law require on AI bias?
| Rule | Who it covers | What it requires on bias | Status |
|---|---|---|---|
| NYC Local Law 144 | Employers and employment agencies using automated employment decision tools in New York City [15] | A bias audit within one year before use, by an independent auditor; a public summary of results; notice to candidates 10 business days before use [15][16] | Enforced since July 5, 2023 [15]; civil penalties up to $500 for a first violation and $500 to $1,500 for each later one, with each day of non-compliant use a separate violation [18] |
| EU AI Act, Article 10 | Providers of high-risk AI systems [19] | Examine training, validation and testing data for possible biases and take measures to detect, prevent and mitigate them; datasets relevant, sufficiently representative and, as far as possible, free of errors [19] | Applies to Annex III systems from December 2, 2027 and Annex I products from August 2, 2028 [21] |
| EU AI Act, Article 4a | Providers of high-risk systems; also deployers and providers of other AI systems and models [20] | Allows processing of special category data (such as ethnicity) where strictly necessary for bias detection and correction, with safeguards; creates no duty to do it [20][21] | Added by Regulation (EU) 2026/1744, in force July 27, 2026 [21] |
| Colorado SB 26-189 | Developers and deployers of automated decision-making technology used in consequential decisions [22] | Notice, explanations after adverse outcomes, data correction and human review; the reasonable-care duty to prevent algorithmic discrimination is gone [22] | Signed May 14, 2026; effective January 1, 2027 [22] |
| US Title VII and the EEOC | Employers, including when they use AI tools | Disparate impact remains part of Title VII, codified in 1991 [24][25]; the EEOC withdrew its AI guidance in January 2025 [23] and closed pending disparate-impact-only charges by September 30, 2025 [24] | EEOC moving to rescind parts of the 1978 Uniform Guidelines; not final when SHRM reported on October 30, 2026 [25] |
| US FTC Act | Businesses deploying AI that affects consumers | The Rite Aid order bans facial recognition for five years and requires safeguards for biometric systems [8] | The FTC set aside its Rytr order on December 22, 2025, citing the AI Action Plan; it says it will continue to hold accountable those who use AI to violate the law [26] |
Federal retreat is not a safe harbor. Law firms tracking the EEOC point out that private plaintiffs and states can still bring disparate impact claims [25], and Constangy advises employers to keep auditing outcomes, “especially when automated tools influence who advances” [36]. The Mobley case tests whether an AI vendor can be liable as an employer’s agent [10]. For the full NYC rules, see our NYC Local Law 144 hub; for the EU, the EU AI Act hub.
How to prevent AI bias: an eight-step program
You can’t remove all bias, for the mathematical reasons above. You can find it, decide what level is acceptable, reduce it and keep watching. This sequence follows NIST’s three challenge areas (data, testing and human factors) and the EU AI Act’s data duties [1][19].
- 01
Inventory decisions, not just models
List every AI system that makes or shapes decisions about people, such as hiring, promotion, credit or access to services. NYC regulates hiring and promotion tools [15]; Colorado’s new law covers tools that materially influence consequential decisions, and makes developers document known limitations for deployers [22].
- 02
- 03
- 04
Get the demographic data you need, lawfully
You can’t measure bias by group without group data. NYC audits must use historical data from your own use where it exists, and may not use imputed or inferred demographics [16]. In the EU, Article 4a allows special category data for bias detection under strict safeguards [20].
- 05
- 06
- 07
Monitor in production
Re-run the metrics on live data on a schedule and when the model, data or population changes. Feedback loops can make bias grow after launch [19].
- 08
Get independent review where it counts
NYC requires an independent auditor [16]. Elsewhere, an outside review supplies the objectivity internal teams lack. Our AI audit guide explains the audit types and process.
Does generative AI have bias too?
Yes. NIST’s report gives the example of a word-embedding model that, asked for doctor minus father plus mother, answered nurse [1]. Measuring bias in generated text is harder because there is no single yes-or-no outcome to count. One approach is to send prompts that differ only in a name or other attribute and compare the answers. Open-source scanners such as Giskard’s generate adversarial test suites that cover stereotypes and harmful content [33].
The EU’s Article 4a extends the bias-detection data exception to providers and deployers of AI systems and models that are not high-risk [20]. US bank model risk guidance leaves generative AI out; see our model risk management guide.
Tools and firms for detecting AI bias
Bias work splits between software you run (testing and monitoring) and firms that audit you. These are examples from our directory, described from their own or public-sector pages; we have not tested them. For ranked comparisons, see AI bias detection tools and AI bias audit firms.
- 01
Best for: NYC bias audits with ongoing monitoring
The UK government’s AI assurance catalogue describes its NYC audits as computing impact ratios for race/ethnicity and sex categories, including intersectional groups, splitting continuous scores at the median to get scoring rates, and highlighting exceptions to the four-fifths threshold. Outputs include a hosted summary of results, an internal report and a dashboard for ongoing monitoring.[30]
- 02
Best for: Fairness monitoring of deployed models
IBM’s documentation says it monitors model fairness across multiple features, tracks drift, and records fairness and drift details in factsheets for each AI use case. It also evaluates generative AI prompt templates.[31]
- 03
Best for: Bias and drift monitoring for predictive models and agents
Its governance page says it monitors predictive models and agents for drift, bias and performance degradation, aims to detect and prevent unfair outcomes across protected attributes, and provides audit evidence.[32]
- 04
Best for: Open-source testing of LLM applications
The open-source library’s version 3 scanner builds adversarial test suites covering prompt injection, harmful content and stereotypes. Its bias scan for tabular models is in version 2, which is no longer actively maintained.[33]
- 05
Best for: Bias audits when you don’t hold demographic data
Offers algorithmic audits and quantitative bias audits on its Pilot platform, with inference of race, ethnicity and gender where clients lack it, plus NYC Local Law 144 audits for employers or vendors.[34]
- 06
Best for: Independent NYC Local Law 144 audits
Describes an independent third-party bias audit in three phases (scoping, review, reporting), producing a public summary for each tool where the law requires one. It says audits typically take two to three weeks once documents are in.[35]
Frequently asked questions
+What is AI bias in simple terms?
AI bias is when an AI system keeps getting things wrong, or treating people worse, for some groups than for others. NIST says it can come from the data and algorithms, from the people who build and use the system, or from the institutions around it [1].
+What are the three types of AI bias?
NIST SP 1270 names systemic bias (from institutions’ procedures and practices), statistical and computational bias (from unrepresentative samples and modeling choices) and human bias (systematic errors in human thought, such as automation complacency) [1].
+What is an example of AI bias?
A widely used US health algorithm predicted health costs as a stand-in for health need. Because less was spent on Black patients with the same need, it rated them as healthier, and the bias more than halved the number flagged for extra care [4]. Changing the target removed the bias [4].
+Can AI bias be completely eliminated?
Not on every measure at once. Researchers have proved that common fairness definitions, such as calibration and equal error rates, cannot all hold when groups have different base rates, except in narrow special cases [12][13]. The practical goal is to pick a definition, measure it, reduce harm and monitor.
+How do you test an AI model for bias?
Split the model’s results by group (and by intersections of groups), then compare a chosen metric such as selection rates, error rates or calibration. NYC Local Law 144 audits, for example, must report selection or scoring rates and impact ratios by sex, race/ethnicity and intersectional categories [16].
+What is the four-fifths rule?
Under the 1978 federal Uniform Guidelines, a selection rate for any race, sex or ethnic group below four-fifths (80%) of the highest group’s rate is generally regarded as evidence of adverse impact [14]. The EEOC is moving to rescind parts of those guidelines, but disparate impact remains part of Title VII [25].
+Is AI bias illegal?
Bias is not itself a legal term; discrimination is. In the US, disparate impact claims can still be brought by private plaintiffs and states, whatever the federal enforcement priorities [25]. AI-specific rules add duties, such as NYC’s bias audits [15] and the EU AI Act’s data rules for high-risk systems [19].
+Does NYC Local Law 144 require you to fix bias found in an audit?
No. DCWP says the law does not require any specific actions based on bias audit results, though employers must still comply with federal, state and city anti-discrimination laws [16]. It does require the audit, a published summary and notice to candidates [15].
+Who is liable for AI bias, the vendor or the employer?
Usually the employer that uses the tool, which is why one law firm warns that outsourcing technology does not outsource risk [36]. In Mobley v. Workday the plaintiff argues the vendor can also be liable as the employer’s agent; the case was conditionally certified as a collective action in May 2025 and is unresolved [10].
Sources
- [1]NIST, SP 1270: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022-03).
- [2]NIST, There’s more to AI bias than biased data, NIST report highlights (2022-03-16).
- [3]Proceedings of Machine Learning Research (Buolamwini and Gebru), Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification (2018).
- [4]UC Berkeley Center for Healthcare Marketplace Innovation, Dissecting racial bias in an algorithm used to manage the health of populations (Obermeyer et al., Science) (2019-10-25).
- [5]ProPublica, Machine Bias (2016-05-23).
- [6]Reuters (via The Irish Times), Amazon scraps secret AI recruiting tool that showed bias against women (2018-10-10).
- [7]Greenberg Traurig, EEOC secures first workplace artificial intelligence settlement (2023-08-23).
- [8]Federal Trade Commission, FTC v. Rite Aid Corporation (case record) (2024-03-08).
- [9]Polsinelli (The National Law Review), FTC Settlement with Rite Aid Mandates AI System Assessment (2023-12-22).
- [10]Proskauer, AI bias lawsuit against Workday reaches next stage as court grants conditional certification of ADEA claim (2025-06-11).
- [11]NIST, NIST releases first-of-its-kind report detailing demographic differences in face recognition algorithms (2019-12-19).
- [12]arXiv (Kleinberg, Mullainathan and Raghavan), Inherent Trade-Offs in the Fair Determination of Risk Scores (2016-11-17).
- [13]arXiv (Chouldechova), Fair prediction with disparate impact: A study of bias in recidivism prediction instruments (2017-02-28).
- [14]Cornell Law School Legal Information Institute, 29 CFR § 1607.4: Information on impact (four-fifths rule).
- [15]NYC Department of Consumer and Worker Protection, Automated Employment Decision Tools (AEDT).
- [16]NYC Department of Consumer and Worker Protection, Automated Employment Decision Tools: Frequently Asked Questions (2023-06-29).
- [17]Office of the New York State Comptroller, Enforcement of Local Law 144: Automated Employment Decision Tools (2025-12-02).
- [18]intro.nyc (NYC Council legislation tracker), Local Law 144 of 2021 (Int. No. 1894-A): text, including § 20-872 penalties.
- [19]European Commission (AI Act Service Desk), AI Act Article 10: Data and data governance.
- [20]European Commission (AI Act Service Desk), AI Act Article 4a: Processing of special categories of personal data for bias detection and correction.
- [21]Cuatrecasas, Digital Omnibus on AI has been published (2026-07-24).
- [22]Holland & Knight, Colorado Governor Signs SB 26-189 (2026-05-18).
- [23]K&L Gates, The Changing Landscape of AI: Federal Guidance for Employers Reverses Course With New Administration (2025-01-31).
- [24]Franczek, EEOC to close all pending disparate impact investigations by September 30, 2025 (2025-09-26).
- [25]SHRM, EEOC moves to roll back longstanding hiring guidelines (2026-10-30).
- [26]Federal Trade Commission, FTC Reopens and Sets Aside Rytr Final Order in Response to the Trump Administration’s AI Action Plan (2025-12-22).
- [27]ISO, ISO/IEC TR 24027:2021: Bias in AI systems and AI aided decision making (2021-11).
- [28]NIST, AI Risk Management Framework (2026-08-13).
- [29]Davis Wright Tremaine, NIST Releases Final Risk Management Framework for Developing Trustworthy AI (2023-01-31).
- [30]UK Department for Science, Innovation and Technology, Holistic AI: NYC Bias Audits (Portfolio of AI assurance techniques) (2023-09-19).
- [31]IBM, IBM watsonx.governance use case (documentation).
- [32]Fiddler AI, Fiddler for AI Governance, Risk Management, and Compliance (GRC).
- [33]Python Package Index (Giskard), giskard 3.0.1 (2026-10-02).
- [34]ORCAA, O’Neil Risk Consulting & Algorithmic Auditing: services.
- [35]BABL AI, NYC Bias Audit.
- [36]Constangy, Brooks, Smith & Prophete, Beyond the Uniform Guidelines: Why employers should continue monitoring for disparate impact (2026-07-30).
How we keep this page current
Every claim links to its source. We re-check this page at least every 60 days, and sooner when a standard, law or vendor changes; vendor names, acquisitions and shutdowns update automatically from our directory. Vendors cannot pay to be included or ranked.