What Is AI Bias and Why Does It Happen?
AI bias is the systematic production of outcomes that unfairly advantage or disadvantage particular groups of people — typically along dimensions such as race, gender, age, disability or socioeconomic status. It is not primarily a technical glitch or coding error; it is a reflection of biases embedded in the data used to train AI systems, the choices made in system design, and the real-world contexts into which systems are deployed. Understanding this is crucial: fixing AI bias is not simply a matter of better algorithms.
The most common sources of AI bias are: historical data bias (if past hiring decisions favoured certain demographics, a model trained on that data will perpetuate those patterns); measurement bias (using proxy variables that correlate with protected characteristics — e.g. using postcode as a credit risk signal); feedback loops (predictive policing AI concentrates surveillance in certain areas, generating more arrests there, which trains future models to over-police the same areas); and selection bias in training data (face recognition systems trained predominantly on lighter-skinned faces performing poorly on darker-skinned individuals).
Real-World AI Bias Cases in the UK
The UK has experienced several high-profile AI fairness failures that have shaped regulatory attitudes and public discourse. The most discussed: the A-level algorithm (2020), in which a statistical model was used to assign grades during the pandemic in a way that systematically downgraded students from state schools and schools with fewer historical data points, disproportionately affecting students from lower socioeconomic backgrounds. The backlash — and U-turn — resulted in billions in additional educational support costs and significant erosion of public trust in algorithmic assessment.
In financial services, the FCA has investigated several insurers and lenders for using AI models with disparate impact on customers of different ethnicities or with disabilities — often introduced unintentionally through price optimisation algorithms that detected and exploited differential price sensitivity correlated with demographic factors. The ICO's 2024 audit of recruitment AI found that 40% of the tested systems showed statistically significant disparities in shortlisting rates for candidates with non-Western names. These are not isolated incidents; they represent systemic patterns across sectors.
The Legal Landscape: Equality Act and UK GDPR
UK organisations deploying AI face obligations under two interlocking legal frameworks. The Equality Act 2010 prohibits indirect discrimination — treating people less favourably through a requirement, criterion or practice (such as an algorithmic decision) that disadvantages a group sharing a protected characteristic, where that disadvantage cannot be justified as a "proportionate means of achieving a legitimate aim." This test applies to AI systems: an AI hiring tool that produces systematically lower scores for female candidates creates indirect sex discrimination liability regardless of whether gender was an input variable.
Under UK GDPR Article 22, individuals have the right not to be subject to solely automated decisions that produce legal or similarly significant effects. Where AI decisions do affect individuals significantly, an accessible opt-out and human review mechanism must be available. The ICO's AI and Data Protection guidance (updated 2025) provides detailed compliance requirements and is, effectively, the published regulatory standard against which enforcement decisions will be made. All Data Protection Officers should be familiar with it.
How to Measure Fairness in Your AI Systems
Fairness cannot be assumed — it must be measured. The primary statistical measures of algorithmic fairness used in UK compliance contexts are: Demographic parity (are positive outcomes equally distributed across demographic groups?); Equalised odds (do true positive and false positive rates differ significantly across groups?); Calibration (are the model's confidence scores equally accurate for different groups?); and Individual fairness (are similarly situated individuals treated similarly, regardless of group membership?).
Note that these metrics can directly conflict — it is mathematically impossible to simultaneously achieve demographic parity, equalised odds and calibration in most real-world settings. Choosing which fairness criterion to optimise for is an ethical decision, not a technical one, and should involve legal, HR, business and (where applicable) affected community input. Documenting this decision — and regular re-measurement of fairness metrics as the model is used and data patterns shift over time — is essential for defensible AI fairness compliance.
Practical Mitigation Strategies
Pre-deployment mitigation: audit training data for underrepresentation and historical biases; remove proxy variables that correlate with protected characteristics (note: model debiasing is complex — removing a variable doesn't remove its predictive effect if correlates remain); conduct adversarial testing specifically for the protected characteristics relevant to your use case; document residual risks explicitly. Post-deployment: establish ongoing fairness monitoring dashboards; create an accessible complaints mechanism for individuals who believe they've been unfairly treated by an AI decision; conduct third-party audits annually for high-stakes applications.
One structural reality: AI fairness work cannot be delegated entirely to data scientists. It requires active involvement from legal, HR and business stakeholders who understand the context and consequences of AI decisions. The organisations making the most progress on AI fairness in 2026 are those that have made it a board-level accountability, not a technical footnote.
