Algorithmic Biases

Learning Objectives

By completing this book, you will be able to:

1. Identify the three principal channels through which bias enters the ML pipeline: historical bias, representation bias, and measurement bias.

2. Analyze how biased algorithms create self-reinforcing cycles that amplify existing inequalities, applying O'Neil's concept of “weapons of math destruction.”

3. Explain why the multiple mathematical definitions of fairness are mutually incompatible, and why choosing between them is an ethical decision, not a technical one.

4. Apply Crawford's framework to analyze how AI systems insert themselves into power structures, and how decisions about classification, optimization, and data are political decisions.

5. Design bias audits for AI systems that interact with the formulation and execution of intentions, integrating tools from all the previous modules.

6. Assume the responsibility of the intention architect: that decisions about fairness in AI systems cannot be delegated to the algorithm; they must be governed with informed human judgment.

1. The Anatomy of Algorithmic Bias

Algorithmic bias is not born of the programmer's malice or of a defect in the code. It is born of a chain of human decisions—each apparently reasonable—that, combined, produce systematically unjust results. Those decisions include what data to collect, how to label it, which variable to optimize, how to define success, and whom the predictions affect. Each is an act of framing (Module II) that includes and excludes, emphasizes and minimizes—and each exclusion can become an injustice amplified at scale.

1.1 Historical Bias

Historical bias occurs when the training data reflect past discrimination. The system does not invent the discrimination; it inherits and perpetuates it. If certain groups historically had less access to credit, a model trained on data from past approvals will learn that belonging to that group is a “predictor of risk”—not because it is true but because the data reflect a past policy, not an objective reality.

“Amazon developed an ML system to evaluate candidates' résumés. It trained the system on historical hiring data: 10 years of human decisions. Since most of the engineers hired historically were men, the system learned that being male was a predictor of success. It penalized résumés mentioning women's colleges and downgraded profiles containing the word ‘women's.’ Amazon cancelled the project. The system was not sexist; the data were sexist. But the result was the same.”

1.2 Representation Bias

Representation bias occurs when the training data overrepresent certain groups and underrepresent others. The model learns to work well for the majority and badly for the minorities—not by design but by statistics. If 80% of the data come from one group, the system will optimize its performance for that group because that is how it minimizes total error. The minority is statistically invisible.

“Facial recognition systems have shown significantly higher error rates for people with darker skin and for women than for men with lighter skin. Joy Buolamwini's research demonstrated that the datasets used to train these systems overrepresented male faces with lighter skin. The system was not designed to discriminate; it was trained with data that did not represent the population equitably. The result is the same: a technology that works better for some than for others, deployed as though it worked equally for all.”

1.3 Measurement Bias

Measurement bias occurs when the variables used as proxies do not adequately capture the concept of interest. It is Goodhart's Law (Module III) applied to fairness: when you measure a proxy instead of the real concept, you optimize the proxy at the expense of what matters.

“A health system used historical healthcare spending as a proxy for medical need in order to prioritize patients. It seemed reasonable: sicker patients spend more on health. But it did not consider that certain groups have systematically less access to the healthcare system and, therefore, spend less—not because they are healthier but because they face more barriers. The proxy ‘spending’ does not measure ‘need’; it measures ‘access.’ The result: the system underestimated the medical need of patients from disadvantaged groups and moved them down the priority queue.”

2. Weapons of Math Destruction

Cathy O'Neil, a mathematician and data scientist, coined the term “weapons of math destruction” (WMDs) to describe algorithms that meet three conditions: they are opaque (the people affected cannot see how they work), they operate at scale (they affect millions), and they create harm (they produce unjust or destructive results). When all three conditions are met, the algorithm becomes an instrument of systemic harm that reinforces itself.

The mechanism of self-reinforcement is a positive feedback loop (Module III): the algorithm produces an unjust result, that result becomes data for the next training cycle, and the system learns to reproduce the injustice with greater confidence. It is the cobra effect (Module III) applied to fairness: the intervention designed to improve a situation worsens it through second-order dynamics nobody anticipated.

“A predictive policing algorithm identifies certain neighborhoods as ‘high risk’ based on historical arrest data. The police send more patrols to those neighborhoods. More patrols produce more arrests. More arrests confirm the algorithm's prediction. The algorithm reinforces its classification. More patrols. More arrests. The cycle perpetuates itself. Does the neighborhood have more crime, or does it have more surveillance? The data cannot distinguish between the two—but the algorithm treats them as if they were the same thing. Historical bias becomes a self-fulfilling prophecy.”

For Intention Architecture, O'Neil poses a question that connects directly with Module II: is the intention behind the algorithm correctly formulated? The predictive policing system has a declared intention (“reduce crime”) and an operational formulation (“predict where arrests will occur”). But arrests are not the same thing as crime—arrests are a measure of police activity, not of criminal activity. The intention is badly formulated at the root: the end is public safety; the chosen means (predicting arrests) is a biased proxy that reinforces what it purports to solve.

3. AI as a Political Instrument

Kate Crawford, in Atlas of AI, broadens the perspective: algorithmic biases are not merely correctable technical errors; they are manifestations of power structures. AI is not neutral: it is a system that classifies, orders, includes, and excludes—and each of those actions is political. Who builds the system, with what data, for what purpose, and who is left outside those decisions are questions of power, not of engineering.

Crawford identifies three dimensions of algorithmic power. The power of classification: AI systems impose categories on reality—solvent/insolvent, healthy/sick, relevant/irrelevant—and those categories determine what opportunities each person receives. The power of optimization: choosing which metric to optimize is a choice of values—maximizing retention is not the same as maximizing wellbeing, although the two are sometimes presented as equivalent. And the power of infrastructure: algorithmic decisions are rendered invisible inside technical systems that appear neutral, making them harder to question.

“A language model that generates job descriptions tends to produce ‘masculine’ language for technical roles and ‘feminine’ language for caregiving roles—not because it was programmed that way but because its training corpus reflects the linguistic patterns of a society with entrenched gender roles. When a company uses that model to write job postings, corpus bias becomes recruitment bias. The amplifier metaphor (Topic 1) operates here with precision: the model amplifies pre-existing social patterns to organizational scale.”

Crawford's contribution to the course is to connect algorithmic bias with the ethics of Module VII and with the pragmatics of Module IV: the classifications an algorithm imposes are performative speech acts (Austin). When the system classifies someone as “high risk,” it does not describe a pre-existing reality; it creates a reality: that person loses access to credit, employment, or opportunities. The classification does not observe; it intervenes. And every intervention is a political act that must be governed as such.

4. The Impossibility of a Single Fairness

Here the topic reaches its most uncomfortable point. Research in algorithmic fairness has produced a result that defies intuition: there exist multiple mathematical definitions of fairness, and some of the most important are mutually incompatible. It is not that we have failed to find the correct definition; it is that no single definition can exist that satisfies all the criteria we consider just. To choose is to renounce.

4.1 Three Definitions of Fairness

Demographic parity: the system must produce the same rate of positive outcomes for all groups. If it approves 60% of one group, it must approve 60% of the others. It sounds fair. But it does not distinguish between qualified and unqualified candidates within each group—it can approve unqualified candidates from one group and reject qualified candidates from another in order to maintain parity.

Equalized odds: the system must have the same error rates for all groups—the same rate of false positives and false negatives. If it makes mistakes, it must make them equally. It sounds fair. But it can produce very different approval rates across groups if the groups have different base rates.

Calibration: when the system says “70% probability,” the real rate must be 70% for every group. The system's confidence must be equally accurate regardless of group. It sounds fair. But it is compatible with groups receiving very different approval rates and very different error rates.

The mathematical result demonstrated by Chouldechova (2017) and by Kleinberg, Mullainathan, and Raghavan (2016) is conclusive: when base rates differ between groups—and they almost always differ for historical, social, and structural reasons—these definitions cannot be satisfied simultaneously. Choosing demographic parity violates calibration. Choosing calibration violates parity. Choosing equalized odds guarantees neither of the other two. There is no technical solution because the problem is not technical: it is a conflict of values.

5. From Cognitive Biases to Algorithmic Biases

The thread connecting Module I with this topic is that algorithmic biases are frequently cognitive biases encoded, amplified, and automated. The confirmation bias of the analyst who selected the variables. The anchoring of the manager who defined the thresholds. The representativeness heuristic that led to using an inadequate proxy. The status quo bias that made nobody question the historical data. Every human bias is transmitted to the algorithm through the design decisions—and there it freezes, scales, and automates.

But there is a crucial difference: cognitive biases are individual, contextual, and correctable in real time. A human can notice their bias and change their mind. An algorithm applies its bias with mechanical consistency to every case, without fatigue, without doubt, without the possibility that the person affected can explain their particular circumstance. Scale transforms the nature of the bias: what in a human is a correctable tendency becomes, in the system, an immutable policy.

“A human credit officer with an unconscious bias might reject a qualified candidate from a minority group. But if that candidate returns with more documentation, the officer can reconsider. A credit algorithm with the same bias rejects thousands of identical candidates with no possibility of appeal, without knowing their circumstances, without any of them being able to explain why their case is different. The consistency that makes the algorithm efficient is the same consistency that makes it inhuman.”

For the intention architect, this establishes a fundamental design principle: when an algorithmic system affects people, the design must include mechanisms of appeal, explanation, and human review. The total automation of decisions with human impact—without supervision, without recourse, without transparency—is not efficiency; it is an abdication of the responsibility that Intention Architecture exists to preserve.

6. Practical Application: TalentFlow's Selection System

TalentFlow is a recruitment platform that uses ML to filter résumés. It receives 10,000 applications a month and reduces them to 200 for interviews. The system was trained on hiring data from the last five years and has 82% accuracy in predicting which candidates pass the probationary period. The company presents it as “objective, data-based selection.” An audit reveals systemic problems.

6.1 Bias Diagnosis

Historical bias: The training data reflect the decisions of the hiring managers of the last five years. If those managers had implicit preferences—for specific universities, for linear career paths, for candidates who “fit” the existing culture—the system encoded those preferences as criteria of success.

Representation bias: The company operates in technology, where certain demographic profiles are historically overrepresented. The system learned that those profiles are “normal” and everything else is deviation. It does not discriminate explicitly; it discriminates statistically.

Measurement bias: The criterion of success is “passing the probationary period.” But passing the probationary period measures compatibility with the existing culture, not the potential for contribution. It is a proxy that favors conformity over diversity of perspective.

Self-reinforcing loop: The system selects profiles similar to the historical ones. Those profiles reinforce the existing culture. The next generation of training data reflects that reinforced culture. The system progressively narrows the “ideal” profile toward homogeneity.

The system meets O'Neil's three criteria: it is opaque (candidates do not know why they were rejected), it operates at scale (10,000 applications a month), and it creates harm (it systematically excludes valuable candidates with unconventional profiles).

6.2 Redesign with Ethical Responsibility

1. Audit the historical data before training. Do they reflect equitable hiring or do they perpetuate the managers' biases? Separate the historical decisions from an independent criterion of candidate quality.

2. Define success beyond survival. Replace “passing the probationary period” with metrics of real contribution: impact on projects, innovation contributed, team satisfaction. Measure what matters, not what is easy to measure.

3. Choose a fairness criterion explicitly. Decide—with the involvement of HR, legal, and leadership—which definition of fairness to prioritize, and document the trade-offs accepted. Do not delegate this decision to the technical team.

4. Preserve human intervention. The system filters; the human decides. No candidate should be rejected exclusively by algorithmic decision. The system reduces 10,000 to 500; human recruiters decide the 200 who advance to interview.

5. Make the system appealable. Rejected candidates must be able to request human review. Automation without recourse is unacceptable when it affects life opportunities.

6.3 Derived Design Principles

Principle 1: Historical data reflect human decisions, not objective truths. Auditing the data is auditing the decisions of those who generated it.

Principle 2: The choice of the success metric is the most important decision in the design. An incorrect proxy produces a system that optimizes the wrong thing with total precision.

Principle 3: Fairness is not a technical problem with a single solution. It is a conflict of values with trade-offs that must be governed explicitly, not delegated to the algorithm.

Principle 4: When an algorithm affects people, the design must include transparency (explain why), appeal (permit recourse), and human supervision (keep responsibility where it belongs).

Principle 5: The “objectivity” of an algorithm is a framing (Module V). Algorithms encode human decisions; calling them objective is hiding responsibility behind arithmetic.

7. Glossary of Key Terms

8. Final Reflection and Self-Assessment

Before moving on to the last topic of this module (Limitations of Models), verify that you can diagnose, evaluate, and govern algorithmic biases with rigor.

9. Bibliography and Recommended Readings

Essential Readings

• O'Neil, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown.

• Crawford, K. (2021). Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press.

• Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking.

Complementary Readings

• Chouldechova, A. (2017). “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments.” Big Data, 5(2), 153–163.

• Kleinberg, J., Mullainathan, S. & Raghavan, M. (2016). “Inherent Trade-Offs in the Fair Determination of Risk Scores.” arXiv:1609.05807.

• Buolamwini, J. & Gebru, T. (2018). “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification.” Conference on Fairness, Accountability and Transparency.

• Barocas, S. & Selbst, A. (2016). “Big Data's Disparate Impact.” California Law Review, 104(3), 671–732.

Next Topic: The last book of this module addresses Limitations of Models: the specific failure modes of current AI systems—hallucination, sycophancy, prompt sensitivity, the incapacity for causal reasoning—and how the intention architect designs workflows that compensate for those limitations. With Mitchell and Russell you will complete your understanding of the tool: what it can do, where it fails, and how to use that understanding to govern without being governed.