Causality and Consequences

Learning Objectives

By completing this book, you will be able to:

1. Explain why the human brain systematically confuses correlation with causality, and how this bias manifests in the formulation of intentions and in the interpretation of AI outputs.

2. Distinguish the three levels of Pearl's causal hierarchy—association, intervention, and counterfactual—and explain why most data and most AI models operate only at the lowest level.

3. Construct and read causal diagrams (DAGs) that make explicit the causal hypotheses behind a formulation or a decision.

4. Identify confounders, mediators, and colliders in a causal chain, and understand how each one can distort conclusions.

5. Apply the analysis of second- and third-order effects to anticipate the unintended consequences of well-formulated actions.

6. Integrate causal thinking with systems thinking (Topic 1) to produce formulations of intention that survive contact with reality.

1. The Causal Brain: Why We See Causes Where There Are None

The human being is, above all, a machine for detecting causality. It is the cognitive ability that allowed us to survive: the ancestor who heard a noise in the bushes and concluded “predator” lived to reproduce; the one who needed a rigorous statistical analysis before fleeing left no descendants. Cause detection is fast, automatic, and operates in System 1. It is, in Gigerenzer's terms which you studied in Module I, an ecologically rational heuristic in environments where the consequences of a false positive (fleeing for no reason) are smaller than those of a false negative (not fleeing when there is a predator).

But that same machinery produces systematic errors when applied to complex systems, statistical data, and business or policy decisions. You already studied the fallacy of post hoc ergo propter hoc in Module II: “after this, therefore because of this.” Here we go much deeper, because causal confusion is not merely an isolated logical fallacy; it is the default mode in which the brain interprets the world.

“A marketing team launches a new advertising campaign in January. Sales rise in February. The team concludes: the campaign caused the increase. But February is always a strong month for that product for seasonal reasons. The campaign coincided with the increase; it did not necessarily cause it. Nevertheless, the causal narrative is now installed in the brain of the whole team, and from now on they will make decisions based on it: more budget for similar campaigns, promotion for the person responsible, replication of the strategy. All of it on an unverified causal foundation.”

Daniel Kahneman, in Thinking, Fast and Slow, documents how System 1 constructs causal narratives automatically. When you see two events in sequence, the brain inserts a causal arrow between them without asking permission from your critical judgment. It is the same WYSIATI mechanism you studied in Module I: the brain constructs the most coherent story possible with the available information, and narrative coherence feels like causal truth.

For Intention Architecture, this has a direct and serious implication: users formulate their intentions on causal hypotheses they rarely verify. When someone says “I need to improve engagement because our content isn't attractive enough,” they are making a causal claim (LOW ENGAGEMENT IS CAUSED BY UNATTRACTIVE CONTENT). But have they verified that causality? Or could the low engagement be caused by the posting schedule, the platform's algorithm, a demographic shift in the audience, or market saturation? An intention architect must make the implicit causal hypothesis visible before accepting it as a premise.

2. Judea Pearl's Causal Revolution

Judea Pearl, a computer scientist and artificial intelligence researcher, led one of the most important intellectual revolutions of the twentieth century, although its impact has not yet fully permeated beyond academia. His fundamental contribution, developed in Causality: Models, Reasoning, and Inference (2000) and popularized in The Book of Why (2018, with Dana Mackenzie), is a mathematical formalization of causal reasoning that solves problems classical statistics could not touch.

To understand the magnitude of what Pearl achieved, you have to understand the problem he solved. Classical statistics—which dominates most data analysis, business reporting, and machine learning models—operates exclusively with correlations. It can tell you that A and B occur together, how often, and how strongly. But it cannot tell you whether A causes B, B causes A, or both are caused by a third factor C. And here is the critical point: decisions require causality, not correlation.

“You observe that patients taking a certain medication have worse outcomes than those who do not take it. Is the medication harmful? The correlation says yes. But the causality could run the other way: perhaps the medication is prescribed to the sickest patients, and it is the severity of their condition—not the medication—that causes the worse outcomes. The same correlation, two opposite causal conclusions, and two radically different decisions: ban the medication, or prescribe it more widely.”

2.1 The Ladder of Causation

Pearl proposes that causal reasoning operates at three hierarchical levels, each more powerful than the last. He calls it the “Ladder of Causation,” and his central thesis is that humans can climb all three rungs but machines (including current AI models) are trapped on the first.

Level 1: Association — What do I observe? At this level, you observe correlations. You see that A and B occur together. You can predict B when you see A. All of classical statistics, supervised machine learning, and most business data analysis operate at this level. It is the world of “customers who buy X also buy Y,” of credit scoring, of Netflix's recommendation system.

“A recommendation algorithm observes that people who buy diapers also buy beer. Pure association. It can use this correlation to place the beer near the diapers in the supermarket. But it cannot answer the causal question: does buying diapers cause the desire for beer? Or is it that the same demographic factor (young parents doing their full weekly shop) explains both purchases?”

Level 2: Intervention — What will happen if I do something? At this level, you no longer merely observe; you act. You intervene in the system and predict the consequences. The question is not “what occurs together?” but “what will happen if I change X?” This level requires a causal model: you need to know not only that A and B are correlated but how A affects B and through what mechanism.

“A hospital observes that patients on supplemental oxygen have higher mortality (Level 1: correlation). Should it stop giving oxygen? The Level 2 question is different: what will happen to mortality if I INTERVENE by withdrawing the oxygen? A causal model reveals that oxygen is given to the sickest patients (a confounder), and withdrawing it would INCREASE mortality. The correlation says one thing; the causal intervention says the opposite.”

Level 3: Counterfactual — What would have happened if I had acted differently? The highest level. It requires imagining alternative worlds: given that X already happened, what would have happened if instead of X I had done Y? This is the reasoning of blame, responsibility, regret, and improvement. It is also the level that distinguishes human intelligence from current artificial intelligence.

“A doctor prescribed treatment A and the patient died. The counterfactual question is: if he had prescribed treatment B, would the patient have survived? You cannot observe this; the patient has already died. You cannot experiment; you cannot rewind time. But you can reason counterfactually using a causal model that includes all the relevant variables. Pearl formalized how to do this mathematically.”

3. Causal Diagrams: Making the Invisible Visible

One of the most powerful tools Pearl introduced into causal thinking is the Directed Acyclic Graph (DAG): a causal diagram that makes explicit the causal hypotheses behind any analysis. A DAG is simply a set of variables connected by arrows indicating the assumed causal direction: if an arrow runs from A to B, you are asserting that A causes B (or contributes causally to B).

The power of the DAG lies not in what it says but in what it forces you to make explicit. When you draw a causal diagram, you are publicly declaring your causal assumptions. And once they are explicit, they can be questioned, tested, and corrected. While they remain implicit—as they do in most decisions and formulations of intention—they are invisible and unchallengeable.

“Suppose you want to understand why staff turnover at your company is high. You could draw a DAG: Salary → Satisfaction → Retention. Also: Culture → Satisfaction. And: Workload → Satisfaction. And perhaps: Labor Market → Retention (directly, without passing through satisfaction). Drawing the DAG forces you to decide: does salary affect retention directly, or only through satisfaction? Does culture affect workload, or are they independent? Each arrow is a hypothesis. Each absent arrow is one too.”

3.1 Three Fundamental Causal Structures

Pearl identifies three basic structures that appear in every causal diagram. Understanding them is essential because each one has different implications for analysis and decision-making.

The Confounder (Fork): A variable C causes both A and B. Result: A and B are correlated, but not because one causes the other—rather because they share a common cause. This is the most frequent causal trap and the source of countless decision errors.

“There is a correlation between ice cream consumption and drownings. Does ice cream cause drownings? Obviously not. The confounder is temperature: heat causes both more ice cream consumption and more visits to pools and beaches, where drownings occur. If you do not control for the confounder, the correlation deceives you. In less obvious contexts, confounders go unnoticed: a correlation between hours of use of an AI tool and productivity could be confounded by technical competence (the most competent use the tool more AND are more productive).”

The Mediator (Chain): A causes M, which in turn causes B. M is the mechanism through which A produces its effect on B. Identifying the mediator is crucial for understanding why something works, not merely that it works.

“A physical exercise program (A) improves cognitive performance (B). What is the mediator? It could be: exercise increases cerebral blood flow (M1), which improves neuronal oxygenation, which improves performance. Or it could be: exercise reduces stress (M2), which reduces emotional interference in cognition. Knowing the mediator matters because if you want to replicate the effect without exercise, you need to know which part of the chain is the active one. If it is blood flow, perhaps other methods that increase it would work; if it is stress reduction, meditation could achieve the same thing.”

The Collider: Both A and B cause C. Here comes the paradox: A and B are not naturally correlated, but if you control for C (if you select or condition on C), A and B become artificially correlated. This is called collider bias and it is one of the subtlest and most dangerous statistical traps.

“A scholarship program selects students who are intelligent OR athletic (or both). Among the scholarship holders (conditioning on C = scholarship), intelligence and athleticism will appear to be negatively correlated: the most intelligent recipients will tend to be less athletic, and vice versa. But this does not mean intelligence and athleticism are opposites in the general population—the negative correlation is an artifact of having selected on a criterion that either one satisfies. If you base decisions on data from scholarship holders, you will believe you must choose between brains and muscle, when in the general reality no such trade-off exists.”

4. Second- and Third-Order Effects

One of the most powerful reasons for combining causal thinking with systems thinking is the anticipation of indirect effects. A first-order effect is the direct and immediate consequence of an action. A second-order effect is the consequence of the consequence. A third-order effect is the consequence of the consequence of the consequence. And so on.

Most people—and most formulations of intention—stop at the first order. Systemic and causal thinking demands going at least to the second order, and preferably to the third.

“First-order effect: a city builds a new highway to reduce traffic. Second-order effect: the highway makes it easier to live further from the center, so more people move to the suburbs and buy cars. Third-order effect: the suburbs grow, generate more traffic, and the highway becomes congested again—now with more cars than before. This phenomenon, documented by the urbanist Anthony Downs as ‘induced demand,’ demonstrates that the solution to traffic is NOT more roads but better public transport and land use. But you only see it if you think in second and third order.”

4.1 Chesterton's Principle

G.K. Chesterton proposed a principle that is essentially a second-order causal principle: before removing a fence, understand why it was put there. If you do not understand the reason for its existence, you cannot predict the consequences of its removal.

“A software company decides to eliminate the weekly status meetings because ‘they are a waste of time.’ First-order effect: everyone recovers an hour a week. Second-order effect: without a regular synchronization point, teams begin to drift out of alignment, problems are not detected early, and dependencies between projects produce blockages. The meeting was not ‘a waste of time’—it was a feedback mechanism (a balancing loop) that maintained the coherence of the system. Removing it without understanding its systemic function produced effects worse than the problem it was trying to solve.”

Chesterton's Principle is a direct application of causal thinking: before intervening in a system, map the existing causal chains. Every element of the system—even those that appear useless or inefficient—may be performing a function that only becomes visible when you remove it.

4.2 Goodhart's Law and the Cobra Effect

Charles Goodhart, a British economist, formulated a law that is a classic example of a second-order effect: “When a measure becomes a target, it ceases to be a good measure.” This occurs because once an indicator is set as a goal, people optimize for that specific indicator—not for the real outcome the indicator was meant to measure. The indicator was a proxy; by turning it into an objective, the proxy divorces itself from what it represented.

“A call center measures ‘customer satisfaction’ with a post-call survey. The operators discover that if they transfer difficult customers (who would give low ratings), their metrics improve. The satisfaction metric rises, but real satisfaction falls—difficult customers are bounced from agent to agent without resolution. The measure has ceased to measure what it claimed to measure.”

The “cobra effect” is the extreme version: the intervention produces exactly the opposite of what was desired. The name comes from a real case during British rule in India: the government offered a reward for each dead cobra in order to reduce the snake population. Second-order result: people began breeding cobras in order to kill them and collect the reward. When the government cancelled the program, the breeders released the cobras, and the population rose beyond its original level.

5. The do-Operator: Intervention vs. Observation

One of Pearl's most important formalizations is the do-operator, which mathematically distinguishes between observing that something occurs and making something occur. This distinction, which seems subtle, is in reality the frontier between correlation and causality.

When you observe, you are gathering information about the system as it is. Confounders are active, sample selections are present, and what you see reflects all the interconnections of the system. When you intervene (do), you are breaking the natural causal chain: you fix a variable at a specific value, cutting the arrows that enter it, and observe what happens to the rest of the system.

“You observe that people carrying umbrellas have a higher probability of getting wet. Spurious correlation: carrying an umbrella does not cause getting wet. The confounder is the rain, which causes both. Now apply the do-operator: force random people to carry umbrellas (do(umbrella = yes)), whether it rains or not. If getting wet were independent of carrying an umbrella under this intervention, you confirm that the correlation was spurious. If not, there is a direct causal relationship. The do-operator breaks the link between the rain and the umbrella, because it is no longer the rain that determines whether you carry an umbrella—you determine it by intervention.”

The do-operator is the formal foundation of randomized controlled trials (RCTs, or A/B tests in the business world). When you randomly assign participants to a treatment, you are applying the do-operator: you break the connection between the treatment and the confounders, isolating the causal effect.

5.1 When You Cannot Intervene

But here is the critical point: there are countless situations where you cannot apply the do-operator directly. You cannot randomly assign people to smoke or not smoke in order to measure the effect of tobacco. You cannot randomly assign countries to different economic policies. You cannot randomly assign a company's customers to receive terrible service in order to measure its effect on retention.

Pearl developed an arsenal of techniques for estimating causal effects from observational data, provided you have a correct causal diagram. These techniques—backdoor adjustment, the front-door criterion, instrumental variables—allow you to simulate the effect of an intervention without physically performing it. But they all depend on one fundamental requirement: you must explicitly declare your causal model. And that model may be wrong.

“A company wants to know whether its mentoring program causes better retention. It cannot randomly assign mentoring (employees choose to participate). It observes that participants stay longer. But is it the mentoring, or is it that the most committed employees (who would have stayed anyway) are the ones who choose to participate? Prior commitment is a confounder. If the company has data on commitment before participation, it can adjust for that confounder and estimate the causal effect. If it does not have that data, the estimate is impossible without a controlled experiment.”

6. Causality and Artificial Intelligence

The relationship between causality and AI is one of the most critical—and least understood—topics in current technology. Pearl has been explicit about it: today's machine learning models, however sophisticated, operate at Level 1 of the causal ladder. They are extraordinarily powerful correlation machines, but they do not reason causally.

A language model such as GPT or Claude can predict which word comes after another with astonishing precision. It can generate text, code, analysis. But when it asserts that “A causes B,” it is reproducing patterns from its training, not performing causal inference. It can tell you that smoking is associated with cancer because it has read this in millions of texts, but it cannot derive that causal conclusion from the data by itself.

“If you ask an AI model, ‘Will reducing the price by 20% increase our sales?’, the model can answer intelligently based on general patterns of basic economics. But it cannot model YOUR specific system with ITS confounders, mediators, feedback loops, and delays. If your customers value exclusivity (the brand is perceived as premium), reducing the price could REDUCE sales—a second-order effect that the general correlation ‘lower price → more sales’ does not capture.”

6.1 What AI Can and Cannot Do Causally

What it CAN do: Identify correlations in large data sets, generate causal hypotheses for humans to evaluate, simulate scenarios under explicit causal assumptions supplied by the human, flag correlations that could be mistaken for causality, and run sensitivity analyses to evaluate the robustness of causal conclusions.

What it CANNOT do: Determine causal direction from observational data alone, identify confounders that are not in the data, distinguish a mediator from a confounder without a causal model provided by the human, predict the effects of interventions never carried out without explicit causal assumptions, and reason counterfactually on its own.

This limitation is not a bug that will be resolved with more data or better hardware. It is a fundamental theoretical limitation: causality is not in the data; it is in the model you impose on the data. Pearl insists that no algorithm can discover causality from purely observational data without additional assumptions. Data is Level 1; decisions require Level 2. The human must provide the bridge.

7. Practical Application: Causal Thinking in Action

7.1 Case Study: The Campaign That ‘Worked’

Valentina, CMO of an e-commerce company, asks an AI system: “Analyze why our Q3 email marketing campaign was so successful and recommend how to replicate those results in Q4.”

The request seems reasonable. But it contains at least three unverified causal assumptions:

Causal assumption 1: The email campaign was the cause of the Q3 success (and not seasonality, a competitor shutting down, a Google algorithm change, or any other factor).

Causal assumption 2: The specific elements of the campaign (subject line, content, segmentation, timing) were causally responsible, rather than unrepeatable contextual factors.

Causal assumption 3: Q4 conditions are similar enough to Q3's for the same cause to produce the same effect.

A system with good intention architecture would respond:

“Before analyzing how to replicate the campaign, we need to verify that the campaign really caused the results. Q3 had 35% more sales than Q2, but it also coincided with: (1) the collapse of your main competitor, (2) the back-to-school season, and (3) a change in Google's algorithm that improved your organic ranking. These are potential confounders. I can analyze correlations in the data, but to establish causality we would need: did you run a control group that did not receive the emails? Do you have data on similar cohorts in Q3 of previous years? Without this evidence, any ‘replication’ will be based on an unverified causal hypothesis.”

7.2 A Causal Analysis Protocol for Intention

Based on everything studied, we can articulate a causal analysis protocol that complements the integrated protocol of Module II:

1. Make the causal hypothesis explicit. Every formulation of an intention contains an implicit causal claim (‘if I do X, Y will happen’). Make it explicit. Draw the DAG.

2. Identify the causal level. Is the evidence supporting your hypothesis Level 1 (association), Level 2 (intervention), or Level 3 (counterfactual)? Does the level of the evidence match the level of the decision?

3. Look for confounders. What third variables could explain the observed correlation without the assumed causal relationship?

4. Trace second- and third-order effects. What will happen AFTER the action ‘works’? Which feedback loops will it activate? Which stocks will it affect?

5. Apply Chesterton's Principle. What function does the current state serve that this intervention might destroy?

6. Design the minimal test. What is the smallest experiment that could refute your causal hypothesis before committing resources?

8. Glossary of Key Terms

9. Final Reflection and Self-Assessment

Before advancing to the next topic (Trade-offs and Decisions Under Uncertainty), verify that you can apply causal thinking not only as academic analysis but as a discipline of formulation.

10. Bibliography and Recommended Readings

Essential Readings

• Pearl, J. (2000). Causality: Models, Reasoning, and Inference. Cambridge University Press. [2nd ed. 2009]

• Pearl, J. & Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.

• Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.

Complementary Readings

• Hernán, M. & Robins, J. (2020). Causal Inference: What If. Chapman & Hall/CRC. [Available free online]

• Morgan, S. & Winship, C. (2015). Counterfactuals and Causal Inference. 2nd ed. Cambridge University Press.

• Angrist, J. & Pischke, J. (2015). Mastering 'Metrics: The Path from Cause to Effect. Princeton University Press.

• Meadows, D. (2008). Thinking in Systems: A Primer. Chelsea Green Publishing. [Connects with Topic 1]

• Sterman, J. (2000). Business Dynamics: Systems Thinking and Modeling for a Complex World. McGraw-Hill. [Connects with Topic 1]

Next Topic

The next book in this series addresses Trade-offs and Decisions Under Uncertainty: how to make decisions when you cannot optimize all variables simultaneously, when information is incomplete, and when the future is fundamentally uncertain. You will study Taleb's concepts of fragility and antifragility, decision theory, and how to design robust formulations that survive surprise. You will connect the causal thinking you have just studied with the irreducible uncertainty that every complex system contains.