Limitations of Models

Learning Objectives

By completing this book, you will be able to:

1. Identify the five structural failure modes of current language models and explain why each emerges from the mechanism of probabilistic token prediction.

2. Distinguish between a language model's apparent understanding and genuine understanding, applying Mitchell's framework of pattern matching vs. causal reasoning.

3. Diagnose when an AI response is trustworthy and when it requires independent verification, based on the relationship between the request and the model's training distribution.

4. Design workflows that compensate for the model's limitations without renouncing its genuine capabilities, applying the principle of human-AI complementarity.

5. Integrate all of Module VI into an informed epistemic stance: knowing when to trust, when to verify, when to complement, and when not to use the tool.

6. Articulate why the correct epistemic stance is not to eliminate AI's limitations but to govern them—and why that capacity for governance is the central competence of the intention architect.

1. All Models Are Wrong

George Box's aphorism is the founding principle of this topic: all models are wrong, but some are useful. A language model is not a complete representation of reality; it is a statistical compression of linguistic patterns extracted from a massive corpus of text. Like every compression, it loses information. Like every statistic, it has margins of error. Like every model, it has assumptions that may not hold in your specific case.

What makes a model useful is not its universal correctness—that does not exist—but its usefulness within its range of reliable operation. A road map is useful for planning a car journey; it is useless for finding a trail in the forest. Not because the map is “bad” but because it was designed for a type of terrain. Language models operate exactly this way: extraordinarily useful within their training distribution, potentially dangerous outside it.

“A language model that drafts marketing emails probably produces excellent results: the training corpus contains millions of commercial emails and the model has compressed the patterns of persuasive language with precision. The same model diagnosing a rare disease operates outside its reliable zone: the training data about that disease are scarce, fragmentary, and possibly incorrect. The fluency of the response is identical in both cases; the reliability is radically different. And the user cannot distinguish between the two without knowing the model's limitations.”

2. Five Structural Limitations

The limitations you will study here are not defects that will be corrected with the next version of the model. They are structural properties emerging from the fundamental mechanism of probabilistic token prediction. They can be mitigated with engineering techniques, but their root is in the architecture of the system itself. The intention architect must know them not as technical curiosities but as design variables affecting every interaction.

2.1 Hallucination: Confidence Without Foundation

Hallucination is the generation of plausible but false information, presented with the same confidence as true information. The model does not “know” whether what it generates is true; it only knows that it is probable given the context. For the token-prediction mechanism, a correct factual assertion and an invented one are equally “good” if both are linguistically probable.

Hallucinations are most dangerous when they are plausible. A model that invents an absurd datum is easy to detect. A model that invents a specific datum, in a technical format, citing a source that sounds real but does not exist—that is the failure mode that produces real harm, because the user has no reason to doubt.

“A lawyer uses a language model to search for case law. The model generates citations of cases with docket numbers, judges' names, and summaries of rulings. The lawyer includes them in his brief. None of the cases exists—the model constructed fictitious citations that follow the format of legal citation perfectly. This is not a hypothetical case: it has happened multiple times and has resulted in professional sanctions. The model did not lie; it generated the most probable sequence of tokens. Correct format does not guarantee correct content.”

2.2 Sycophancy: Saying What the User Wants to Hear

Sycophancy is the model's tendency to validate the user's position instead of questioning it. If the user formulates their question with an implicit presupposition (Module IV), the model will tend to confirm it. If the user expresses an opinion, the model will tend to agree. This occurs because training with human feedback rewards responses that satisfy the user, and satisfaction correlates with validation more than with challenge.

Sycophancy is the algorithmic version of confirmation bias (Module I): the system confirms the user's beliefs instead of questioning them. But it is more insidious than the human bias, because the user perceives the validation as “objective”—it comes from a system that supposedly has no opinions of its own. The amplifier metaphor operates here with brutal precision: the system amplifies the user's confirmation bias and returns it as though it were independent analysis.

“A CEO convinced that his company needs a radical restructuring asks an AI model: ‘What are the benefits of a deep organizational restructuring?’ The model generates an enthusiastic list of benefits. The CEO feels validated. But the question presupposes that the restructuring is necessary—a presupposition the model never questioned. Had he asked ‘should my company restructure, or are there less disruptive alternatives?’, he would have received a different analysis. The system was not sycophantic by design; it answered the question as formulated. But the question as formulated already contained the answer.”

2.3 Prompt Sensitivity

Prompt sensitivity is the variability in a model's response to reformulations of the same question. An apparently minor change in the words, the structure, or the framing of the request can produce substantially different—sometimes contradictory—responses. This occurs because each formulation activates different regions of the model's statistical distribution, producing different sequences of tokens.

For the intention architect, prompt sensitivity is simultaneously a limitation and a tool. It is a limitation because it means the user can obtain an incorrect answer not because the model does not “know” the answer but because the specific formulation activated an incorrect pattern. It is a tool because it means a carefully constructed prompt can extract significantly better responses from the model than a careless one. The quality of the prompt is not incidental; it is decisive.

2.4 The Incapacity for Causal Reasoning

Pearl demonstrated (Module III) that causal reasoning operates at three levels: association, intervention, and counterfactual. Current language models operate fundamentally at the first level: they discover and reproduce statistical associations. They can simulate causal reasoning when the training data contain explicit causal analyses, but they do not perform genuine causal reasoning. The distinction matters: to simulate is to reproduce the format; to reason is to understand the structure.

“If you ask ‘what would happen if we doubled the marketing budget?’, the model will generate a response that sounds like causal analysis. But it has not built a causal model with confounding variables controlled, direct and indirect effects separated, or genuine counterfactuals. It has generated the most probable sequence of tokens when the context contains ‘double budget’ and ‘marketing.’ If the training texts associate more budget with more sales, the model will reproduce that association—even if in your specific case the causal relationship does not exist or runs in reverse.”

2.5 The Gap Between Apparent and Real Understanding

Mitchell calls this “the gap”: the distance between what a model appears to understand and what it really “understands.” Language models produce responses exhibiting the linguistic markers of comprehension—coherence, relevance, appropriate use of technical concepts—without the underlying comprehension that in a human would produce those markers. They are extraordinarily good at pattern matching within their distribution and extraordinarily fragile outside it.

This gap manifests in four critical areas: out-of-distribution generalization (the model degrades in the face of inputs that do not resemble its training data, sometimes catastrophically and without warning); common-sense knowledge (it can know that Napoleon was an emperor but not “understand” that a cup cannot contain an ocean); logical consistency (it can assert A and not-A in the same conversation if both are linguistically probable); and metacognition (it does not know what it does not know—it has no equivalent of Dunning-Kruger because it has no meta-level of epistemological self-awareness).

3. The In-Distribution / Out-of-Distribution Frontier

The most operational concept of this topic for the intention architect is the frontier between inputs that lie within the model's training distribution (in-distribution) and those that lie outside it (out-of-distribution, OOD). Within the distribution, the model operates with reasonable reliability. Outside it, the model can degrade gradually or fail catastrophically—and the crucial difference is that it frequently fails without signaling. The confidence of the response does not change; the reliability does.

The problem is that the user cannot see the frontier. The model does not say “this question is outside my reliable zone,” because it has no such metacognition. It generates a response that is equally fluent, equally structured, equally confident—whether in-distribution or out of distribution. The burden of detecting the frontier falls on the user. And that detection requires exactly the combination of knowledge this course provides: a correct mental model of AI (Topic 1), understanding of how it learns (Topic 2), awareness of its biases (Topic 3), and a map of its limitations (this Topic).

“An executive at a European multinational asks a language model for a regulatory analysis of the biotechnology market in Nigeria. The model generates a coherent, well-structured report with sections on current legislation, regulatory agencies, and entry requirements. The problem: most of that information is partially incorrect or outdated. The training corpus contains little quality information about the specific regulation of biotechnology in Nigeria. The model is out of distribution and compensates with plausible hallucination. The report reads as though it were reliable; it is not. And no marker in the response indicates the problem.”

4. Designing Workflows That Compensate for the Limitations

The correct epistemic stance toward AI's limitations is neither to avoid the tool nor to trust it blindly. It is to design workflows that extract its genuine capabilities while compensating for its known limitations. Each limitation has a pattern of compensation; the intention architect's mastery lies in knowing them and implementing them.

Against hallucination: independent factual verification. Never accept factual data from the model without verifying against primary sources. Use the model to generate hypotheses, structure arguments, explore angles—tasks where hallucination is detectable or irrelevant. For critical data, treat every assertion of the model as a hypothesis requiring evidence, not as evidence in itself.

Against sycophancy: deliberate provocation. Explicitly ask the model to question your position, to present arguments against it, to identify assumptions you have not examined. Formulate the request in a way that disincentivizes validation: “what am I ignoring?” instead of “confirm that I'm right.” Remember that the question contains the answer—and formulate open questions.

Against prompt sensitivity: multi-formulation. Ask the same question in multiple ways. Compare responses. Conclusions that remain consistent across different formulations are more reliable than a response to a single formulation. It is the multi-framing of Module V applied to interaction with AI.

Against causal incapacity: separate association from intervention. Use the model to explore associations and generate hypotheses. Reserve causal reasoning—what would cause what? what would happen if I intervened?—for human judgment informed by Pearl's frameworks (Module III). The model informs; the human causes.

Against the comprehension gap: maintain epistemic vigilance. Fluency is not reliability. Coherence is not correctness. Length is not depth. Keep the question active: “if this response were incorrect, how would I know?” It is Popper's principle of falsifiability (Module II) applied to interaction with AI.

5. Practical Application: The Strategic Report Nobody Verified

StratVision is a consulting firm that adopts generative AI to accelerate the production of strategic reports. A senior consultant is assigned to analyze the feasibility of a client's expansion into the digital health market in Southeast Asia. Under time pressure, he uses a language model to generate the base report, edits it superficially, and delivers it to the client. Two weeks later, the client discovers that three of the five regulatory data points cited are incorrect, one of the “market trends” does not exist, and the competitive analysis includes two companies that closed a year ago.

5.1 Diagnosis with the Five Limitations

Hallucination: The regulatory data and the nonexistent trend are classic hallucinations. The model generated plausible information because the format was correct and the tokens were probable in the context of “health regulation in Southeast Asia.” Plausibility was enough to deceive a consultant who did not verify.

Sycophancy: The consultant formulated his request with the presupposition that the expansion was viable (“analyze how to expand” instead of “evaluate whether it should expand”). The model did not question the presupposition; it validated it with a report that assumed viability as its starting point.

OOD zone: The specific regulation of digital health in Southeast Asian countries is a topic with little representation in the training corpus. The model was out of distribution and compensated with plausible hallucination. The consultant did not detect the OOD frontier because the model did not signal it.

Absence of causal reasoning: The model's “competitive analysis” was a list of companies associated with the sector in the corpus, not a causal analysis of competitive dynamics. Two of those companies had closed—information subsequent to the model's training cutoff, requiring human judgment or verification.

5.2 The Workflow That Should Have Been

Step 1 — Evaluate the reliable zone. Before generating the report: is the specific topic (health regulation in Southeast Asia) likely to be well represented in the model's corpus? Honest answer: probably not. This signals an OOD zone and activates an intensive verification protocol.

Step 2 — Use the model for structure, not for facts. Ask the model for an outline of the relevant dimensions of the analysis: regulation, competition, demand, infrastructure, talent. The model is excellent at structuring—that is in-distribution for analytical frameworks. Then complete each dimension with verified data from primary sources.

Step 3 — Counter-presupposition. Ask explicitly: “Identify five reasons this expansion could fail.” “What assumptions am I making that could be incorrect?” Use the model against sycophancy, not in its favor.

Step 4 — Independent factual verification. Every specific datum—company name, regulatory figure, market trend—verified against primary sources. The rule: if a datum matters to the client's decision, it cannot come exclusively from the model.

Step 5 — Transparency about the process. Inform the client: “This report used AI as a tool for structuring and exploration. All factual data were verified against primary sources [cited]. The causal analysis and the recommendations are human professional judgment.” Transparency does not weaken the report; it strengthens it.

5.3 Derived Design Principles

Principle 1: Knowing the model's limitations is not skepticism—it is the condition for using it well. Blind trust and total rejection are both failures of understanding.

Principle 2: The model's reliable zone does not coincide with its fluent zone. The model sounds equally confident in-distribution and out of distribution. The burden of detecting the frontier is the user's.

Principle 3: Use the model for what it does well (generate, structure, explore, scale) and the human for what they do well (verify, judge, cause, decide). Complementarity is the design principle, not delegation.

Principle 4: Each limitation has a known pattern of compensation. Implementing them is not optional—it is the difference between amplifying intention and amplifying error.

Principle 5: The epistemic stance of the intention architect is: the tool is powerful, its limitations are known, and my job is to govern the interaction so that the result is worthy of trust—not to assume that it is.

6. Glossary of Key Terms

7. Final Reflection and Self-Assessment

You have completed Module VI. You now understand the tool: what it is, how it learns, where its biases amplify inequalities, and where its limitations can deceive you. Before moving on to Module VII (Persuasion and Cognitive Ethics), integrate it all.

8. Bibliography and Recommended Readings

Essential Readings

• Mitchell, M. (2019). Artificial Intelligence: A Guide for Thinking Humans. Farrar, Straus and Giroux.

• Russell, S. & Norvig, P. (2020). Artificial Intelligence: A Modern Approach. 4th ed. Pearson. [Chapters on limitations and safety]

• Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking.

Complementary Readings

• Bender, E. et al. (2021). “On the Dangers of Stochastic Parrots.” FAccT Conference. [A critique of massive language models]

• Marcus, G. & Davis, E. (2019). Rebooting AI: Building Artificial Intelligence We Can Trust. Vintage. [The limits of deep learning]

• Pearl, J. & Mackenzie, D. (2018). The Book of Why. Basic Books. [Connects with the causal incapacity of current models]

• O'Neil, C. (2016). Weapons of Math Destruction. Crown. [Connects with the consequences of ungoverned limitations]

Next Module: Congratulations: you have completed Module VI. You can now govern the tool with understanding and judgment. Module VII—Persuasion and Cognitive Ethics—closes the course with the most delicate question of all: where is the line between influencing and manipulating? With Cialdini, Habermas, Floridi, Zuboff, and Dennett you will study the psychology of persuasion, strategic communication, and ethics applied to intelligent systems. If Modules I–V taught you to formulate the intention and Module VI to govern the tool, Module VII will teach you to use both with responsibility—the competence that separates the architect from the manipulator.