Scientific Methodology

Learning Objectives

By completing this book, you will be able to:

1. Distinguish between deductive, inductive, and abductive reasoning, and recognize which one you are using at each moment in the formulation of a problem.

2. Identify the most frequent logical fallacies in everyday reasoning and in interactions with intelligent systems.

3. Apply the scientific method as a disciplined framework for advancing from hypothesis to evidence, integrating Popper's falsifiability with real practice.

4. Distinguish between correlation and causality, and recognize when a causal inference is justified and when it is premature.

5. Evaluate the quality of the evidence supporting any claim, using criteria of internal validity, external validity, and reliability.

6. Integrate the tools of all four topics of Module II into a complete protocol for the formulation and validation of problems.

1. Introduction: What Does It Mean to Reason Well?

Module I showed you how the brain actually reasons: with shortcuts, emotions, and biases. The preceding topics of this module taught you to evaluate the quality of your knowledge, to separate ends from means, and to detect the frames that limit your formulation. But one piece is missing: the rules of valid reasoning. Knowing that your brain takes shortcuts does not tell you which path is the correct one. For that you need logic.

Logic is not an abstract subject for philosophers. It is the discipline that studies the rules of valid inference: how to move from premises to conclusions in such a way that, if the premises are true, the conclusion cannot be false (deduction) or is probably true (induction). It is, literally, the science of drawing good conclusions.

“A sales director observes that his three best salespeople have MBAs. He concludes: ‘We need to hire more people with MBAs.’ It sounds reasonable. But is it valid? The sample is tiny (three people). The correlation could be spurious: perhaps the three are good salespeople because of their personalities, not their MBAs. Perhaps the MBA gave them access to contacts, not to selling skills. Perhaps the other salespeople with MBAs were let go years ago and the director does not remember them (survivorship bias). The reasoning seems logical but is riddled with inferential flaws.”

1.1 Three Forms of Reasoning

Deduction: from the general to the particular. If the general premise is true, the particular conclusion is necessarily true. It is the strongest form of reasoning but also the most limited, because the conclusion never contains new information—it only makes explicit what was already implicit in the premises.

“Premise 1: All contracts signed after 2024 include an arbitration clause. Premise 2: This contract was signed in 2025. Conclusion: This contract includes an arbitration clause. The conclusion is necessarily valid—if the premises are true, the conclusion cannot be false. But the conclusion tells us nothing we did not already know. All of the risk lies in the truth of the premises.”

Induction: from the particular to the general. From specific observations, a general pattern is inferred. It is the form of reasoning that generates new knowledge, but its conclusions are always probable, never certain. Induction is the engine of empirical science—and also the source of its most persistent errors.

“Observation 1: Campaign A increased sales by 15%. Observation 2: Campaign B increased sales by 12%. Observation 3: Campaign C increased sales by 18%. Inductive conclusion: Campaigns of this type increase sales by between 12% and 18%. This is probably true given the evidence, but the next campaign could fail completely. Induction gives you reasonable expectations, not guarantees.”

Abduction: from the observation to the best explanation. Charles Sanders Peirce, the American philosopher and logician, identified a third form of reasoning: abduction, or inference to the best explanation. You observe a surprising phenomenon and generate the hypothesis that would best explain it. It is the reasoning of the detective, the physician, and the innovator. And it is the form of reasoning that Dorst (Topic 3) places at the center of frame innovation.

“A product that sold well for years suddenly loses 30% of its sales in two months. There were no changes in price, quality, or distribution. Abductive hypothesis: a competitor launched a substitute product that is capturing our market. This hypothesis is not proven—it is a conjecture that explains the observation. But it is the best available explanation and it orients the investigation. Abduction generates hypotheses; deduction extracts their consequences; induction tests them against data. The three forms work together.”

2. Logical Fallacies: The Paths That Look Valid and Are Not

A fallacy is an error of reasoning that appears valid but is not. Fallacies are dangerous precisely because they are persuasive: they sound good, they feel right, and they frequently produce conclusions that coincide with what we want to believe. Learning to detect them does not make you immune to them—System 1 produces them automatically—but it gives you the tools for your System 2 to intercept them before they become premises of your decisions.

2.1 Fallacies of Relevance

Ad hominem: Attacking the person instead of the argument. “That proposal comes from someone who has never led a team” does not refute the proposal; it refutes the proposer. The proposal may be brilliant regardless of who makes it.

“In a meeting, a junior analyst suggests that the company should reconsider its pricing strategy. The director replies: ‘When you have ten years of experience, we'll talk.’ This is ad hominem: the analyst's experience bears no logical relationship to the validity of her pricing analysis. A beginner with good data can be right; an expert with bad data can be wrong.”

Appeal to authority: Accepting a claim because someone with authority says it, without evaluating the argument. It is fallacious when the authority lies outside their domain of competence, or when it is used as a substitute for evidence.

“A CEO famous for creating a successful technology company opines about macroeconomic policy in an interview. His followers treat that opinion as expert knowledge. But competence in technology does not transfer to macroeconomics. Authority is contextual, not universal.”

Appeal to popularity (ad populum): “Everybody is doing it” or “the market is moving in this direction” are not logical arguments. The majority can be wrong. Every financial bubble in history was backed by the majority.

2.2 Causal Fallacies

Post hoc ergo propter hoc: “After this, therefore because of this.” Assuming that because B occurred after A, A caused B. It is the most common causal fallacy and the most costly in business settings.

“A company implemented a new CRM in January. In March, sales rose 20%. Conclusion: the CRM increased sales. But March is always a good month (seasonality), a competitor shut down operations in February (external factor), and the sales team received a new commission scheme, also in January (confounding variable). Temporal sequence does not establish causality.”

Correlation is not causality: Two variables can move together (correlation) without either causing the other. Both can be caused by a third factor, the relationship can be coincidence, or the causality can run opposite to the direction assumed.

“There is a correlation between the number of ice creams sold and the number of drownings in swimming pools. Do ice creams cause drownings? Obviously not. Both are caused by a third factor: summer heat. This example seems obvious, but in business contexts, correlations just as spurious go unnoticed because they sound plausible: ‘teams that use our tool perform better’ could mean that high-performing teams choose better tools, not that the tool causes the performance.”

2.3 Structural Fallacies

False dichotomy: Presenting only two options when there are more. “Either we adopt AI or we become obsolete” ignores intermediate, gradual, or entirely different options.

Slippery slope: Assuming that a small step will inevitably lead to extreme consequences without justifying the intermediate steps. “If we allow remote work one day a week, soon nobody will come to the office and we will lose our entire culture” is a causal chain with no evidence at any link.

Straw man: Distorting the other person's argument to make it easier to refute. “You want us to invest in innovation, but we can't throw money away on crazy projects” transforms “investment in innovation” into “throwing money away on lunacy.”

Begging the question (circular reasoning): Assuming as a premise what you are trying to prove. “Our strategy is correct because the results show that it works”—but which results do you count as success? The ones your strategy defined as success. The argument bites its own tail.

3. The Scientific Method: From Hypothesis to Evidence

The scientific method is not a ritual exclusive to laboratories. It is a disciplined protocol for moving from conjecture to knowledge, from hypothesis to evidence, from “I believe that” to “the evidence shows that.” And like every discipline, its value lies not in the moments when it is easy to apply but in the moments when it is hard—when intuition shouts one thing and the data say another.

3.1 The Stages of the Method

Observation: You notice a phenomenon you cannot explain with your current knowledge. This stage connects with abduction: something surprises you and triggers the search for an explanation.

Hypothesis: You formulate a tentative and falsifiable explanation (here Popper from Topic 1 enters). A good hypothesis specifies what it predicts and what observation would refute it.

Prediction: You deduce the observable consequences of your hypothesis. If the hypothesis is true, then you should observe X. If you do not observe X, the hypothesis has a problem.

Experimentation: You design a test that could refute your hypothesis. The key is control: isolating the variable you want to test from all the other variables that could explain the result.

Analysis and conclusion: You evaluate the results. Does the evidence support the hypothesis, refute it, or is it ambiguous? Are there alternative explanations? Are the results replicable?

“A product team notices that users abandon the app after the third session (observation). They hypothesize that the onboarding is confusing (hypothesis). They predict that if they simplify the onboarding, third-session retention will rise by at least 15% (falsifiable prediction). They launch an A/B version: half of new users receive the current onboarding and the other half receive the new one (experimentation with a control). After four weeks, group B shows 22% higher retention (analysis). But before celebrating, they ask: are there confounding variables? Did group B also receive more push notifications? Did the test period coincide with a marketing campaign? The legitimate conclusion is: the evidence is consistent with the hypothesis, but it does not definitively confirm it (remember: corroboration, not confirmation).”

3.2 The Scientific Method and the Formulation of Intentions

For Intention Architecture, the scientific method offers a framework for validating intentions before executing them. Every intention contains implicit hypotheses: “I believe the problem is X, I believe the cause is Y, I believe the solution is Z.” The scientific method says: do not execute Z until you have tested hypotheses X and Y.

“An entrepreneur wants to launch a healthy-food subscription platform. Her intention contains at least three hypotheses: (1) people want to eat healthier but have no time to cook (problem hypothesis), (2) a subscription with weekly deliveries would solve that problem (solution hypothesis), (3) they are willing to pay a premium price for it (viability hypothesis). The scientific method asks: before building the platform, can you test each hypothesis separately? Interviews with potential users for (1), an MVP or a manual pilot for (2), and price testing for (3). This is what Eric Ries called ‘validated learning’ in the Lean Startup method—which is nothing other than the scientific method applied to entrepreneurship.”

4. Correlation and Causality: The Most Ignored Distinction

No other inferential error causes as many problems in the formulation of intentions as the confusion between correlation and causality. It is so ubiquitous, so intuitive, and so costly that it deserves a full section.

4.1 Why the Brain Confuses Correlation with Causality

The human brain is wired to detect patterns and attribute causes. It is an extraordinarily useful evolutionary adaptation: the ancestor who heard a noise in the bushes and assumed “predator” survived more often than the one who assumed “spurious correlation with the wind.” But this adaptation produces a systematic bias in modern contexts: we see causality where there is only correlation, and we act on that false causality with unjustified confidence.

Judea Pearl, a computer scientist and philosopher, developed the most influential formal framework for reasoning about causality. In his book The Book of Why (2018), Pearl distinguishes three levels of causal reasoning:

Level 1 — Association: What occurs together with what? Seeing correlations in the data. “Clients who buy shoes also buy socks.”

Level 2 — Intervention: What will happen if I do X? Predicting the effect of an action. “If we offer free socks with every shoe purchase, will we sell more shoes?”

Level 3 — Counterfactual: What would have happened if I had done something different? “If we had not changed the price, would sales have fallen anyway?”

Most of the data we have gives us Level 1 information (correlations). But decisions require Level 2 information (interventions). The leap from Level 1 to Level 2 is exactly where the correlation-causality confusion occurs.

“A company observes (Level 1) that employees who use its corporate wellness app have lower absenteeism. It decides (Level 2) to make use of the app mandatory, expecting absenteeism to fall for everyone. But perhaps the causality runs the other way: the healthiest and most motivated employees choose to use the app. Forcing everyone to use it will create neither health nor motivation—only resentment. An intervention based on correlation can produce exactly the opposite of what was expected.”

4.2 Tools for Evaluating Causality

Although establishing causality with certainty is extremely difficult outside a controlled experiment, there are criteria that strengthen or weaken a causal inference. The most useful for everyday practice are:

Temporality: The cause must precede the effect. If B occurs before A, A cannot cause B. This is necessary but not sufficient (post hoc ergo propter hoc).

Magnitude of the association: Stronger correlations are more suggestive of causality than weak ones, although a strong correlation does not guarantee causality.

Dose-response: If more of A produces more of B in a graded way, the causal relationship is more plausible.

Plausible mechanism: Is there a mechanism that explains how A would cause B? A correlation without a plausible mechanism is suspect.

Elimination of alternatives: Have you ruled out the most probable alternative explanations? Confounding variables, reverse causality, a third factor, coincidence.

5. The Quality of Evidence: Not All Data Are Equal

Having data is not the same as having evidence. Evidence is data organized with rigor to answer a specific question. And not all evidence has the same quality. Understanding this hierarchy is essential for evaluating whether the information supporting a problem formulation is solid or fragile.

5.1 The Hierarchy of Evidence

Evidence-based medicine developed a hierarchy of evidence quality that applies far beyond medicine. From lowest to highest quality:

Expert opinion: What someone with experience believes. Useful as a starting point but vulnerable to personal biases, unverified tacit knowledge, and the Dunning-Kruger effect.

Single case studies: Detailed observations of one particular case. Rich in detail but impossible to generalize. “It worked for us” does not mean it will work for others.

Observational studies: Measuring correlations in existing data without manipulating variables. Better than single cases but vulnerable to the correlation-causality confusion.

Controlled experiments: Manipulating one variable while controlling the rest. The gold standard for establishing causality. A/B tests are the business version.

Meta-analyses and systematic reviews: Synthesizing all available studies on a question. The most robust evidence, because it integrates multiple sources and reduces the bias of any individual study.

“An entrepreneur hears on a podcast that ‘emails sent on Tuesdays have better open rates.’ That is expert opinion (the lowest level). If he reads it in a Mailchimp study drawing on data from thousands of campaigns, it is an observational study (better). If Mailchimp ran a controlled A/B test varying only the day of sending, it is an experiment (better still). But even the experiment does not guarantee that Tuesday is best for his own audience—that requires his own test. Someone else's evidence is a starting point, not a destination.”

5.2 Internal Validity, External Validity, and Reliability

Internal validity: Does the study measure what it claims to measure? Are the results really due to the variable that was studied, or are there alternative explanations? An A/B test in which group A received the new email and a discount has low internal validity: you do not know whether the effect comes from the email or the discount.

External validity: Are the results generalizable to other contexts? A study with American university students may not apply to European professionals. An A/B test on your technology web page may not apply to your agricultural products page.

Reliability: If you repeat the study, do you obtain the same results? Reliability is consistency. A result that appears once and is never replicated is suspect, however promising it may look.

6. Practical Application: The Complete Protocol of Module II

This is the final topic of Module II. You now have all the tools for formulating problems with rigor. Let us integrate everything into a final case study and into a protocol you can apply to any formulation.

6.1 Case Study: The Hiring Nobody Questioned

Ricardo, CTO of a 200-person company, asks an AI system: “I need to hire 10 senior machine learning engineers in the next three months. Help me design the hiring strategy.”

Let us apply the complete protocol of Module II:

Epistemology (Topic 1): What does Ricardo actually know? His epistemic assumptions: (1) he knows he needs ML (or does he assume it because it is a trend?), (2) he knows he needs 10 (based on what calculation?), (3) he knows they must be senior (or could they be mid-level with good mentorship?), (4) he knows three months is enough (or is that an arbitrary deadline imposed by the budget?). The dominant type of knowledge may be tacit knowledge (“I feel that we need AI”) disguised as propositional knowledge (“we need 10 ML engineers”). Falsifiability: what result would lead Ricardo to conclude that he did not need 10 ML engineers?

Ends vs. Means (Topic 2): Hiring 10 engineers is a means. What is the end? If we climb the ladder: why 10 ML engineers? To build AI capabilities. Why? To improve the product. Why? To retain clients and win new ones. The real end is business growth. Alternative means: outsource ML to consultants? Use pre-trained models without an in-house team? Hire 3 seniors and 7 juniors with a training program?

Framing Errors (Topic 3): Narrow frame: it looks only at hiring, not at capabilities. Solutionist frame: he has already decided that the answer is to hire. Single-actor frame: what do the current engineers think? Perhaps they already have ML skills that are not being used. Metaphor: “hiring” is a metaphor of acquisition—what if the metaphor were cultivation? Growing talent internally instead of buying it.

Scientific Methodology (this Topic): The claim “we need ML” is a hypothesis. Can it be tested before hiring 10 people? A pilot with 2 engineers and one specific project would test whether ML really improves the product. The available evidence: is it the CTO's opinion, a market study, or data from clients asking for ML-based features? The assumed causality (“more ML = better product = more clients”) needs validation at every link.

“A system with Intention Architecture would respond: ‘I can help you with the hiring strategy. But first, I want to make sure we are solving the right problem. I see four implicit hypotheses in your request: that ML is the right technology for your product, that you need in-house rather than external capability, that the number is 10, and that they must be senior. Each one is testable. Do you have client data indicating demand for ML-based features? Have you explored alternatives such as pre-trained ML APIs that would require no in-house team? Does the number 10 come from a capacity calculation or an intuitive estimate? I suggest we begin with a small pilot that validates the central hypothesis before committing to a mass hire.’”

6.2 The Integrated Formulation Protocol of Module II

Here is the complete protocol you can apply to any formulation of a problem or intention:

Step 1 — Epistemic audit: Which assumptions about the situation, the causes, the solution, and competence are implicit? What type of knowledge dominates—propositional, procedural, or tacit? Is the formulation falsifiable?

Step 2 — Means-ends ladder: Is the request an end or a means? Climb with “why?” until you find the optimal level. Generate alternative means.

Step 3 — Frame analysis: What does the frame include and exclude? Are there framing errors (narrow, symptomatic, solutionist, temporal, single-actor, paradigmatic)? Which metaphors are operating? Generate at least one alternative reframing.

Step 4 — Logical validation: Are there fallacies in the reasoning? Are the causal inferences justified? Is correlation being confused with causality?

Step 5 — Evidence evaluation: What type of evidence supports the formulation? Where does it sit in the hierarchy? Does it have internal validity, external validity, and reliability for this context?

Step 6 — Test design: Can you test the critical hypotheses before executing? What is the minimum viable experiment? What result would make you change course?

7. Glossary of Key Terms

8. Final Reflection and Self-Assessment

You have completed Module II. Before moving on to Module III (Applied Logic and Systems Thinking), verify that the tools of formulation have been integrated into the way you think.

9. Bibliography and Recommended Readings

Essential Readings

• Pearl, J. & Mackenzie, D. (2018). The Book of Why. Basic Books.

• Popper, K. (1963). Conjectures and Refutations. Routledge.

• Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.

Complementary Readings

• Peirce, C.S. (1878). “Deduction, Induction, and Hypothesis.” Popular Science Monthly, 13, 470–482.

• Ries, E. (2011). The Lean Startup. Crown Business.

• Taleb, N.N. (2007). The Black Swan. Random House.

• Sackett, D. et al. (1996). “Evidence Based Medicine: What It Is and What It Isn't.” BMJ, 312(7023), 71–72.

• Sagan, C. (1996). The Demon-Haunted World. Ballantine Books.

Next Module

You have completed Module II: Correct Formulation of Problems. You now know how to evaluate the quality of your knowledge, how to separate ends from means, how to detect and correct frames, and how to reason validly about problems and evidence. Module III—Applied Logic and Systems Thinking—will take you to the next level: how correctly formulated problems interact with complex systems, producing second- and third-order consequences that no linear formulation can anticipate. You will study formal causality with Judea Pearl, system dynamics with Donella Meadows and Peter Senge, leverage points with Jay Forrester, and antifragility with Nassim Taleb.