The Scientific Method — The Skill of Checking Claims

1.Science is a method before it is knowledge

Science tends to bring formulas and jargon to mind, but its real power lies in its method for checking claims. However plausible an idea sounds, until it has been confirmed by observation and experiment, it is still only a hypothesis.

The scientific method is usually described in the following order. Real research moves back and forth through these steps and repeats them many times, but the framework is useful for getting the overall flow.

• Observation: You notice a phenomenon, such as “Bananas kept in the refrigerator quickly turn black on the outside.”
• Question: Why does that happen? Is it the temperature, or the humidity?
• Hypothesis: You propose an answer in a form that can be tested, such as “Low temperature damages the cells of the peel and turns it black.”
• Experiment: You decide in advance what result should appear if the hypothesis is right, then compare under controlled conditions.
• Analysis and conclusion: You check whether the results match the prediction, then accept, revise, or discard the hypothesis.
• Sharing and replication: You publish your method so others can repeat it and see whether they get the same result.

2.Controlling variables — change only one thing at a time

The heart of an experiment is a fair comparison. You change only the one condition you think is the cause and keep everything else the same. That way, if the results differ, you can narrow the cause down to that one condition.

The condition you deliberately change is called the independent (manipulated) variable, the value you measure as a result is the dependent variable, and the other conditions you keep the same are the controlled variables. The group in which the condition is changed is the experimental group, and the baseline group in which it is not changed is the control group.

In experiments on people, expectations alone can change the results. The classic example is the placebo effect: people feel better even after taking a fake pill that has no effect. That is why medical studies assign participants to groups at random and use a double-blind design, in which neither the participants nor the researchers know who received the real drug.

3.Correlation is not causation

When two values rise and fall together, they are correlated. In summer, both ice cream sales and swimming accidents go up. That doesn't mean ice cream causes accidents. Both rise together because of a third cause: hot weather.

A hidden common cause like this is called a confounding variable. Cause and effect may also be reversed (reverse causation), or the two may simply have moved together by chance. Even if a survey finds that “students who eat breakfast get better grades,” the survey alone cannot tell you whether breakfast raised their grades or whether some other factor in families with regular routines affected both.

The surest way to establish causation is the controlled experiment described above. When groups are assigned at random, confounding variables are spread evenly across both groups, so any difference that remains can be attributed to the condition you changed. For topics where experiments are impossible (such as smoking and disease), scientists gather evidence instead: Do many different kinds of studies point in the same direction? Does the effect grow as the amount increases?

• Confounding: Is there a hidden cause that affects both?
• Reverse causation: Could cause and effect be the other way around?
• Chance: Is the data too small, or was this one result cherry-picked from many comparisons?

4.How to read “cuts risk by 50%” — relative risk and absolute risk

Health articles and ads often say things like “cuts the risk of disease by 50%.” The number may not be wrong, but on its own it doesn't tell you how big the effect is. The impression changes dramatically depending on whether the same result is described as relative risk or absolute risk.

Relative risk reduction tells you by what percent the risk fell compared with the original risk; absolute risk reduction tells you how many percentage points it actually fell. The smaller the original risk, the bigger the relative risk reduction looks and the smaller the absolute risk reduction becomes. You need to look at both numbers to judge the size of the effect properly.

5.Reproducibility and the nature of scientific knowledge

The result of a single experiment could be a fluke or a mistake. You can trust a result only when other people get the same result using the same method. This is called reproducibility. That is why scientific papers describe not just their results but their methods in detail, and go through review by other researchers.

In everyday speech, “theory” is used to mean something close to a guess, but in science a theory is an explanatory framework supported by a great many observations and experiments. Even so, scientific knowledge can be revised when new evidence appears. That is not a weakness; it is how science corrects its own errors.

Many scientific explanations are also models — simplified pictures of reality. Drawing the atom as a little solar system, or describing motion as if there were no friction, are examples. A model is not wrong; it is a tool built to be exactly as accurate as needed. Throughout this course, we will point out when something is a model.

6.Questions for checking an AI's scientific explanations

AI produces plausible scientific explanations very quickly. Most are correct, but they sometimes include figures of unclear origin or sentences that present a correlation as if it were causation. Just apply the questions of the scientific method directly to the AI's answer.

Was this claim confirmed by an experiment, or does it come from an observational study? Was there a control group? Have other studies found the same result? If there is a number, where did it come from? Asking just these four questions will filter out a large share of exaggerated health information and advertising copy.