Reading Data: The Traps in Graphs

1.Truncated Axes: Making Small Differences Look Big

The heights of the bars in a bar chart are there so you can compare the sizes of the values by eye. But if the vertical axis starts somewhere other than 0, the ratio of the bar heights no longer matches the ratio of the actual values. A small difference ends up looking like a difference of several times.

Truncating the vertical axis is not always wrong. Line graphs that show small changes in values that never come near 0, such as body temperature or stock prices, often cut the axis. The problem is cutting the axis while inviting you to compare sizes by bar length. When you look at a graph, check the starting value and the tick marks on the vertical axis first.

2.Correlation Is Not Causation

When two values rise and fall together, they are correlated. In summer, ice cream sales go up and so do swimming accidents. The two are clearly correlated, but ice cream does not cause accidents. A 3rd factor, temperature, moves them both. A hidden factor like this is called a confounding variable.

Before reading a correlation as causation, ask three questions. Is there a hidden common cause? Could the direction of cause and effect be reversed (are people who exercise healthy, or do healthy people exercise more)? Could it be a coincidence? The surest way to establish causation is an experiment comparing two randomly assigned groups, but such an experiment isn't always possible.

• Common cause: temperature → ice cream sales, temperature → swimming accidents
• Reversed direction: "People who go to the hospital often are sicker"; they go to the hospital because they are sick
• Coincidence: two unrelated statistics can move in step if you choose the time period carefully

3.Sampling Bias: Whom Did You Ask?

Since you can't survey everyone, you draw a part (a sample) and use it to estimate the whole. If the sample doesn't resemble the whole, the results are skewed no matter how much data you collect. A survey run only inside a particular app reflects the opinions of people who use that app, and a survey of only people who volunteered to answer is closer to the opinions of people with a strong interest in the topic.

Looking only at the survivors is also sampling bias. The claim "successful people all got up at dawn" leaves out the people who got up at dawn and did not succeed. When you look at data, ask "who was chosen, how, and who was left out?" before you look at the numbers. How a sample is drawn often matters more than how big it is.

AI has the same problem. If the training data is skewed toward a particular group, the AI's judgments are skewed in that direction too. The data bias covered in the AI Basics course is exactly this sampling bias.

4.Rates vs. Counts, Relative vs. Absolute Change

If you directly compare the number of accidents in a city of 1,000,000 people and a city of 100,000 people, the big city will always look worse. For a fair comparison, you need rates converted to the same base, such as per 100,000 people. Conversely, if you look only at rates and miss the counts, you misjudge the actual scale.

"The risk has doubled" carries the same problem. Relative change (doubling, a 100% increase) sounds big, but if the original risk was very small, the absolute change may be tiny. You need to look at both to judge.

5.The Traps of Averages and Grouping

Can you wade across a river whose average depth is 1 m? If it is shallow at the edges and 3 m deep in the middle, you can't, even though the average is 1 m. An average tells you nothing about spread or the maximum. As you saw in Lesson 8, the median and the spread belong next to the mean.

Sometimes the results for grouped data and for the separate parts point in opposite directions. This is called Simpson's paradox. In the example below, Hospital X has a higher success rate than Y for both easy and difficult surgeries, but combined, Y looks far better. That's because X took on far more difficult surgeries.