Statistical Reports, Claims and Bias
Theory
Statistics in the media and in reports need checking before they are believed. This Year 10 maths page shows how to judge whether a sample is representative or biased, spot unfair survey questions, test a claim against the data, recognise misleading graphs and averages, and explain why an association does not prove a cause. In the Victorian Curriculum this is Level 10 content; in the Australian Curriculum it is Year 10 content. Curriculum: VC2M10ST04 / AC9M10ST01.
A sample is representative when every part of the population can appear in it and the way it is chosen has nothing to do with the question. Otherwise it is biased: its results lean one way.
Common causes of a biased sample: a convenience sample (whoever is easy to reach), a self-selected sample (people choose to reply, such as an online poll), under-coverage (some groups cannot be reached, such as a landline-only phone survey), and non-response (many people chosen do not reply).
A bigger sample does not fix bias. Asking more of the same kind of people gives the same wrong answer with more confidence.
A question is biased when it is leading (it suggests the answer), loaded (it uses emotive words), ambiguous (words like 'often' mean different things to different people), or double-barrelled (two questions joined by 'and').
Answer options must not overlap, must cover every possible answer, and should be balanced, with as many negative as positive choices. Sensitive questions get more honest answers when the survey is anonymous.
A report's claim must match its evidence. Check the display (does the vertical axis start at zero?), the statistic (is the mean dragged by a few extreme values, so the median would be fairer?), and whether counts should be rates, such as cases per \(1000\) people.
An association between two variables does not prove that one causes the other. A third variable may drive both: towns with more cafés also have more dentists, because both grow with the number of people.
Comparing the expected number with the actual number shows which groups are under- or over-represented.
Use the groups' true shares of the population, not their shares of the replies.
| Problem | Example | Fix |
|---|---|---|
| leading | 'Don't you agree …?' | ask neutrally, offer all answers |
| ambiguous | 'Do you read often?' | ask for a number in a time period |
| double-barrelled | '… quieter and cleaner?' | split into two questions |
| overlapping options | \(0\)–\(3\), \(3\)–\(6\) | \(0\)–\(3\), \(4\)–\(6\) |
| unbalanced scale | great / good / OK | equal positive and negative options |
Is the sample biased?
- Name the population the question is about.
- Name who was actually asked, and how they were chosen.
- Ask who is left out, and whether they would answer differently. If so, the sample is biased.
- Check the response: a low response rate or self-selection makes any result doubtful.
- Limit the conclusion to the group that was really sampled ('shoppers at that market on a Saturday').
Is the questionnaire fair?
- Read each question as a reader would: does it suggest an answer, use emotive words, or ask two things at once?
- Check the options: no overlaps, nothing missing, balanced.
- Sensitive topics: make the survey anonymous.
- Look for a conflict of interest: who paid for the survey, and who was asked?
(i) Adults leaving the town's gym one evening.
(ii) Every \(20\)th name on the town's electoral roll.
(iii) Adults who answer a poll on a local sports club's social-media page.
The population is every adult in the town. For each sample, ask who is left out and whether they would answer differently.
(i) Biased. It leaves out everyone who does not go to the gym, and people at a gym play more sport than most. The estimate would be too high.
(ii) Representative. The electoral roll lists the town's adults, and choosing every \(20\)th name has nothing to do with how much sport they play.
(iii) Biased. It is self-selected: only people who follow a sports club page and choose to answer take part. They are likely to play more sport than other adults.
(ii) Explain why the graph makes the increase look larger than it is.
(iii) How should the graph be drawn?
| \(\text{(i) increase}\) | \(=\) | \(430 - 400 = 30\) |
| \(\dfrac{30}{400} \times 100\%\) | \(=\) | \(\mathbf{7.5\%}\) |
(ii) The vertical axis starts at \(390\), not \(0\). The \(2023\) column is drawn \(10\) units tall and the \(2025\) column \(40\) units tall, so visits look as if they have quadrupled, when they rose by only \(7.5\%\).
(iii) Start the vertical axis at \(0\), or show a clear break in the axis and give the actual values.
(i) What percentage of households replied?
(ii) How many households said they oppose it, and what percentage of all \(1800\) is that?
(iii) The council reports '\(65\%\) of residents oppose the dog park'. Explain why this is unreliable.
(i) and (ii)
| \(\text{response rate}\) | \(=\) | \(\dfrac{360}{1800} \times 100\%\) |
| \(\) | \(=\) | \(\mathbf{20\%}\) |
| \(\text{oppose}\) | \(=\) | \(65\% \times 360 = \mathbf{234}\) |
| \(\text{of all } 1800\) | \(=\) | \(\dfrac{234}{1800} \times 100\%\) |
| \(\) | \(=\) | \(\mathbf{13\%}\) |
So \(234\) households oppose the park: \(13\%\) of all households.
(iii) \(80\%\) of households did not reply. People who feel strongly, often those against a change, are more likely to send a survey back, so the replies are a self-selected sample. All we know for sure is that at least \(13\%\) of households oppose the park.
(i) Find the mean and the median salary.
(ii) Is the advertisement true? Is it fair?
| \(\text{total}\) | \(=\) | \(10 \times 55\,000 + 120\,000 + 365\,000\) |
| \(\) | \(=\) | \(1\,035\,000\) |
| \(\text{mean}\) | \(=\) | \(\dfrac{1\,035\,000}{12} = \mathbf{\$86\,250}\) |
| \(\text{median}\) | \(=\) | \(\text{6th and 7th salaries} = \mathbf{\$55\,000}\) |
(ii) The claim is true for the mean, but misleading. Ten of the twelve staff earn \(\$55\,000\), well below the mean; two very high salaries drag it up. The data are positively skewed, so the median, \(\$55\,000\), is the fair measure of a typical salary.
Common pitfalls
Frequently asked questions
What makes a sample biased?
A sample is biased when some part of the population is left out or over-represented, and those people would answer differently from the rest.
Does a bigger sample remove bias?
No. A bigger sample reduces random variation, but if the way people are chosen is biased, adding more of the same people repeats the same bias.
What is a leading question?
A question that pushes people towards one answer, such as one starting with 'Don't you agree that'.
Why are online polls often unreliable?
The people who answer choose themselves, so the poll over-represents people who feel strongly about the topic or who visit that website.
Why use an anonymous survey?
People answer sensitive questions more honestly when nobody can tell which answer is theirs.
What is a double-barrelled question?
A question that asks about two things at once, so someone who agrees with one part and not the other cannot answer honestly.
Does a strong association prove that one thing causes another?
No. A third variable may cause both, or the link may be a coincidence. Only a fair, controlled experiment can show a cause.