Statistical Reports, Claims and Bias
Theory
Statistics in the media and in reports need checking before they are believed. This Year 10A maths page shows how to judge whether a sample is representative or biased, spot unfair survey questions, test a claim against the data, recognise misleading graphs and averages, and explain why an association does not prove a cause, including rates per head of population and ethical issues. It is Level 10 content in the Year 10A course (Victorian Curriculum 10A) and Year 10 content in the Australian Curriculum. Curriculum: VC2M10ST04 / AC9M10ST01.
A sample is representative when every part of the population can appear in it and the way it is chosen has nothing to do with the question. Otherwise it is biased: its results lean one way.
Common causes of a biased sample: a convenience sample (whoever is easy to reach), a self-selected sample (people choose to reply, such as an online poll), under-coverage (some groups cannot be reached, such as a landline-only phone survey), and non-response (many people chosen do not reply).
A bigger sample does not fix bias. Asking more of the same kind of people gives the same wrong answer with more confidence.
A question is biased when it is leading (it suggests the answer), loaded (it uses emotive words), ambiguous (words like 'often' mean different things to different people), or double-barrelled (two questions joined by 'and').
Answer options must not overlap, must cover every possible answer, and should be balanced, with as many negative as positive choices. Sensitive questions get more honest answers when the survey is anonymous.
A report's claim must match its evidence. Check the display (does the vertical axis start at zero?), the statistic (is the mean dragged by a few extreme values, so the median would be fairer?), and whether counts should be rates, such as cases per \(1000\) people.
An association between two variables does not prove that one causes the other. A third variable may drive both: towns with more cafés also have more dentists, because both grow with the number of people.
Comparing the expected number with the actual number shows which groups are under- or over-represented.
Use the groups' true shares of the population, not their shares of the replies.
| Problem | Example | Fix |
|---|---|---|
| leading | 'Don't you agree …?' | ask neutrally, offer all answers |
| ambiguous | 'Do you read often?' | ask for a number in a time period |
| double-barrelled | '… quieter and cleaner?' | split into two questions |
| overlapping options | \(0\)–\(3\), \(3\)–\(6\) | \(0\)–\(3\), \(4\)–\(6\) |
| unbalanced scale | great / good / OK | equal positive and negative options |
Is the sample biased?
- Name the population the question is about.
- Name who was actually asked, and how they were chosen.
- Ask who is left out, and whether they would answer differently. If so, the sample is biased.
- Check the response: a low response rate or self-selection makes any result doubtful.
- Limit the conclusion to the group that was really sampled ('shoppers at that market on a Saturday').
Is the questionnaire fair?
- Read each question as a reader would: does it suggest an answer, use emotive words, or ask two things at once?
- Check the options: no overlaps, nothing missing, balanced.
- Sensitive topics: make the survey anonymous.
- Look for a conflict of interest: who paid for the survey, and who was asked?
(i) Adults leaving the town's gym one evening.
(ii) Every \(20\)th name on the town's electoral roll.
(iii) Adults who answer a poll on a local sports club's social-media page.
The population is every adult in the town. For each sample, ask who is left out and whether they would answer differently.
(i) Biased. It leaves out everyone who does not go to the gym, and people at a gym play more sport than most. The estimate would be too high.
(ii) Representative. The electoral roll lists the town's adults, and choosing every \(20\)th name has nothing to do with how much sport they play.
(iii) Biased. It is self-selected: only people who follow a sports club page and choose to answer take part. They are likely to play more sport than other adults.
(i) Don't you agree that the library should open on Sundays?
(ii) How often do you use public transport? Rarely / Sometimes / Often
(iii) Your age: \(10\)–\(20\), \(20\)–\(30\), \(30\)–\(40\)
(iv) Should the council fix the footpaths and plant more trees?
(i) Leading: 'Don't you agree' pushes people towards yes. Fair version: Should the library open on Sundays? Yes / No / Unsure.
(ii) Ambiguous: 'sometimes' means different things to different people. Fair version: How many trips did you take on public transport last week? \(0\), \(1\)–\(2\), \(3\)–\(5\), \(6\) or more.
(iii) Overlapping and incomplete: a \(20\)-year-old fits two boxes, and anyone under \(10\) or over \(40\) fits none. Fair version: under \(20\), \(20\)–\(29\), \(30\)–\(39\), \(40\) or over.
(iv) Two questions in one: someone who wants new footpaths but not trees cannot answer. Ask two separate questions.
(i) Find the number of accidents per \(1000\) residents in each suburb.
(ii) Is the claim supported?
(iii) The report also printed the street and age of every injured rider. Give one ethical concern.
| \(A\) | \(:\) | \(\dfrac{96}{32\,000} \times 1000 = \mathbf{3}\) |
| \(B\) | \(:\) | \(\dfrac{54}{12\,000} \times 1000 = \mathbf{4.5}\) |
(ii) No. \(A\) has more accidents only because it has more people. Per \(1000\) residents, \(B\)'s rate is higher.
(iii) Privacy: with a street and an age, readers could identify the riders without their consent.
(i) What percentage of the replies say yes?
(ii) Why does this overstate support?
(iii) Assuming each suburb's percentage is accurate, estimate the support in the region, as a percentage and a number of adults.
| \(\text{yes}\) | \(=\) | \(80\% \times 600 + 40\% \times 200\) |
| \(\) | \(=\) | \(480 + 80 = 560\) |
| \(\dfrac{560}{800}\) | \(=\) | \(\mathbf{70\%}\) |
(ii) Suburb \(A\) gives \(\dfrac{600}{800} = 75\%\) of the replies but has only \(30\%\) of the adults, and it is the suburb most in favour. The poll is self-selected and over-represents \(A\).
(iii) Weight each suburb's percentage by its true share of the adults.
| \(\text{support}\) | \(=\) | \(0.3 \times 80\% + 0.7 \times 40\%\) |
| \(\) | \(=\) | \(24\% + 28\% = \mathbf{52\%}\) |
| \(52\% \times 20\,000\) | \(=\) | \(\mathbf{10\,400}~\text{adults}\) |
Common pitfalls
Frequently asked questions
What makes a sample biased?
A sample is biased when some part of the population is left out or over-represented, and those people would answer differently from the rest.
Does a bigger sample remove bias?
No. A bigger sample reduces random variation, but if the way people are chosen is biased, adding more of the same people repeats the same bias.
What is a leading question?
A question that pushes people towards one answer, such as one starting with 'Don't you agree that'.
Why are online polls often unreliable?
The people who answer choose themselves, so the poll over-represents people who feel strongly about the topic or who visit that website.
Why use an anonymous survey?
People answer sensitive questions more honestly when nobody can tell which answer is theirs.
What is a double-barrelled question?
A question that asks about two things at once, so someone who agrees with one part and not the other cannot answer honestly.
Does a strong association prove that one thing causes another?
No. A third variable may cause both, or the link may be a coincidence. Only a fair, controlled experiment can show a cause.