Bias and Questionnaires
A survey is only useful if its sample represents the population and its questions are fair. This NSW Year 9 Advanced maths subtopic looks at how bias gets into samples and questionnaires, and how to avoid it (NESA outcome MA5‑DAT‑P‑01).
You will decide whether a sample is biased or representative, spot vague and two-part questions, one-sided answer options and overlapping intervals, and rewrite questions fairly. You will judge advertising claims, online polls, landline surveys, low response rates and face-to-face questions on sensitive topics, and weigh a large biased sample against a small random one. It is part of Year 9 Advanced Investigating Data.
Every question with a fully worked solution.
- Bias and Questionnaires - Video - Bias and Questionnaires Watch
Theory
A survey can give a wrong answer for two reasons: the wrong people were asked, or the right people were asked the wrong question. Both are called bias. This Year 9 Advanced maths page shows how to spot bias in a sample and in a questionnaire, how to fix it, and how to judge a claim built on a survey. It is useful for students heading towards Standard or Advanced maths.
A sample is representative when every part of the population can appear in it and the way it is chosen has nothing to do with the question. Otherwise it is biased: its results lean one way.
Common causes of a biased sample: a convenience sample (whoever is easy to reach), a self-selected sample (people choose to reply, such as an online poll), under-coverage (some groups cannot be reached, such as a landline-only phone survey), and non-response (many people chosen do not reply).
A bigger sample does not fix bias. Asking more of the same kind of people gives the same wrong answer with more confidence.
A question is biased when it is leading (it suggests the answer), loaded (it uses emotive words), ambiguous (words like 'often' mean different things to different people), or double-barrelled (two questions joined by 'and').
Answer options must not overlap, must cover every possible answer, and should be balanced, with as many negative as positive choices. Sensitive questions get more honest answers when the survey is anonymous.
Comparing the expected number with the actual number shows which groups are under- or over-represented.
Use the groups' true shares of the population, not their shares of the replies.
| Problem | Example | Fix |
|---|---|---|
| leading | 'Don't you agree …?' | ask neutrally, offer all answers |
| ambiguous | 'Do you read often?' | ask for a number in a time period |
| double-barrelled | '… quieter and cleaner?' | split into two questions |
| overlapping options | \(0\)–\(3\), \(3\)–\(6\) | \(0\)–\(3\), \(4\)–\(6\) |
| unbalanced scale | great / good / OK | equal positive and negative options |
Is the sample biased?
- Name the population the question is about.
- Name who was actually asked, and how they were chosen.
- Ask who is left out, and whether they would answer differently. If so, the sample is biased.
- Check the response: a low response rate or self-selection makes any result doubtful.
- Limit the conclusion to the group that was really sampled ('shoppers at that market on a Saturday').
Is the questionnaire fair?
- Read each question as a reader would: does it suggest an answer, use emotive words, or ask two things at once?
- Check the options: no overlaps, nothing missing, balanced.
- Sensitive topics: make the survey anonymous.
- Look for a conflict of interest: who paid for the survey, and who was asked?
(i) Adults leaving the town's gym one evening.
(ii) Every \(20\)th name on the town's electoral roll.
(iii) Adults who answer a poll on a local sports club's social-media page.
The population is every adult in the town. For each sample, ask who is left out and whether they would answer differently.
(i) Biased. It leaves out everyone who does not go to the gym, and people at a gym play more sport than most. The estimate would be too high.
(ii) Representative. The electoral roll lists the town's adults, and choosing every \(20\)th name has nothing to do with how much sport they play.
(iii) Biased. It is self-selected: only people who follow a sports club page and choose to answer take part. They are likely to play more sport than other adults.
(i) Don't you agree that the library should open on Sundays?
(ii) How often do you use public transport? Rarely / Sometimes / Often
(iii) Your age: \(10\)–\(20\), \(20\)–\(30\), \(30\)–\(40\)
(iv) Should the council fix the footpaths and plant more trees?
(i) Leading: 'Don't you agree' pushes people towards yes. Fair version: Should the library open on Sundays? Yes / No / Unsure.
(ii) Ambiguous: 'sometimes' means different things to different people. Fair version: How many trips did you take on public transport last week? \(0\), \(1\)–\(2\), \(3\)–\(5\), \(6\) or more.
(iii) Overlapping and incomplete: a \(20\)-year-old fits two boxes, and anyone under \(10\) or over \(40\) fits none. Fair version: under \(20\), \(20\)–\(29\), \(30\)–\(39\), \(40\) or over.
(iv) Two questions in one: someone who wants new footpaths but not trees cannot answer. Ask two separate questions.
(i) What percentage of households replied?
(ii) How many households said they oppose it, and what percentage of all \(1800\) is that?
(iii) The council reports '\(65\%\) of residents oppose the dog park'. Explain why this is unreliable.
(i) and (ii)
| \(\text{response rate}\) | \(=\) | \(\dfrac{360}{1800} \times 100\% = \mathbf{20\%}\) |
| \(\text{oppose}\) | \(=\) | \(65\% \times 360 = \mathbf{234}~\text{households}\) |
| \(\dfrac{234}{1800} \times 100\%\) | \(=\) | \(\mathbf{13\%}~\text{of all households}\) |
(iii) \(80\%\) of households did not reply. People who feel strongly, often those against a change, are more likely to send a survey back, so the replies are a self-selected sample. All we know for sure is that at least \(13\%\) of households oppose the park.
(i) What percentage of the replies say yes?
(ii) Why does this overstate support?
(iii) Assuming each suburb's percentage is accurate, estimate the support in the region, as a percentage and a number of adults.
| \(\text{yes}\) | \(=\) | \(80\% \times 600 + 40\% \times 200 = 480 + 80 = 560\) |
| \(\dfrac{560}{800}\) | \(=\) | \(\mathbf{70\%}\) |
(ii) Suburb \(A\) gives \(\dfrac{600}{800} = 75\%\) of the replies but has only \(30\%\) of the adults, and it is the suburb most in favour. The poll is self-selected and over-represents \(A\).
(iii) Weight each suburb's percentage by its true share of the adults.
| \(\text{support}\) | \(=\) | \(0.3 \times 80\% + 0.7 \times 40\%\) |
| \(\) | \(=\) | \(24\% + 28\% = \mathbf{52\%}\) |
| \(52\% \times 20\,000\) | \(=\) | \(\mathbf{10\,400}~\text{adults}\) |
Common pitfalls
Frequently asked questions
What makes a sample biased?
A sample is biased when some part of the population is left out or over-represented, and those people would answer differently from the rest.
Does a bigger sample remove bias?
No. A bigger sample reduces random variation, but if the way people are chosen is biased, adding more of the same people repeats the same bias.
What is a leading question?
A question that pushes people towards one answer, such as one starting with 'Don't you agree that'.
Why are online polls often unreliable?
The people who answer choose themselves, so the poll over-represents people who feel strongly about the topic or who visit that website.
Why use an anonymous survey?
People answer sensitive questions more honestly when nobody can tell which answer is theirs.
What is a double-barrelled question?
A question that asks about two things at once, so someone who agrees with one part and not the other cannot answer honestly.