Bias and Questionnaires
A biased sample or a badly worded question can make a survey give the wrong answer, however carefully it is analysed. This NSW Year 9 Core maths subtopic examines both, and goes beyond the Core outcomes, coming from NESA outcome MA5‑DAT‑P‑01.
You will decide whether a sample is biased or representative by naming the population first and asking who could never be chosen, as in a survey about dog ownership taken at a dog park. You will explain which way bias pushes a result, see why a huge self-selected online poll still cannot speak for the whole country, find problems with leading or vague questions such as asking whether you exercise often, fix overlapping or missing answer options, and pick the fairest survey plan. It is part of Year 9 Core Investigating Data.
Every question with a fully worked solution.
- Bias and Questionnaires - Video - Bias and Questionnaires Watch
Theory
A survey is only as good as who was asked and how they were asked. This Year 9 maths page shows how to tell a biased sample from a representative one, say which way the bias pushes the result, and fix survey questions that are leading, vague or have faulty options. This content comes up if you're heading for the Standard or Advanced path in Year 10.
Bias is anything that makes a sample, or a question, lean towards one answer.
A representative sample is like the whole population. Every part of the population has a chance to be chosen, and the way people are chosen has nothing to do with the question.
A biased sample leaves out a whole sub-group. The classic case is surveying at a place linked to the question, such as asking about pets at a pet shop. Everyone there already leans one way.
A self-selected sample, such as an online poll, is made up of the people who choose to answer. People with strong views reply and others do not, so the result cannot be generalised. A huge number of replies does not fix this.
A questionnaire can be biased too, even with a perfect sample. A fair question is neutral, precise, and has a box for every honest answer.
| Fault in a question | Why it spoils the data |
|---|---|
| leading wording, such as “Don't you agree…” | it pushes the reader towards one answer |
| a vague word, such as “often” | each person reads it differently |
| overlapping options, such as \(0\)–\(5\) and \(5\)–\(10\) | one answer fits in two boxes |
| missing options | some honest answers cannot be given |
| two questions in one | a single answer cannot cover both |
How to judge a survey
- Name the population the conclusion is about.
- Describe who was actually asked, and how they were chosen.
- Ask who could never be chosen. If they differ on the question, the sample is biased.
- Say which way the bias pushes the result: too high or too low.
- Read the question twice: once for what it pushes you to say, and once to check your honest answer fits exactly one box.
- Suggest a fix: a random or systematic sample from the whole population, and neutral, precise wording.
Name the population, describe the sample, then ask who is left out.
| population | \(:\) | everyone in the city |
| sample | \(:\) | people at a bicycle shop |
Everyone who never goes near a bicycle shop is left out, and that is most of the people who do not ride. The sample is biased, and the estimate will be far too high.
Name the population, then look at who is at a skate park.
| population | \(:\) | everyone living in the town |
| sample | \(:\) | people at the skate park |
A skate park is used mostly by young people, and older residents are rarely there. The sample leans young, so the average age will come out too low.
A fairer method: choose addresses at random from a list of every address in the town and ask the age of each person living there. Every resident then has a chance of being included.
Read the wording and ask what it pushes you to say.
| “Don't you agree” | \(:\) | assumes a yes |
| “excellent” | \(:\) | praises the library first |
Both push the reader towards agreeing, so the result will overstate support. It is a leading question.
A neutral version: “Should the library's opening hours be extended, reduced, or left as they are?”
Check where one option ends and the next begins, then check the two ends.
| \(1\)–\(10\) and \(10\)–\(20\) | \(:\) | both hold \(10\) |
| \(10\)–\(20\) and \(20\)–\(30\) | \(:\) | both hold \(20\) |
| more than \(30\) | \(:\) | no box at all |
The options overlap and do not cover every answer. Use \(1\)–\(10\), \(11\)–\(20\), \(21\)–\(30\) and “more than \(30\)”.
Common pitfalls
Frequently asked questions
What is a biased sample?
A sample that leaves out part of the population, or favours one part, so the results lean one way. Surveying people at a place linked to the question is the most common cause.
How do you know if a sample is representative?
Every member of the population must have a chance to be chosen, and the way people are chosen must have nothing to do with the question. Choosing at random from a list of the whole population does this.
Why are online polls unreliable?
The people who answer choose themselves, so mostly people with strong views reply. However many people click, they do not represent everyone.
What is a leading question?
A question worded to push people towards one answer, such as one starting “Don't you agree”, or one that praises or criticises the thing being asked about.
What makes a good survey question?
It is neutral, uses precise words instead of vague ones like “often”, and offers options that do not overlap and cover every possible answer.