Unit 01 · Day 3
Fair Tests, Controls & Sampling
Design comparisons that reduce alternative explanations, collect enough evidence to expose variation, and sample in ways that actually match the population being studied.
Learning objectives
- Explain when a control or comparison condition strengthens an investigation and when a separate control is not meaningful.
- Distinguish repeated measurements, repeated trials, sample size, and independent replication.
- Identify convenience sampling, volunteer bias, undercoverage, and other selection problems.
- Use random selection or random assignment appropriately rather than treating the words as interchangeable.
- Revise a flawed investigation and justify how each repair improves the strength of the evidence.
Materials
- Science notebook or digital portfolio
- Coins, shuffled numbered slips, or another simple randomizer
- Provided investigation and sampling scenarios
- Calculator optional for proportions and sample summaries
This lesson uses design scenarios and low-risk randomization only. Do not collect private personal information, involve human participants in unapproved experiments, or substitute an unsupervised biological, chemical, electrical, or physical procedure for a paper design exercise.
1. A fair comparison is a design argument
A fair test is not a perfect test. It is a comparison in which the intended explanatory difference is clear and important alternative explanations have been reduced, measured, randomized, matched, or openly acknowledged.
Comparison: several rotor lengths made from the same paper template.
Held sufficiently consistent: body width, paper type, paper clips, release height, timing rule, and indoor location.
Still possible: small fold differences, human release variation, and stopwatch reaction time. These become uncertainties to manage rather than facts to hide.
A control condition is useful when it provides a meaningful baseline. In some descriptive or observational studies, the stronger design may be a carefully defined comparison group rather than a traditional “control.”
For each case, decide whether the study needs a control, a comparison group, or neither: (a) testing a new fertilizer, (b) mapping average noon temperature across a school campus, (c) comparing two insulation materials, (d) measuring lunar phases over a month. Explain one choice.
2. Four quantities that students often confuse
| Term | Meaning | Example |
|---|---|---|
| Repeated measurement | Measure the same item or event more than once. | Measure one ruler-marked object three times. |
| Repeated trial | Run the procedure again under the same condition. | Drop the same helicopter design ten times. |
| Sample size | Number of distinct units, organisms, sites, people, or observations represented. | Measure 40 different leaves. |
| Independent replication | Repeat the investigation with a new set of trials, samples, investigators, times, or settings. | A second class repeats the helicopter investigation independently. |
More trials can improve precision and reveal variation, but they do not automatically make a biased, confounded, or badly measured design valid.
A learner measures the same incorrectly calibrated thermometer 100 times and reports a very stable average. Is the evidence precise, accurate, both, or neither? What would improve the design?
3. Sampling must match the population
A population is the full group relevant to the question. A sample is the subset actually observed. A large sample can still mislead if the selection method systematically excludes part of the population.
- Convenience sample: selected because it is easiest to reach.
- Volunteer sample: people choose themselves, which may overrepresent strong opinions or particular characteristics.
- Undercoverage: important groups have little or no chance to be included.
- Random selection: uses chance to select members of a population and can improve representativeness when the sampling frame itself is appropriate.
A student wants to estimate average backpack mass for Grade 9 but measures only five students arriving early for sports practice. Identify at least three weaknesses. Then propose a practical sampling plan that gives a broader cross-section of Grade 9 students a chance to be selected.
4. Random selection is not random assignment
Random selection addresses who or what enters a sample. Random assignment addresses how experimental units are placed into conditions. They solve different design problems.
Experiment: Randomly assign already-selected seedlings to light conditions → treatment-allocation question.
Create eight labels numbered 1–8. Shuffle them or use a coin-based rule to assign four labels to Condition A and four to Condition B. Record the rule before you randomize. Explain why changing the rule after seeing the assignments would weaken the design.
5. Detect the hidden flaw
For each design below, name the strongest alternative explanation and propose one repair.
- A plant-growth study gives Group A more light and a different soil than Group B.
- A noise study compares Monday morning readings with Friday afternoon readings.
- A survey about school lunch quality is posted only in the culinary club.
- A material-strength comparison uses one sample of Material A and twenty samples of Material B.
For one scenario, complete: “If we do not repair ________, then the result could be explained by ________ instead of ________.”
6. Interactive Science Studio
Use the Science Studio on this page to support—not replace—your design reasoning. The Scientific Calculator may help check proportions or summary values. Atlas Evidence Analysis or the Investigation Design Task can help organize evidence, assumptions, alternative explanations, and design repairs.
Use one recommended tool only after writing your own design first. Record one thing the tool helped you notice and one limitation the tool could not decide for you.
7. Mentor conference
Before opening the Mentor, prepare a concise design defense:
- What population or system are you trying to understand?
- What comparison makes the question answerable?
- Where could selection bias or confounding enter?
- What would count as enough replication to make the evidence more trustworthy?
Ask the Mentor to challenge one assumption in your sampling or comparison plan. Record the challenge and your revision.
Portfolio evidence
Return to the investigation you began on Days 1–2. Add or revise:
- population or system of interest;
- sample or experimental units;
- comparison/control condition;
- trial count and what counts as an independent trial;
- sampling or assignment method;
- at least two possible sources of bias or alternative explanation;
- one design limitation that will remain even after your revisions.
Exit check
- Why can 1,000 observations from a biased sample still give a misleading conclusion?
- What is the difference between random selection and random assignment?
- When would a comparison condition be more appropriate language than a “control group”?
- Explain why repeating a flawed procedure does not repair the flaw.