Part 1 Non-sampling errors in polls & surveys
EVALUATION OF
STATISTICAL
REPORTS
Non-sampling errors as they apply to polls
and surveys and observational studies
This unit is designed to be taught over a one and a half week period. A possible teaching plan
is given below and associated resources are provided in the Resource Pack.
1
Unit Overview
UNIT OVERVIEW
Lesson
1
Topic
Introduction
2
Sampling in
the wider NZ
population
Non-sampling
errors
Literacy
Strategies
3
4
5
6
7
Practice using
worry Qs and
writing frame
Practice with
less familiar
context
Introduce
Political polls
Practice Exam
style question
Description
Class poll to review and extend
ideas on sampling
Structure of a media article: driving
infringements article + template
Extend ideas on sampling to
samples of the NZ populationdistribution of key characteristics
Power-point presentation with
examples and stories
Pre-reading to get the gist, read
aloud asking worry Qs, note-taking
(e.g.Road Tolls article + worry Qs sheet
+ writing frame)
e.g.DVT article: What is DVT?
get the gist, ask and answer worry
Qs, note 2 positive aspects and 2
concerns
e.g.ANZUS article: What is ANZUS?
Reading to gain meaning from
context
Critically evaluate article
Power-point presentation on
research companies-compare
methodologies
Compare 2 recent political polls
from different companies
See new NZQA exemplar
2
Teaching
Notes

Notes about activity
objectives will be
made in this panel.
References to
resources used such as
power-point slides will
be referenced in this
panel.
How is a typical media report on a poll or survey organised?
Read the media article and identify the following information: (Hint: draw round the
information on the article as you locate it and number the section using the codes below)
1. The main finding of the survey
 What is it?
 Where in the article is it located?
2. Who funded (commissioned) this survey?
 Why do you think this survey was carried out?-what is its purpose?-who are the target group?
3. Method: Who was sampled ?
Who conducted the survey?
Sample size
Random/self-selection/man-in-street
Telephone/on-line/face-to-face/posted
Margin of error:
Date:
4. Questions Where located –in graphic/ in text/ both ?
Are the questions clearly stated?
Can they be inferred from reading the text?
5. Summary of results Where located?-e.g. graphic/ text/ both
6. Comment from an expert Who was the expert(s) consulted in this report?
7. Links to other studies/surveys?
 What other links are made?-do they seem appropriate?
8. Any other information? Is there any other information not yet highlighted?How would you describe it?
Read this report with a critical eye.
Comment on 2 good aspects
Comment on 2 concerns
Are there aspects of this report which you are unfamiliar with?
3
“Worry” questions for evaluating a Poll or Survey
1. Source:
 Who funded it?
 Who carried it out?
Potential for a hidden agenda
2. Methodology:
1. Who is the target group?
2. Who was sampled?
 How was the sample selected?- random/representative?
self-selection/man-in-street?
- Phone/online/face-to-face
Potential for selection bias, self-selection bias, interviewer effects
 What was the sample size?
 margin of error?
Potential for non-response bias
3. What questions were asked?
Potential for question effects, survey format effects, behavioural bias
 How were the “measures” defined?
e.g. a “measure” could be ‘bullying’, ‘violence on TV’
4. What are the findings?
 are graphics/ summaries appropriate ?
5. What claims (inferences) are made?
 are the claims for the target group?
Potential for transfer of findings
 can I verified the claims made?
6. What information is missing?
7. Am I willing to believe the findings of this survey or do I need more
convincing?
Can I make a judgement and justify my judgement with at least 3 pieces of evidence
from 1 to 6 above?
4
Writing frame for critically evaluating a report about a Poll or Survey
Pre-Reading: “Getting the gist”
Read the media report and summarise what it is about in 3 sentences or bullet points.
While reading: “Worry Questions”
Read the media report again, asking appropriate “worry questions” as you go.
Record your answers in the boxes below:
Source
Method
Target Group
Who was sampled?
Selection method
Sample size
Margin of error
Questions asked
Key Findings
Claims Made
What is missing?
Critical Evaluation:
Discuss 2 good aspects of this report
Discuss 2 concerns
5
Part 2 Observational studies and Experiments
Unit Overview
UNIT OVERVIEW
Lesson
1
2
Topic
Introduction
Experiments
3
Experiments –
design issues
Observational
Studies
Experiment vs OS
4
5
6
7
8
Other
Resources
Other
Resources
Evaluation of
Report
Attack of killer
frying pans
Exam Style
Questions
Teen drinking
may cause
irreversible brain
damage
More exam style
questions &
review
Description
Review & discuss some media articles
What are they? Causal claims. Points to Ponder + identification in
media articles. Potential Problems
Design Issues - Control Group, Placebo effect. Blinding. Review
articles that use these terms
Look forward/back. Why OS conducted in preference to an
Experiment? Discussion of potential problems
Summarise differences between the two types of studies. Review
articles and classify study type with justification.
Evaluation steps, Worry questions, Work-sheet.
Begin first in depth evaluation.
In depth evaluation of media report. Examples of possible student
responses provided
Examples of shorter exam style
Second in depth analysis of media report.
Examples of shorter exam style questions & review of Unit as a
whole.
In an assessment a student will not have to read a complete report but will have to evaluate
selected extracts from media reports, press releases and research papers.
This teaching unit will demonstrate in-depth evaluations of media reports and will then give
examples of assessment style questions.
Assessment style questions will usually contain only an extract from an article and will focus on
just a few of the evaluation steps.
6
IN-DEPTH EVALUATION OF STATISTICAL REPORTS
Evaluation of a statistical report will involve students answering
a number of questions. These are presented in the Resource Pack
Teaching
Notes
under ‘Worry Questions’. Students can use these as a template
to get started on evaluation but they will not be available in an

assessment.
LESSON 6 – Activity 1 – How to evaluate a report
Worry question page
in Resource Pack.
Often useful to print
Evaluation Steps
this in a different
Step One
Is the report based on sound research? Find the
colour, then can say
original research article to determine this. Was it published in a
“Use the yellow page
reputable Journal? Was it conducted by reputable researchers
to evaluate your
from a University or Research Institute for example?
report”
Step Two
Identify the type of study undertaken –
experiment or observational study?
Slide 16 covers steps 1
Step Three
Who funded the research? Did they have a vested
–5
interest in the outcome? For example, was it a drug company
trying to compare their new drug over an existing treatment?
Step Four
Were reliable data collection methods used? If
Reputable journals
data was collected from the past, how was accuracy checked?
include The Lancet,
Could confounding variables have changed over time?
BMJ or any other peerreviewed journal.
Step Five
Who or what was studied? How were they
selected?
Step Six
What measurements were taken and how? Are
questions defined and responses provided? Are the
measurements appropriate for the effect that is being observed?
For example, is an IQ test an acceptable measure of intelligence.
How were measurements taken – interview, online,
questionnaire. Could this have had an impact on responses?
Step Seven
Where and when were measurements taken or
Context is crucial in
engaging students.
Start by evaluating
articles about familiar
contexts and move
towards more
questions asked? Sometimes if measurements are taken in an
unfamiliar contexts.
unfamiliar setting for the participant, atypical responses can be
Unfamiliar contexts
observed. Some measurements will vary according to the time
will require some
they are taken – morning/evening, summer/winter,
research prior to
weekend/weekday.
evaluation.
Step Eight
Is the size of the observed effect quantified or just
described? Is a claim made that Drug A reduces blood pressure
7
OR that Drug A reduces blood pressure by 15%. Beware of
unquantified claims.
Step Nine
Teaching
Notes
Are there any confounding variables that should be
considered?
Step Ten
Have the results been extended inappropriately?
Step Eleven
Is the information complete? Was non-response or
missing data a problem?
Step Twelve

Slide 17 covers steps 6
Do the results make sense? Is there a scientific or
– 12
biological reason to support the evidence? Is there any alternative
explanation.
Step Thirteen Would the results persuade you or someone you know
Slide 18 cover steps 13
to modify or change their behaviour?
& 14
Step Fourteen Overall conclusion about the research findings. Justify
why the study was good or why it was not. Do other studies reveal
similar findings? What further information about the study might be
useful.
Worry Questions &
Discuss Worry Questions and Report Evaluation template and
Report Evaluation
demonstrate relationship between the two and the steps identified
template
above.
8
WORRY QUESTIONS FOR EVALUATING MEDIA REPORTS ABOUT
OBSERVATIONAL STUDIES
1. Source
 Who funded it?
 Who were the researchers? Do they work for a reputable organisation?
 Was there a hidden agenda?
 Have you found original research paper or press release?
2. Methodology
 Were individuals studied over a period of time?
 How was data collected?
3. Potential Problems
 Confounding variables – as randomisation not used, it is impossible to
separate effects of treatment from those of confounding variables. A causal
link cannot be claimed.
 Extending results inappropriately – results of an observational study cannot
be extended to a larger population unless the participants were randomly
selected and can be considered representative of that larger population.
 Using the past as a source of data – often participants cannot recall details
from the past particularly accurately. If past records are used, such as
medical records, then accuracy is not such a problem.
4. Critical Components
 Is non-response a problem? Have non-response figures been provided? Are
the non-respondents different from the participants in any way?
 Are there any missing values? Could these values show different results?
 Do the results make sense? Can you think of any scientific or biological
reason that might support the results found?
 Can you think of any alternative explanation for the results?
 Would the results persuade you or anyone you know to modify their
behaviour?
5. Conclusion
 Given that a perfect observational study is difficult to achieve, how do you
rate this one?
 Can you suggest any improvements to the study?
 Do you have any suggestions for what researchers could look at next?
9
MEDIA REPORT EVALUATION
Media Headline
Observational Unit
Explanatory variable
Response variable
Treatment(s)
Study Type
Original Source Found - YES/NO
Design of study - describe methodology of observational study
OBSERVATIONAL STUDY - potential problems
Evaluate critical components
CONCLUSION - good or bad study? Any areas of improvement?
10
EXAMPLE OF ASSESSMENT STYLE QUESTION
Extract from Newspaper article
“GRADUATING IS GOOD FOR YOUR HEALTH”
College graduates feel better emotionally and physically than their high-school drop-out
counterparts, a study of over 400,000 US adults reveals. Participants were asked a series of
questions including level of education and on how many of the past 30 days they felt physically
and emotionally healthy.
1. Define the explanatory and response variables
2. Explain whether any of the potential problems that observational studies can suffer from such
as using the past as a source of data, extending results inappropriately or confounding
variables, might apply to this study.
3. Are there any concerns regarding the methodology: who was studied and how measurements
were taken? Discuss this in relation to this article.
11
EXEMPLAR PAPER 2013
QUESTION THREE
Obesity linked to less sleep in childhood
Less sleep in childhood equals more obesity in adulthood. That is the conclusion Otago
University researchers have come to – showing a possible link between the obesity epidemic and
a suggested decline in the number of hours children are sleeping.
The Dunedin Multidisciplinary Health and Development Study used data collected from 1 000
Dunedin people born between 1972 and 1973. The researchers used the "body mass index" of the
group at 32 years old when 53 per cent were overweight including 18 per cent who were obese.
They set those data against "time in bed" data from group members when they were 5, 7, 9, and
11 years old.
After discounting lifestyle factors, the study showed that "short sleep time in childhood was
associated with an increased adult body mass index". "Our findings suggest that sleep restriction
could be a plausible contributing factor to the current epidemic of obesity," the study said.
The paper acknowledged that the correlation between lack of sleep in childhood and obesity in
adulthood was "weak". "In view of the increasing prevalence of obesity, an effect of this size may
have important public health implications." Researcher Dr Erik Landhuis said it was well known
children needed an "adequate" amount of sleep.
Source: Otago Daily Times, November 4, 2008 (abridged)
(a)
Identify the explanatory and response variables for the study. Discuss any potential issues
with the measurement of the explanatory variable.
(b)
Explain whether this study is an observational study or an experiment. Give an implication of
using this type of study for the relationship investigated.
(c)
The report begins with the statement “less sleep in childhood equals more obesity in
adulthood”. Comment on the quality of this statement.
(d)
The report states that after discounting lifestyle factors, the study showed that "short sleep
time in childhood was associated with an increased adult body mass index".
Give examples of two related lifestyle factors that may need to be taken into account for this
study, and explain how these may be confounding variables.
12