Evaluation considerations
FACULTY OF ARTS &
SOCIAL SCIENCES
Tess Lea QEII Fellow
Department of Gender and Cultural Studies

1

Comes down to:
› What do we want to do?
› How much time have we got?
› Who have we got?
› What resources do we have?
› Who is our target audience?
› Best imaginable outcome for results?
What’s the best approach?

Some techniques
› Interviews
› Observations
› Focus Groups
› Small questionnaires with crisp, manageable questions
› Review of documents, records, media
› Limited archival research
› Performance based methods
Interactive role playing
› Follow up (cohort) surveys to map changing decisions/perceptions
› Participant led data gathering

› Focus group
Economical but risk getting dominant ‘group think’
› Observations
less intrusive but takes longer saturation to get to patterns
› Interview format/interactive performance based
flexible and rich but time intensive
› Survey
larger potential population
useful for longitudinal or cohort analyses
not ideal for understanding detail or social complexity
Also dealing with questionnaire-fatigued target groups
Pros and cons
5

Sampling: size versus depth
› If want larger sample, must tame the monster: ask mainly yes/no questions; have only one open-ended question
› If want detail, limit the sample
› Most reliable method for young people is face to face interviews and participatory research
› In any case, design data analysis before you administer instrument of choice
Clarify what data you actually want and ask for it, not what you think should be asked
Involve target group in the revisions
Anticipate and design the data analysis tools (the database and coding systems) before going live

Some protocols
› Research shows that poorly worded questionnaires bore young people, or provokes conventional answers – needs to be attractive and easily interpreted
› Also important to avoid having it be another piece of school work
› True consent not possible in power structures
› Thus even more essential instruments are well designed
Conduct a small trial: test the questions – do people respond as you had hoped?
Ask young people how they found it. Gauge reactions and modify
Limit time required to 15 minutes or less
Be aware of biased terms – e.g. ‘assistance to the poor’ (positive) versus
‘welfare’ (negative)
Keep it uncluttered – less is more
Think about the order – make it a logical progression, not choppy
7

What are the right questions?
› What is the age group? Gender? Social background?
› What data story do you need to be able to tell and with what ambition
(For further funding? Publications?)
Can’t over-state need for clarity about purpose
› Vignettes of transformational stories are excellent
Write with journalistic flair
› Combine these with other impact data
The more ‘factual’ and repeatable over time, the better
8

Youth oriented evaluations that yield convincing data
› Given reality of need to maximise responses in shortest time frame and at least cost, how to get the holy grail of youth-centred evaluations that yield convincing data?
› Many interventions presume a deficit (in esteem, in awareness, in comfort) and rush to fill that, without any primary research
› Foregrounding the students and making the evaluations as studentcentred as the interventions vital especially given academic persona as another teacher/adult
› Be creative
9

2

Widening Participation
Are we making an impact? How known?
What is meant by/needed for sustainability?
Is it worth expanding/scaling?
What do we not want to be doing more of?

Brainstorming time
› Group discussion of project status in relation to the timelines and matching methods to questions to desired outcomes
› Contact details: tess.lea@sydney.edu.au
; 9351 6777