Evaluation of the project
Annamari Ylonen and Brahm Norwich
Session outline
Evaluation at different levels
Goal Monitoring and Evaluation – principles and
guidelines
Ethnographic research – what is this and what does it
involve?
Process evaluation
Ethical issues
Timeline of evaluation activities
One survey to be filled in: concepts of inclusion & MLD
Different levels of evaluation

Pupil level
o Goal Monitoring and Evaluation (GME) by teachers
as part of the Lesson Study
 Teacher level
o Questionnaires /surveys : online (Survey-Monkey),
email or hard copies
 Classroom and school level
o Ethnographic research: observations of LS meetings
and classes; some interviews
 Process level (‘realistic evaluation’)
o A survey/interviews about the Lesson Study process
Goal Monitoring and evaluation

Developed from Goal Attainment Scaling (GAS) for the
Lesson Study project
 Method of monitoring the learning outcomes of the case
pupils (2 in each class identified as having MLD)
•2-3 research questions for each pupil (related to LS aims)
•3 goals for each pupil (related to LS aims and RQs). Within each
goal, 3 levels: 2 before the LS (baseline and expected levels) and 1
after the LS (an achieved level)
 The process is teacher/LS team-led
How?
Detailed instructions are in the pack together with a template to be
filled in for each Lesson Study/case pupils
Example of a GME goal and levels
GOAL 1
Be more frequently engaged in group activities when emphasis is on using key vocabulary
Rating scale:
Low
Baseline
Expected
Achieved
High
X
X
X
Baseline
descriptor
Is often disengaged in group activities when she needs to use vocabulary
Specific
evidence for
descriptor
Expected
descriptor
Specific
evidence for
descriptor
Achieved
descriptor
Specific
evidence for
descriptor
Because of problems in written and verbal language tends to be disengaged
and not participate
More frequent engagement in group activities
Increased participation, improved speech, increased confidence
More frequent engagement in group activities
Participated more during question and answer sessions; vocabulary still
limited but conversations contains more basic structure; more willing to write
and less anxious about her work
Ethnographic research
 Real time observations of the Lesson Study process (LS team
meetings and classes) in schools
 The focus is on the process, not on outcomes
How?
School visits (estimated 2-4 visits per school in each LS)


Observations in a small number of case study schools
Short interviews:
a. with teachers, other LS team members and some senior leaders:
views about the process, impact on pupil learning, issues arisen
etc.
b. with the case pupil focusing on their views about LS and how it
affects their learning
Ethnographic research: case study schools
 There will be 5 case study schools (same for both LS
cycles)
 How are these schools selected?
• By distance to Exeter: time constraints/logistics –
easy access
• If your school fits this criteria, you may be asked to
take part
 How are the visits planned?
• By negotiating suitable times to fit within teaching
timetables etc.
• Suggestion: initial visit ASAP to plan dates/times
Process evaluation
Focus on the LS process at teacher and school levels
 why and how the LS works in schools/impact on teaching
and teachers
 construction of process theory (links between contexts,
mechanisms and outcomes)
How?
1. An online survey at the end of the project (Feb/March
2012) OR at the final conference
2. Short interviews with a sample of teachers – possibly by
phone
Ethical issues
 The research and evaluation in the project follows
ethical guidelines (by the British Educational
Research Association)
 Consent from research participants:
• Understand what the project is, what it is aiming
to do and its purposes
• What taking part involves
• Right to confidentiality and anonymity (e.g. in any
subsequent reporting of research findings)
• Right to withdraw from the study at any time
 For case pupils interviews: written parental consent
• template available from project website
TIME
PERIOD
STUDENT
TEACHER
ETHNOGRAPHIC
PROCESS
EVALUATION
AT THE 1st
CONFERE
NCE
n/a
1. MLD teaching
approaches
questionnaire
2. Inclusion
survey
n/a
n/a
LESSON
STUDY 1
GME – by
teachers
Visits to case study
schools
(observations and
interviews
LESSON
STUDY 2
GME – by
teachers
Visits to case study
schools
(observations and
interviews
AT THE
FINAL
CONFERE
NCE
(OR
AFTER)
GME – final
details (if
not
finished)
1. MLD teaching
approaches quest.
repeat
2. Inclusion survey
repeat
OTHER
School
identification
of MLD
questionnaire
Process survey
(if not done)
Process
interviews
(sample)