Evaluating Educational
Interventions
Paul Stevenson
Background
• Local evaluation of EBP
training showed:
Skills & Knowledge
improved ….
..but limited change in
behaviour.
•Miller, G E The assessment of clinical skills/performance
•Academic medicine, 1990, 65, S63-7
• How can I make the training more effective?
• I had used the Kirkpartick Model for Evaluation
Kirkpatrick Model
Kirkpatrick Model – Learner Focused
Level 1 – Learner reaction
Level 2 – Modification of learner attitudes & perceptions,
and learner acquisition of knowledge & skills
Level 3 - Change in learner behaviour
Level 4 - Benefits to the organisation/patient resulting
from learner performance
Kirkpatrick’s four levels of evaluation (1967,1994), adapted by Barr et al (2000)
Kirkpatrick Model
Tendency for declarative outcome measures
Shows if the intervention has been effective…
…but provides little information about why
Learner Focused
Fails to consider contextual/environmental variables
View that training is solely responsible for any outcome
One Directional
Tendency to focus on the impact of the learner on their
environment
• What alternative methods could be used?
Transfer of Learning
Baldwin & Ford (1988) transfer of training into practice.
Three areas of influence on transfer :
1. Aspects of the course
2. Characteristics of the learner
3. Features of the work environment
Common Barriers
• The most commonly mentioned barriers to
EBP are:
Learner
Training
Environment
•Personal lack of interest
•Appropriate content
•Organisational culture
•Lack of skills
•Delivery method
•Insufficient time
•Low self-efficacy
•Lack of resources
•Previous experience
•Lack of support from
colleagues and management
Salbach et al, 2007; Pollock & Legg, 2000; Jette et al, 2003; McColl & Smith, 1998; Freeman & Sweeny,2001; Retsas, 2000;
Cranney & Warren, 2001; Metcalf et al, 2001
Diffusion of Innovation
Diffusion of innovation model, Rogers (1995)
LET evaluation
Rather than focus solely on the impact of the learner on the
Lewin’s (1951) equation
(P,E) the impact of the environment on
environment,
we alsoB=f
measure
Behavior
is a function of the person and the environment
the
learner.
Adapting this to evaluate the impact of an educational
intervention we get: BC= f (L,E,T)
Behavioral change is a function of:
L – Learner E – Environment T- Training
Systems Approach
• System: Integrated whole composed of several
parts
• Look at the educational intervention holistically
• Look for relationships with other parts of the
organisation (L&D, HR, Clinical Audit)
Example Measures
Learner:
Pre-post course evaluation form
Test exercise/paper to show learning has occurred
Likert/VAS measure of self-efficacy, desire for change and
other psychological aspects of the learner.
Post-course follow up – has learning been retained, examples
of how training has impacted on work, reasons/barriers if new
learning has not been used.
Monitor usage statistics such as Athens Usage
Example Measures
Environment:
Floor Walking – observe the working environment: is there
access to appropriate resources, space/time for staff to
implement EBP.
Dialog with Line Management – Creates a two way exchange
of information
Integrate Library Services with the wider organisation – If you
are working in the same environment as your learners you will
experience the culture/environment first-hand.
Example Measures
Training:
Pre-post course evaluation form
Test exercise/paper to show learning has occurred
Post-course follow up – has learning been retained, examples
of how training has impacted on work.
Peer-observation
Advantages
Truly understand the wider impact of the training
Identify and have a better understanding of the
barriers which exist and can start to resolve them.
Increases profile and importance of library
services within organisation / departments
Further Information
paul.stevenson@bradford.nhs.uk