Government Social Research Unit
Randomised Controlled Trials Conference
University of York, 14-15 September 2006
Why Governments Need RCTs
Philip Davies PhD
Government Social Research Unit
HM Treasury
London SW1A 2HQ
www.gsr.gov.uk
1
Answer
Because RCTs can help get policy initiatives
right more often than wrong
www.gsr.gov.uk
2
Caveats
• Governments have got to want to get things right
• ‘Getting things right’ is more than achieving an effect
size
• Governments do things other than achieving impacts
• RCTs are not a panacea
• RCTs are only as good as their design and
implementation
www.gsr.gov.uk
3
Governments Have Got to Want to Get Things Right
“Social science should be at the heart of policy
making. We need a revolution in the relationship
between government and the social research
community - we need social scientists to help
determine what works and why, and what type of
policy initiatives are likely to be most effective …”
(David Blunkett, 2000)
www.gsr.gov.uk
4
Governments Have Got to Want to Get Things Right
Government policy must be:
• Evidence-based
• Properly evaluated
• Based on best practice
www.gsr.gov.uk
5
Governments Have Got to Want to Get Things Right
• Policy making must be soundly
based on evidence of what
works
• Improve departments’ capacity
to make best use of evidence
• Improve the accessibility of the
evidence available to policy
makers
6
Governments Have Got to Want to Get Things Right
Professional Skills for Government (2005)
Analysis and Use of Evidence
•
•
•
•
•
•
Anticipate and secure appropriate evidence
Test for deliverability of policy/practice – and evaluate
Use evidence to challenge decision making
Identify ways to improve policy/practice
Champion a variety of tools for collecting/using evidence
Ensure use of evidence is consistent with wider government
requirements
• Work in partnership with a wide range of experts/analysts
7
What ‘Getting It Right’ Means
The ‘Experimenting Society’
(Donald T. Campbell)
“…a society that would use social science methods and
evaluation techniques to vigorously try out possible
solutions to recurrent problems and would make hardheaded, multidimensional evaluations of outcomes, and
when the evaluation of one reform showed it to have
been ineffective or harmful, would move on and try
other alternatives”
8
What ‘Getting It Right’ Means
“A social experiment benefits society by providing better
information on which to base public policy. Such
information can improve policy in one of two ways: it
can lead policymakers to adopt a program or policy that
is found to have net social benefits, or it can lead to the
termination of an existing program that is found to have a
net social cost”
(Orr, L., 1999).
9
Why RCTs?
(Recap)
Randomised Controlled Trials:
• Provide most precise estimates of the likely effects of
policy interventions.
• Against a strong counterfactual – i.e. the net effect/impact
• Reduce/account for various biases.
• Establish cause and effect of policy interventions and
outcomes.
• Raise the quality of evidence to support policy making.
10
So, Why Don’t Governments Use RCTS More Often?
• RCTs not on most policy makers’ radar
• RCTs not on many government researchers’ radar
• RCTS not on many academic researchers’ radar
• For some (many?) RCTs are anathema:
 Don’t like ‘random’ (too ‘chancey’)
 Don’t like ‘control’ (no ‘wiggle room’)
 Don’t like ‘trials’ (sounds inhumane / 1940s Germany)
• There are things other than ‘impact’
• There are things other than ‘evidence’
11
Things Other Than Impact Evidence
Experimental
Quasi-Experimental
Social Ethics
Public Consultation
Ethical
Evidence
Experimental
Qualitative
Theories of Change
Impact
Evidence
Implementation
Evidence
Research
Evidence
Cost-Benefit
Cost-Effectiveness
Cost-Utility
Economic
and
Econometric
Evidence
Statistical
Modelling
Surveys
Admin Data
Comparative
Qualitative
Descriptive
Analytical
Evidence
Multivariate
Analysis
12
Things Other Than Evidence
“Research of any kind is seldom the
determining factor in shaping public
policy. Experimentation is no exception
to this rule” (Orr, L., 1999)
13
Things Other Than Evidence
Pragmatics &
Contingencies
Lobbyists &
Pressure Groups
Habits &
Tradition
Experience &
Expertise
Judgement
Evidence
Resources
Values and
Policy
Context
14
Different Notions of Evidence
Policy Makers’
Evidence
Researchers’
Evidence
• Colloquial (Contextual)
• ‘Scientific’ (Context free)
• Anything that seems reasonable
• Proven empirically
• Policy relevant
• Theoretically driven
• Timely
• As long as it takes
• Clear Message
• Caveats and qualifications
15
Where Do Policy Makers Go For Evidence?
Policy or Practice
Special Advisers
'Experts'
Think Tanks/Opinion Formers
Lobbyists and Pressure Groups
Professional Associations
Media
Constituents, Consumers and Users
16
Academics
But Not All Is Lost
Examples of Recent UK Government RCTs
• Employment Demonstration Project (ERA)
• Job Retraining and Rehabilitation Project (JRRP)
• Restart evaluation (1990)
• BA Visiting Officer Pilot
• Evaluation of Employment Zones
• New Deal 25+ Pilots
• Evaluation of the New Deal for Young People Intensive
Gateway
• Evaluation of the Restorative Justice programme
17
RCTs Are Only As Good As Their
Design and Implementation
Common Design and Implementation Faults:
•
•
•
•
•
•
•
•
•
Unclear policy objectives, populations, outcomes
Insufficient time  (lack of strategic policy making)
Test condition often pre-determined
Sample sizes too small – no statistical power
Problems of randomisation, concealment, cross-over
High attrition (especially in control groups)
Intention to Treat analysis not used
Limited capacity (in the UK) to do high quality RCTs
Poor counterfactual too often the result
18
RCTs Are Only As Good As Their
Design and Implementation
These faults can usually be overcome!
•
•
•
•
•
•
•
•
•
Better strategic policy making
Better planning and protocol development
Involvement of researchers, policy makers, front-line staff
High quality piloting / pre-fieldwork testing
High quality research/project management
Careful real time data management
Use of experienced technical experts throughout
Policy Trials Service
Professional development of researchers and policy makers
19
Take Home Message
• RCTs have enormous potential for governments
• RCTs can help governments get things right more often
• RCTs are currently used considerably to UK Govnt.
• But there are still non-believers
• And those who remain to be convinced
• RCTs, like other types of research, must compete with
other notions of evidence
• And with factors other than evidence
• There is a need to improve the capacity to do high
quality RCTs in the UK
20
Contact
philip.davies@hm-treasury.x.gsi.gov.uk
Government Social Research Unit
HM Treasury
1 Horse Guards Road
London SW1A HQ
England
Tel: +44 (0)20 7270 5156
www.gsr.gov.uk
21