Tensions of Measurement:
community investment activities
in housing associations
Funded by:
Hosted by:
David Mullins and Vanessa Wilkes, TSRC
Housing Studies Association Conference
15th April 2011
Summary
• Sector Context
• Evidence Base
• Levels of Measurement
– 5 levels of measurement
– Questions and Issues
• Tensions
– External Drivers
– Internal Drivers
• Conclusions
• Further Research
Sector Context
• Well established industry, large enterprises
(50% market share in social housing)
• Entrenched hybridity (social & commercial)
• Core business highly regulated & measured
• Non-core ‘community investment’ (CI)
activities symbol of independence
• Legitimacy & control issues brought to fore –
2008 Housing & Regeneration Bill & NHF Audit
• New tools emerging to measure social impact
of CI activities (in-house & off the shelf)
Evidence Base
• Scoping study – emerging approaches to
measurement of social performance (2010)
–
literature review & checklist (17 tools to set priorities, measure
outcomes or link these to corporate goals)
– 7 Case studies (organisation/tool based: 3 in-house, 4 outsourced)
– 2 workshops (now 3, plus follow up tracking e.g. of first sector specific
tool)
• NHF Neighbourhood Audits 2008, 2011
– Sector overview of CI (64% coverage by stock)
– Inputs/Outputs (£435mil CI in 6,800 projects 2008; leverage 1.6:2.7)
• Linked PhD
– Impacts & outcomes, depth case studies
– More on this later!
The levels of impact measurement
OECD definition of impact measurement
“the impact of all changes, positive or negative,
attributable to the intervention”
• Also – intended and unintended effects and long-term as well as shortterm
Original argument (Zappala and Lyons 2009):
• 3 levels of impact measurement
– Whole sector
– Individual organisations
– Individual projects
• Tools, frameworks and methodologies not easily transferred between
levels
• Framework developed in relation to housing study (Mullins 2010) to 5
levels
5 levels of impact measurement
1. Sector level
2. Organisational level
3. Team level
4. Project level
5. Individual
Transferability between levels
is dependent on:
• How the organisation
adopts and uses the tool
• Having the necessary skills
and expertise
• Relationship between
internal and external data
needs
• A corporate awareness of
what is required from
impact measurement
• The understanding and
support by staff
• Maximising the usage of the
data at more than one level
Questions and Issues
Whole sector
–
Is it possible or feasible to measure impact?
•
•
•
–
Dependent on collective approach and understanding
Expensive
Housing sector – disparate projects and circumstances
Draw on information from level 2
Organisation / Team / Project
–
Reason / motivation
•
•
•
–
Availability of skills and expertise
•
•
–
External requirement?
Internal legitimisation / accountability?
A learning tool?
Not necessarily new methodologies but alternative application
Interpretative and subjective decisions
Draw on information from levels below
External Tensions
External Drivers
•
•
•
•
Funding
Sector Promotion
Dances around regulation
Common outcomes /
community budgeting
• Wider third sector policy
influences (e.g. promotion
of SROI)
Emerging Tensions
• Data for Funders
–
–
–
–
Conflicts with internal needs
Multiple monitoring
Timescales
Mission Drift
• Sector promotion
– ‘more than a good story’
– Limited no of PIs obscure
complexities of context &
impact
Internal Tensions
Internal Drivers
• Accountability & Reporting
(variety of audiences)
– Boards & Trustees
– Management
– Making sure people enjoy the
projects
• Benchmarking
• Organisational Power Play
– Promoting visibility of CI
within corporate scorecards
– Aligning CI activities to
corporate goals
Emerging Tensions
Commercial & Social Goals
• Social Dividend from
commercial success
– Tendency toward harder
measures – leverage
• Contribute to core business
– Attribution problems – leads
to focus of CI on tenants
Corporate v Project Levels
– Who sets the goals?
– which CI outcomes get
reported?
Conclusions
• Early days but likely to grow
• High Expectations in advance of evidence!
• Tensions around types of measures and uses
made of the data (both external & internal)
• Is it worth it? (practical issues, costs
motivation/training)
• Underlying tensions reflect hybrid and large
scale of these organisations & competing
interests (e.g. Core business v CI)
Further Research
1. The role of community investment measurement
tools within the organisation
– How is community investment integrated and understood?
– What measurement tools are used, what is their strategic
fit within the organisation?
2. Data considerations (internal and external)
–
–
How is data reported and used?
What level of understanding does it provide?
3. The practical implications of measurement
–
–
Does impact measurement cause organisations to rethink
their mission, goals or values?
What are the operational issues and practical
considerations which affect measurement, such as length
of contracts
Thanks for Listening
Please Keep in Touch
David Mullins D.W.Mullins@bham.ac.uk
Vanessa Wilkes vew930@bham.ac.uk