Measuring the impact of community investment activities in housing by malj


									               Tensions of Measurement:
             community investment activities
                 in housing associations
               David Mullins and Vanessa Wilkes, TSRC
               Housing Studies Association Conference
                           15th April 2011
                                Funded by:
Hosted by:
• Sector Context
• Evidence Base
• Levels of Measurement
  – 5 levels of measurement
  – Questions and Issues
• Tensions
  – External Drivers
  – Internal Drivers
• Conclusions
• Further Research
             Sector Context
• Well established industry, large enterprises
  (50% market share in social housing)
• Entrenched hybridity (social & commercial)
• Core business highly regulated & measured
• Non-core ‘community investment’ (CI)
  activities symbol of independence
• Legitimacy & control issues brought to fore –
  2008 Housing & Regeneration Bill & NHF Audit
• New tools emerging to measure social impact
  of CI activities (in-house & off the shelf)
                     Evidence Base
• Scoping study – emerging approaches to
  measurement of social performance (2010)
  –  literature review & checklist (17 tools to set priorities, measure
    outcomes or link these to corporate goals)
  – 7 Case studies (organisation/tool based: 3 in-house, 4 outsourced)
  – 2 workshops (now 3, plus follow up tracking e.g. of first sector specific

• NHF Neighbourhood Audits 2008, 2011
  – Sector overview of CI (64% coverage by stock)
  – Inputs/Outputs (£435mil CI in 6,800 projects 2008; leverage 1.6:2.7)

• Linked PhD
  – Impacts & outcomes, depth case studies
  – More on this later!
 The levels of impact measurement
                   OECD definition of impact measurement
                “the impact of all changes, positive or negative,
                       attributable to the intervention”

• Also – intended and unintended effects and long-term as well as short-

Original argument (Zappala and Lyons 2009):
• 3 levels of impact measurement
    – Whole sector
    – Individual organisations
    – Individual projects
• Tools, frameworks and methodologies not easily transferred between
• Framework developed in relation to housing study (Mullins 2010) to 5
   5 levels of impact measurement
                          Transferability between levels
1. Sector level              is dependent on:
                          • How the organisation
2. Organisational level      adopts and uses the tool
                          • Having the necessary skills
3. Team level                and expertise
                          • Relationship between
4. Project level             internal and external data
5. Individual             • A corporate awareness of
                             what is required from
                             impact measurement
                          • The understanding and
                             support by staff
                          • Maximising the usage of the
                             data at more than one level
                Questions and Issues
Whole sector
  –       Is it possible or feasible to measure impact?
      •      Dependent on collective approach and understanding
      •      Expensive
      •      Housing sector – disparate projects and circumstances
  –       Draw on information from level 2
Organisation / Team / Project
  –       Reason / motivation
      •      External requirement?
      •      Internal legitimisation / accountability?
      •      A learning tool?
  –       Availability of skills and expertise
      •      Not necessarily new methodologies but alternative application
      •      Interpretative and subjective decisions
  –       Draw on information from levels below
                   External Tensions
External Drivers               Emerging Tensions
                               • Data for Funders
• Funding                         –   Conflicts with internal needs
                                  –   Multiple monitoring
• Sector Promotion
                                  –   Timescales
• Dances around regulation
                                  –   Mission Drift
• Common outcomes /
  community budgeting          • Sector promotion
• Wider third sector policy       – ‘more than a good story’
  influences (e.g. promotion      – Limited no of PIs obscure
  of SROI)                          complexities of context &
                  Internal Tensions
Internal Drivers                    Emerging Tensions
• Accountability & Reporting        Commercial & Social Goals
   (variety of audiences)           • Social Dividend from
   – Boards & Trustees                commercial success
   – Management                        – Tendency toward harder
   – Making sure people enjoy the        measures – leverage
                                    • Contribute to core business
• Benchmarking                         – Attribution problems – leads
• Organisational Power Play              to focus of CI on tenants
   – Promoting visibility of CI     Corporate v Project Levels
     within corporate scorecards       – Who sets the goals?
   – Aligning CI activities to         – which CI outcomes get
     corporate goals                     reported?
• Early days but likely to grow
• High Expectations in advance of evidence!
• Tensions around types of measures and uses
  made of the data (both external & internal)
• Is it worth it? (practical issues, costs
• Underlying tensions reflect hybrid and large
  scale of these organisations & competing
  interests (e.g. Core business v CI)
                Further Research
1. The role of community investment measurement
   tools within the organisation
   – How is community investment integrated and understood?
   – What measurement tools are used, what is their strategic
     fit within the organisation?
2. Data considerations (internal and external)
   –   How is data reported and used?
   –   What level of understanding does it provide?
3. The practical implications of measurement
   –   Does impact measurement cause organisations to rethink
       their mission, goals or values?
   –   What are the operational issues and practical
       considerations which affect measurement, such as length
       of contracts
          Thanks for Listening
Please Keep in Touch

David Mullins
Vanessa Wilkes

To top