Strategies for measuring the outcomes of the Communities for

Strategies for measuring the outcomes of the Communities for Children program Ben Edwards National evaluation consortium   Social Policy Research Centre (SPRC), UNSW: Ilan Katz (Lead Investigator), Kristy Muir, Christiane Purcal, Roger Patulny, Saul Flaxman, David Abello, Natasha Cortis, Cathy Thomson, Ioana Oprea and Sharni Chan, Thomas Longden, Catherine Spooner and King Tam Australian Institute of Family Studies: Ben Edwards, Sarah Wise, Matthew Gray, Alan Hayes, Sebastian Mission The Communities for Children initiative  NGOs were funded as Facilitating Partners (FPs) in communities to develop and implement a wholeof-community approach to early childhood development, in consultation with local stakeholders. FPs established a CfC Committee (CCC) with broad representation from stakeholders in their communities. FPs oversaw development of 4-year plans with the CCC and managed the overall funding allocation. Most of the funding was allocated to other local service providers known as Community Partners (CPs) to deliver the programs identified in the plans. Examples of programs included home visiting; early learning and literacy; parenting and family support; child nutrition; and community events. Service effectiveness was dependent not only on the nature and number of services, but also on coordinated service delivery. CfC logic = provide new services + increased service coordination and cooperation = improved outcomes for children and families.      Theory of change Improved services Better outcomes for children & parents FP model (leads to) Contributes to Stronger communities (more social capital) Distinguishing between process and outcomes  Outcomes included: children and parent’s mental and physical health, parenting, parent relationship conflict, joblessness, children’s receptive vocabulary, home learning, social capital and neighbourhood quality. (Stronger Families in Australia study) Process such as number, focus, reach and coordination of services (Service Mapping and Service Coordination studies)  The Stronger Families in Australia Study Three-wave longitudinal study of 2,202 families living in 10 CfC sites and five sites that did not but were in other ways comparable (contrast sites). Some strengths of the study include: 1. An appropriately matched comparison group 2. Sample represented 42% of the population of 2 year-old children in CfC and contrast sites 3. Low attrition (that was not systematic) at longitudinal follow-up 4. Robust measurement of outcomes (using LSAC measures) Establishing the counterfactual, the role of matched comparison sites The counterfactual, is what would happen in the absence of the intervention  Validity depends on:    Selection of comparator and, Whether there were pre-existing differences prior to the intervention. The validity of contrast sites in SFIA  Site selection:   Same states and territories as CfC and similar in location, size and SEIFA Selected from the pool of possible communities considered for CfC funding or from comparable communities where the Australian Early Development Index was administered.   Testing for systematic differences in outcomes at wave 1 after adjusting for demographic characteristics of families No evidence of systematic attrition Secondary data: Outcome indicator framework Secondary data to measure community outcomes and compare them to other similar areas. Promise     Cheaper Measure changes over longer periods. Relieve the burden on service providers to collect data Data from the whole population Different spatial units don’t easily map onto CfC sites Different definitions of measures across states Timescales problematic, at least 2 years in arrears Sensitivity about the release of small area level data Problems     Only baseline could be collected Roger Patulny, David Abello. Framework Report. (forthcoming). Appendix 4: Outcome Indicators Lessons for the evaluation of NGO’s Theory of change  Well defined population  Good measures (LSAC may be useful)  Longitudinal data necessary  Have a comparison group  Secondary data on communities problematic  References Edwards, B., Wise, S., Gray, M., Hayes, A., Katz, I., Misson, S., Patulny, R. and Muir, K. (forthcoming), The Stronger Families in Australia Study: the Impact of Communities for Children, Department of Families, Housing, Community Services and Indigenous Affairs. Muir, K., Katz, I., Purcal, C., Patulny, R., Flaxman, S., Abello, D., Cortis, N., Thomson, C., Oprea, I., Wise, S., Edwards, B., Gray, M. and Hayes, A. (forthcoming). National evaluation (2004–2008) of the Stronger Families and Communities Strategy 2004–2009, Department of Families, Housing, Community Services and Indigenous Affairs.