Using a Program Theory Model to Clarify the Collaboration, Guide the Program, and Direct an Outcomes-based Evaluation
Kathryn Race Race & Associates, Ltd.
Presentation at the Annual Conference of the American Evaluation Association, Baltimore MD November 2007
4/14/2009
Overview
Park Voyagers is a collaborative effort between the Chicago Park District and Museums In the Park.
4/14/2009
Collaborative Partners
Museums In the Park (a consortium of 10 Museums in Chicago) The Chicago Park District Local Participating Parks
4/14/2009
Program Overview
A 3-year, hybrid after-school and family outreach program organized into four phases.
4/14/2009
Program Phase 1
10
week after school program for children 8 to 10 years of age. Program sessions are tied to a field trip that occurs once a month.
4/14/2009
Program Phase 2
Family workshops are offered once a month for 3 mos.
Workshop tied to a family museum visit that occurs once a month.
4/14/2009
Program Phase 3 and 4
Phase 3 – Family museum visits once a month for a year. Phase 4 – 1-year family passes.
4/14/2009
Participating Local Parks
Number of participating parks varied from: 9 new parks each year (27 total) 3 new parks each year (9 total) 5 new parks each year (15 total) (went from 9 to 10 participating museums)
4/14/2009
Criteria for Participating Local Parks
Facilities and staff to support program Able to recruit children/families Committed to program Selected to balance the racial and ethnic mix of Chicago family residents
4/14/2009
Approach to Evaluation
Aligned with IMLS’ outcomes-based evaluation directive Sought to build evaluation capacity into the consortium Not a culture of assessment
4/14/2009
Approach to Evaluation (con’t.)
Prospective approach – archived program records revised during evaluation Continuing management tool rather than a discrete study (Scheirer, 2005) Evaluation data collected as an integral part of program operations
4/14/2009
Program Model
Helped to: Clarify the collaborative Explain and describe the program (Used to revise Phase 1 curriculum) Used to guide and direct the evaluation
4/14/2009
Scope of Evaluation
Process outcomes were integrated into the outcomes assessment Level of participation Focus on outcomes assessment
4/14/2009
Scope of Evaluation (con’t.)
Identified target behavior(s) used to infer future behavior
4/14/2009
Lessons Learned
CMT
approach to evaluation was successful. Required commitment from Executive and Program directors. Program educators were integrally involved in the collection of data.
4/14/2009
Lessons Learned (con’t.)
Not sure if evaluation efforts will be sustained. Processes now in place that will facilitate the collection of improved program data.
4/14/2009
Lessons Learned (con’t.)
Process outcome assessment suggested
implementation differences (as compared to designed program) especially for Phase 1. But there was good alignment of designed versus implemented program for Phase 2. Target behaviors were evident during program phases – inferred for future.
4/14/2009