Working With Performance Measures
Lawrence Fiedler, OJJDP Ryan Clinton, Juvenile Justice Specialist from Texas Katherine Guidry, Juvenile Justice Specialist from Louisiana Susan Jenkins, CSR, Inc. Sarah Wygant, CSR, Inc. Marcia Cohen, DSG, Inc.
Session Overview
• State-Level Data Reporting Strategies
• Uses for OJJDP’s Performance Data • Tips for Long-Term Data Collection
2
Overview of OJJDP Data Requirements
• Mandatory Only
• Year-Round Data Entry • Marking Funds as ―Not Expended‖
• Updated Data Reports
3
OJJDP Data Reporting Deadlines
• Formula Grants – Reporting period 10/1/2006–9/30/2007; due: 12/31/2007 • Title V – Reporting period 10/1/2006–9/30/2007; due: 11/30/2007 • JABG – Reporting period 4/1/2007–3/31/2008; due: 6/30/2008 • EUDL Block Grants – Reporting period 7/1/2007–12/31/2007; due: 1/31/2008
4
Methods to Collecting Data
• States either
– Collect and enter data into DCTAT – Have subgrantees enter data directly to DCTAT – Contract services to another party
5
State-Level Data Reporting Strategies:
Contracting Out Example From Texas
Ryan Clinton, Juvenile Justice Specialist from Texas
6
Paper System
7
Routine Program Reports
8
Data Entry
9
Objections to Change
Analysis
10
Evaluation Online System
Grant notification
Data downloaded Reports entered
11
Grant Reporting
12
Inactive Grant Records
13
Grant Notes
14
15
Report Status
16
Report Validation
17
18
Administrative Access
19
System Reports
20
21
Objectives Comparison Report
22
Management Functions
23
Benefits of Online Evaluation
• Decreased missing data • Decreased reporting errors
• Decreased reporting delinquency
• Decreased management costs • Decreased general costs
24
State-Level Data Reporting Strategies:
State Collecting Data Example From Louisiana
Katherine Guidry, Juvenile Justice Specialist from Louisiana
25
Louisiana
• Subgrantees provided with
– Performance indicators and measurements table for their program area – DSG’s Web site – Assistance
26
Louisiana’s Reporting Method
• Subgrantees are required to
– List outputs and outcomes in application – Report data in quarterly progress reports – Have JJ Specialist enter all data into DCTAT
27
Louisiana’s QPR
•
# 9 11 12
Output Measurements
Measurement Number of program youth served Number of service hours completed Average length of stay in program Results During This Quarter
A. B. C.
28
•
Outcome Measurements
Measurement # & % of program youth exhibiting change in school attendance
Short-Term Results (During the Qtr) A. B. C. A. B. C. A. B. C. Long-Term Results (6–12 Mos.) A. B. C. A. B. C. A. B. C.
1
2
# & % of program youth completing program requirements
3
# & % of program families satisfied with
29
Uses for OJJDP’s Performance Data
Susan Jenkins, CSR, Inc. Sarah Wygant, CSR, Inc.
30
Overview
• • • • Grantee demographics Data report examples New DCTAT data reports Ways to use the DCTAT data
31
Grantee Demographics
Response Rates
• • • • Title V 49 (88 percent) Formula Grants 48 of 56 (86 percent) JABG 50 of 56 (89 percent) EUDL Block Grant 33 of 51 (65 percent)
32
Grantee Demographics (cont’d)
Most Common Program and Purpose Areas • Title V and Formula Grants—Delinquency Prevention (PA 9) • JABG accountability—Programming by Law Enforcement (PA 11), Accountability Programming for Courts and Probation Staff (PA 15) • EUDL Block Grants—Enforcement
33
Grantee Demographics
Source of Funding:
• Title V—fiscal years 2004 and 2005 • Formula—FYs 2004 and 2005
• JABG—FY 2003
34
Grantee Demographics (cont’d)
Use of Evidence-Based Models:
• Title V—42 percent of projects • Formula—19 percent of projects
35
Grantee Demographics (cont’d)
Implementation by Faith-Based Organizations:
• Title V—2 percent of projects • Formula—1 percent of projects
36
Data Report Examples
2006 PART of Juvenile Justice Programs
http://www.whitehouse.gov/omb/expectmore/summary/10003813.2006.html
37
Data Report Examples (cont’d)
2006 PART of Juvenile Justice Programs
http://www.whitehouse.gov/omb/expectmore/summary/10003813.2006.html
38
Data Report Examples (cont’d)
Explanatory Text: • Respondent demographics • Subgrant distribution
39
Data Report Examples (cont’d)
Summary of the data for mandatory measures
Performance Indicator Percent of subgrants using an evidence-based model Number of program youth served Percent of program youth completing program requirements Reduced antisocial behavior Improved employment Improved family relationships Improved GED attainment Improved GPA Reduced suspension from school Reduced pregnancies Improved school attendance Improved social competence Improved body Image Reduced bullying Improved conflict resolution skills Improved coping skills Reduced gang activities Improved job skills Improved family functioning Improved self esteem Improved perception of social support Reduced substance use/abuse Percent of program youth who offended (short term): Percent of program youth who offended (long term): Data 19% 1,826,543 80% 66% 65% 77% 0% 15% 13% 40% 39% 39% 45% NR NR 0% NR 85% 47% 78% 45% 37% 9% 8%
40
Data Report Examples (cont’d)
Reports to Congress • Title V 2004/2005 • JABG 2004, 2005, 2006
41
New DCTAT Data Reports
• Performance Data Summary Report
• Performance Data Report • Subgrant Detail Data Report
42
Ways to Use DCTAT Data
• Reports to SAG or Governor’s Office. • Grant applications. • Compare subgrantee performance.
How do you use performance data?
43
Contact Information
• DCTAT Helpdesk:
ojjdp-dctat@csrincorporated.com OR 1.866.487.0512
44
Tips for Obtaining Long-Term Outcome Data
Marcia Cohen, DSG, Inc.
45
Overall OJJDP Logic Model
46
Monitoring Subgrantees Is Key to Obtaining Long-Term Outcome Data
Ensure that subgrantees submit a data collection plan that states:
• • • • • •
1. What are the sources of data? 2. What are the data collection methods? 3. Who will collect the data? 4. How often will the data be collected? 5. Who will analyze the data? 6. Who will report the data?
47
Data Sources
• Is that data accessible?
• Has someone been designated to obtain the data?
• Can it be accessed in a timely manner? • Is it feasible and cost effective to collect the data?
48
Data Collection Methods
• What are the strategies and instruments for data collection? • How will the subgrantee obtain the data? • Have arrangements been made to obtain the data?
• What procedures will be used to obtain the data?
49
Site Visits
Site visits are necessary to examine whether the subgrantee is implementing its project as planned. • (a) Key types and levels of monitoring • (b) Links between implementation monitoring and results monitoring • (c) Key principles in building a monitoring system • (d) The needs of every monitoring system • (e) Analyzing performance data • (g) Pretesting data collection instruments and procedures
50
Monitoring
Two Types of Monitoring: • Implementation monitoring
– Tracks inputs, activities, and outputs
• Results monitoring
– Policy monitoring – Project monitoring
51
Results-Based Monitoring
Needs of Every Results-Based Monitoring System: • Ownership • Management • Maintenance • Credibility
52
Sustaining Outcomes
• Accountability • Capacity • Incentives
53