Docstoc

ALABAMA CTF 2007-2008 CTF Grantee Training Presentation FINAL--Handout

Document Sample
ALABAMA CTF 2007-2008 CTF Grantee Training Presentation FINAL--Handout Powered By Docstoc
					The Alabama Children’s Trust Fund Evaluation Project PY 2006-2007 Results and 2007-2008 Training

The Value of Evaluation
Evaluation is your friend: Completes the implementation of a program
–

Tests your theory; your hypotheses

Francesca Adler-Baeder, Ph.D., CFLE Jennifer Kerpelman, Ph.D., CFLE Melody M. Griffin, M. S. Craig Holloway, M. S. Alexa Calligas Catina Lynaum, M. S. Eugenia Parrett, M. S. Human Development and Family Studies Auburn University, Alabama

Quantifies program effects Tells your participants’ story Tells your story Tells CTF’s story Ultimately, becomes a tool for you to use to engage broader support

A New Direction in 2006
Utilize an involved research team Central data collection and management Mail delivered on 1 day!

A New Direction in 2006
An emphasis on change and benefit – due to the program Aggregation of information through assessment of common program objectives

Our Approach
A balance of “user-friendly” methods and systematic assessment of impact/change evidence to meet accountability requirements of funding sources Partnership with CTF and Grantees
– –

Overview: 3-Tiered Approach
1. All grantees:
– –

Use a short list of questions that assess changes in targeted knowledge, commitment, and ability. Track demographics of participants and program “dosage.” Use additional true pre/post survey to assess more behavioral measures Qualitative interviews with participants; tell the human story

2.

A few from each program type:
–

Direct involvement with CTF staff Direct involvement with Grantees

3.

A few from each program type:

1

Results to Date
42,091 Adults and 63,340 Youth participated in Community Awareness Activities. Activities 8,960 Adults and 69,827 Youth participated in program offerings (typically multi-session programs).

Demographic Characteristics Adults/Parents

Demographic Characteristics Adults/Parents

Demographic Characteristics Adults/Parents

Demographic Characteristics Adults/Parents
Adult/Parent Education

Demographic Characteristics Adults/Parents

Adult/Parent Work Status

2

Demographic Characteristics YOUTH

Demographic Characteristics YOUTH

Demographic Characteristics YOUTH

PARENT EDUCATION AND HOME VISITATION PROGRAMS
sample of participants (n = 3,058) responded to an assessment of 24 learning objectives. Analysis of paired-sample t-tests revealed statistically significant (p < .001) improvements on the average level of commitment, skill, and knowledge in ALL targeted areas.

PARENT EDUCATION AND HOME VISITATION PROGRAMS

PARENT EDUCATION AND HOME VISITATION PROGRAMS

Proportion of p parents who had “good” or “excellent” knowledge nearly

100% improvements
on both assessments

tripled

KNOWLEDGE AND USE OF  SUPPORT SERVICES

STRESS MANAGEMENT

3

PARENT EDUCATION AND HOME VISITATION PROGRAMS

PARENT EDUCATION AND HOME VISITATION PROGRAMS
From 73% and 76% to 97% ability to recognize and “stop myself”

Almost 100% improvements in all areas

PARENTING SKILLS AND CHILD DEVELOPMENT KNOWLEDGE

INDIVIDUAL RISK OF  MALTREATMENT

2nd Level of Evaluation
A selected group in each program category:
Standard methods of valid measurement of outreach (who and where) – Retrospective pre-post design In addition: – True pre-post design – Assess behavioral variables
–

PARENT EDUCATION AND HOME VISITATION PROGRAMS
A sample (n=109) completed multi-item assessments at pre and at post-program Paired sample t tests revealed statistically t-tests significant (p<.001) increases in level of:
Parental Involvement Positive Parenting Practice Parental Efficacy

and statistically significant (p< .001) decrease in level of:
Parent Distress Level

Parent Education and Home Visitation - Qualitative Interviews
Less “reactionary” parenting
–

RESPITE CARE PROGRAMS
sample of participants (n = 212) responded to an assessment of 20 learning objectives. Analysis of paired-sample t-tests revealed statistically significant (p < .001) improvements on the average level of commitment, skill, and knowledge in ALL targeted areas.

“I take more time to cool off and communicate better with my kids ” kids.”

Greater awareness of their responsibility as a role model
–

“…kids get into trouble at school by mimicking parents' bad behavior!“ “I learned things about being a parent …I didn't know how to be a parent originally.”

Learned what nurturing parenting is
–

4

RESPITE CARE PROGRAMS

RESPITE CARE PROGRAMS
Reports of stress management ability as “good” or “excellent” doubled - and proportions reporting “good” or “excellent” anger management ability increased by 50%

Proportion of p parents who had “good” or “excellent” knowledge more than doubled

KNOWLEDGE AND USE OF  SUPPORT SERVICES

STRESS  and ANGER MANAGEMENT

RESPITE CARE PROGRAMS

Qualitative Interviews – Respite Care
Benefit for parents and children:

Almost 100% improvements in all areas

–

not only helps the parents to cope, but it also helps the children with special needs as they look forward needs, to spending time with their care providers.

PARENTING SKILLS AND CHILD DEVELOPMENT KNOWLEDGE

“Respite has allowed us to accept change and understand the need for adjustments. The program helps us to navigate.” high demand of parents with special needs children seeking assistance. Many eligible parents are not able to access the services.

FATHERHOOD PROGRAMS
sample of participants (n = 370) responded to an assessment of 24 learning objectives. Analysis of paired-sample t-tests revealed statistically significant (p < .001) improvements on the average level of commitment, skill, and knowledge in ALL targeted areas.

FATHERHOOD PROGRAMS

More than 100% improvement

COOPERATION WITH  CSE

KNOWLEDGE AND USE OF  SUPPORT SERVICES

5

FATHERHOOD PROGRAMS
67% to almost 300% more fathers reported “good” or “excellent” levels of commitment

FATHERHOOD PROGRAMS

More than 100% improvements in all areas

Work, Education, and Child Support Commitment

PARENTING SKILLS AND CHILD DEVELOPMENT KNOWLEDGE

FATHERHOOD PROGRAMS
Almost 100% improvement in all areas

FATHERHOOD PROGRAMS
More than 100% improvement

INVOLVEMENT WITH  FAMILY

Co‐Parenting Relationship

FATHERHOOD PROGRAMS 2nd Tier Evaluation
Paired sample t-tests on data provide by 370 fathers revealed statistically significant (.001) increases in level of: Parental Involvement Beliefs about Father Responsibilities Parental Efficacy

FATHERHOOD - Qualitative Interviews
The program improved and/or increased their family time and relationships with spouses/partners, spouses/partners their children’s other parent, and their children.
–

“It [the program] has changed my outlook and how I look at my kids. I didn’t used to spend time with them at all, but now I spend time with them playing games like basketball and also interacting with them in other ways.”

6

FATHERHOOD - Qualitative Interviews
Promoted cooperation with CSE
–

YOUTH K- 5th GRADE
A sample of youth (n = 3,410) in grades K-5 responded to an assessment of 13 learning objectives. Analyses of paired-sample ttests revealed statistically significant (p < .001) improvements on the average level of knowledge in ALL targeted areas.

“The program gives us a place to vent. After that, the instructor begins to show us our options He options. helps us not to beat the system, but work within the system.”

“This program helps to prepare us to be selfreliant, self-confident, and self-sufficient.” Motivated to reach out to other fatherless boys in the community

YOUTH K- 5th GRADE
Percentage reporting “good” or “excellent” knowledge increased by nearly 100% on all items

YOUTH K- 5th GRADE
Percentage reporting “good” or “excellent” knowledge 42% at pre; 77% at post

Social Skills

Abuse Awareness

YOUTH K- 5th GRADE
Approximately 100% increases in percentage reporting “good” or “excellent” knowledge

YOUTH K- 5th GRADE
Nearly 100% increases in percentage reporting “good” or “excellent” knowledge

Assertiveness (in the face of  bullying)

EMOTION IDENTIFICATION AND REGULATION

7

YOUTH K- 2nd GRADE
13 item “test” option
Given pre-program and again post-program – n = 687 45% (Pre) vs. 65% (Post) answered 8 or more correctly 23% (Pre) vs. 44% (Post) answered 10 or more correctly. 44% improvement on scores of 8 or higher and 91% improvement on scores of 10 or higher.
–

YOUTH GRADES 6-12
A sample of youth (n = 7,018) in grades 6-12 responded to an assessment of 24 learning objectives. Analyses of paired-sample t-tests revealed statistically significant (p < .001) improvements on the average level of knowledge, ability, or commitment in ALL targeted areas.

YOUTH GRADES 6-12
Percentage reporting “good” or “excellent” knowledge increased by an average of 40% on these items

YOUTH GRADES 6-12
Percentage reporting “good” or “excellent” knowledge and ability in
– – –

COOPERATIVE BEHAVIOR EMOTION KNOWLEDGE SELF-CONFIDENCE

increased by an average of 40%

Social Competence

YOUTH GRADES 6-12

YOUTH GRADES 6-12
Started out at a higher level However, changes in commitment levels were statistically significant and meaningful
–

- Started out at a higher level - Mean levels changes were statistically significant - Meaningful changes in important areas

–

–

ABUSE AWARENESS AND RESOURCEFULNESS

My commitment to staying in or returning to school (81% - up to 91% post) My commitment to stay free of drug abuse (81% up to 92% post) My commitment to stay tobacco free (81% - up to 89% post)

8

YOUTH GRADES 6-12 2ND Tier Evaluation
n-= 181 Analyses of multi-item measures utilizing paired sample t-tests revealed statistically significant increases (p< .001) in level of:
– – – – –

YOUTH GRADES 6-12 2ND Tier Evaluation
Examination of details:
–

–

Self-Esteem Commitment to Education Accountability for Actions Conflict Management Skills Assertiveness Skills

–

25% decrease from pre-program to post- program in youth reporting getting into fights “a lot.” lot ” 13% decrease from pre- to post-program was documented for youth reporting receiving suspension or detention “a few times” or “a lot.” a 25% increase in youth from pre-program to postprogram whose average score for accountability for actions was 4 or higher (scale of 1-5).

YOUTH GRADES 6-12 QUALITATIVE INTERVIEWS
learned a lot about appropriate anger management techniques, techniques effective conflict resolution skills, and felt that they had an increase in selfesteem.
–

YOUTH GRADES 6-12 QUALITATIVE INTERVIEWS
Program improved to their current school performance
–

“[The “[Th program] helped me complete my h ]h l d l t homework, k perform better on tests, and improve my grades.” Many, who before participation in the program had not thought about going to college, now express a greater interest in attending college and pursing dreams they might not otherwise have thought possible.

Opened their eyes to new possibilities
–

“I learned how not to argue….how to ‘tune someone out’ when getting into a conflict with them…. I can express myself better.”

COMMUNITY AWARENESS
A sample of participants (n = 476) completed surveys to assess 6 learning objectives. Analyses of paired-sample t-tests revealed statistically significant (p < .001) improvements on the average level of knowledge, ability, or commitment in ALL targeted areas.

COMMUNITY AWARENESS
My ability to recognize abuse and/or neglect. Percentage responding “good” or “excellent” nearly doubled.

9

COMMUNITY AWARENESS
My knowledge of intervention strategies to use in the event of ab se and/or neglect abuse neglect. Percentage responding “good” or “excellent” doubled.

COMMUNITY AWARENESS
My knowledge of ways to prevent child abuse and/or neglect. neglect Percentage responding “good” or “excellent” nearly doubled.

Still to come….
Final report with all information and analyses completed Reports by district Individual reports

VIDEO

Programmatic Reporting

Evaluation Procedural Guide
Make copies of every sheet for your records before mailing to AU. Expect an email from AU confirming each report received.

10

Revised Target Data/Objectives
Re-submit only if (1) stipulated to do so, or (2) need to revise for any reason. Due Monday, October 15, 2007 Includes:
– – –

Data Report
Due Nov. 15, Feb. 15, May 15, and July 31 Includes the following:
– – – – – –

Cover Sheet (noting package contents) Target Data Form Target Program Objectives Checklist(s)

Cover Sheet (noting package contents) Electronic Master List(s) Electronic Master Output Report Individual Demographic Forms Individual Retrospective Pre-Post Surveys Individual Pre- or Post-Program Surveys

Electronic Master Lists
Continuously add information to the lists on all participants served since Aug. 1, 2007. Three types of lists:
– – –

Electronic Master Lists (cont.)
For grantees funded PY 2006-2007:
– –

Continue numbering of ID #’s from July, 2007. Keep same ID #’s for participants continuing from #s last year.

General Master List School-Based Classroom Master List School-Based Large Presentation Master List

Use updated lists for PY 2007-2008. General Master List – assign ID #’s

Do NOT reuse ID #’s – must be unique to individual. Once someone is added to the Master List during PY 2007-2008, do NOT remove the person.

Electronic Master Output Report
Provides information on individuals served through community awareness activities. Continuously add information to the lists on activities and participants served since Aug. 1, 2007.

Demographic Form
Electronically add contract # before printing copies. Use updated form for PY 2007 2008 2007-2008. Have participants complete ONCE— at first session. Use appropriate form for participants:
– –

Adult/Parent (use for teen parents) Child/Youth

11

Retrospective Pre-Post Survey
Electronically add contract # before printing copies. Use updated survey for PY 2007-2008. 1 page survey for your program type (or additional component). Have participants complete survey ONCE— at last session—during PY 2007-2008. Ask participants to complete ALL items. Explain to them how to complete the survey.

Retrospective Pre-Post Instructions
From Page 2 of retrospective packet: This survey asks about things related to the program y you have just finished. For each q j question, p , provide two responses. The left side of the page is where you will place a checkmark to tell us about these things before you participated in this program. The right side of the page is different. Place a checkmark on that side to let us know about these things now that you have participated in the program.

Approved Modifications to Surveys
List of Approved Additional Retrospective Questions provides optional items to ADD. Who can use ONLY targeted items?
– –

Child-Focused Programs: Pre- or Post-Program Surveys
For participants in Pre-K – 2nd Grade. Electronically add contract # before printing copies. Use updated survey for PY 2007-2008. See updated instructions for PY 2007-2008:
–

Programs surveying 5th Grade and below. Programs granted permission by CTF and CTF Evaluation Team due to special circumstances.

– –

May edit to include items consistent with 7 target program objectives. Mark whether Pre- or Post-Test. Make sure ID #’s used are unique to an individual.

IMPORTANT GENERAL INFORMATION
• •

Unique Methods for COMMUNITY AWARENESS PROGRAMS Master Output Report provides information on activities and participants served. No master list is needed. Demographic forms and retrospective surveys used when appropriate for activity.

•

Mark in heading on all forms/surveys what quarter you submitted to AU. Do NOT staple demographic form to retrospective survey, or do double-sided copies to join different forms/surveys. Spanish versions of forms/surveys will be made available this Fall at: http://www.ctf.alabama.gov/Forms.htm.

12

Site Visits
The Evaluation Team is not the Big Bad Bully! Our purpose is NOT to investigate!! We are here to provide technical assistance with the evaluation process.

Site Visits (cont.)
The Evaluation Team is available to during Monday – Friday to provide assistance as needed. Please feel free to contact any Team member at

334-844-3299.

13


				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:45
posted:1/12/2008
language:English
pages:13