Guiding Principles From Theory to Reality by variablepitch336


									Evidence and Feasibility

A Systematic Approach to Balance
David Andrews Cynthia Buettner

Choose your favorite.

Divergent Perspectives on Defining “what works”
No Child Left Behind “If we can reach only one child”

No Child Left Behind: Key Characteristics of Reliable Research
Scientific Method
A hypothesis is formulated; a treatment group and control group are used in a study to try to disprove the hypothesis.

Several studies find the same result.

Study findings can be applied broadly to other populations.

Meets Rigorous Review Standards
The study's design, measures, and interpretation of results meet rigorous standards of peer review.

Convergent findings
Results found using variations of the approach all point to the same conclusion.

“If we can reach only one child.”
"Scientists tell you that bumblebees can't fly, but we know better," declared D.A.R.E. Executive Director Glenn Levant upon release of the governmentsponsored report that D.A.R.E. doesn't work (USA Today, October 11, 1994). D.A.R.E. officers claim that the anecdotal evidence is convincing that D.A.R.E. is working extremely well, citing the warm reception they have received by schools and parents. "Besides, even if we are reaching only one kid, it's worth all the effort.”

Best Practice, Promising, Model: Evidence by Any Other Name?
Everyone has an evaluation system

National Agencies
Mental Health Juvenile Justice Substance Abuse Education


Local Systems

Prototype Evidence-based Lists
Youth Violence: Surgeon General's Report (composite) Blueprints Violence Prevention (OJJDP) Positive Youth Development (NICHHD) Preventing Crime: What Works (NIJ) Safe and Drug-Free Schools (DOE) Prevention Research Synthesis (CDC) National Registry of Effective Prevention Programs (CSAP)

Top 10 (almost) Programs/Approaches
Life Skills Training SOAR Project Star Treatment Foster Care PATHS Nurse-Family Partnership Multisystemic Therapy Bullying Prevention Project Northland Botvin Hawkins Pentz Chamberlain Greenburg Olds Henngeler Olweus Perry (6) (4) (4) (4) (3) (3) (3) (3) (3)

Community Implementation and Effectiveness Barriers
Program doesn’t exist Fidelity Expense Qualifications of staff Statistical versus practical significance Misrepresentation of target impact

Community Responses to Implementation Barriers
Discount the importance of evidence Adopt fall back stance “if we can only reach one child” Build local approaches (hopefully with evaluation) Delay implementation of any new programs

What is the best way to assist communities
Develop more evidence-based programs in collaboration with communities to insure feasibility (Hoagwood). Assist communities in developing evaluation skills Provide tools and develop skills to create a more balanced approach emphasizing evidence and feasibility

Choose your favorite

1910 Kadinsky
Russian artist Wassily Kandinsky’s famous work of 1910. Generally recognized as one of the first abstract paintings.

1940 Unknown Artist
Artist created this beautiful Japanese garden using paint, a brush, and newly created tool – Paint-by-Number

2002 - Camryn

Three year old artist used mostly her fingers to create this piece - she was unable (or unwilling) to label or describe her work

Evidence Checklist
Implementable. Based on Effective Principles. Customer Satisfaction. Change Reports. Comparison Group. Random Assignment to Control Group. Longitudinal Impact. Multiple Site Replication. Dosage Analysis. Meta-analysis, Expert review, Consensus.

Evidence Scoring
O-2 checks = Unproven Approach – Such an approach has either 1) no documentation that it has ever been used (regardless of the principles it is based upon), or 2) has been implemented successfully with no evaluation.

Evidence Scoring
3-5 checks = Promising Approach – Programs with this score have been implemented and significant impact evaluations have been conducted. While the data supporting the programs is promising, its scientific rigor is insufficient to suggest causality. Multiple, undefined factors may be contributing to the success of participants.

Evidence Scoring
6-10 points = Evidence-Based Approaches – Programs scoring in this range have compelling evidence of effectiveness. Those with the highest scores (8-10) can attribute participant success to the program itself, and have evidence that the approach will work for others in different environments.

Feasibility Checklist - Available
Detailed descriptions of implementation procedures are available and understandable. Training is available when described as a necessary component of the program. Curriculum materials are available when necessary for implementation. Any other support materials described as necessary for implementation are available.

Feasibility Checklist - Affordable
The total costs of program materials are affordable given our organization's budget. The total costs of training are affordable given our organization's budget. The training time commitment of new or existing personnel is affordable given our organization's budget. The implementation time commitment of new or existing personnel is affordable given our organization's budget. The time commitment of participants is feasible given our capacity. The time commitment for administering the program is feasible.

Feasibility Checklist - Feasible
The underlying principles of the program being evaluated are consistent with our organization’s approach to meeting the needs of high risk youth. The approach used in the program being evaluated is consistent with existing policies and procedures currently in place within the organization. The implementation of this program will not create insurmountable internal political challenges. The implementation of this program is consistent with the current priorities of the organization.

Feasibility and Evidence Matrix
Evidence Feasible
Untested and Available Promising and Available Evidenced Based and Available

Untested and Affordable

Promising and Affordable

Evidenced Based and Affordable

Untested and Feasible

Promising and Feasible

Evidence Based and Feasible

A dearth of feasible science-based programs prohibits a universal mandate for such programs. Home grown prevention/intervention needs to be evaluated. Communities should adopt a portfolio management approach to investing in prevention/intervention

To top