Health Disparities and

Document Sample
Health              Disparities and Powered By Docstoc
					 Community Participatory
 Approaches to Evaluation:
Addressing Health Disparities
  in Hawaii and the Pacific

Public Health Colloquium Series
           August 28, 2008

            Angela Sy, DrPH
    Department of Public Health Sciences
           University of Hawaii
What Is Participatory Evaluation?

Evaluation that involves all the stakeholders
in a project - those directly affected by it or
by carrying it out - in every phase of
evaluating it, and in applying the results of
that evaluation to the improvement of the
       Empowerment Evaluation

The use of evaluation concepts, techniques, and
findings to foster improvement and self
determination (Fetterman et al. 1996)

An evaluation approach that aims to increase the
probability of achieving program success by
  Providing program stakeholders with tools for assessing
  the planning, implementation, and self-evaluation of
  their program
  Mainstreaming evaluation as part of the planning and
  management of the program/organization.
    Empowerment Evaluation

Perhaps what distinguishes EE most
clearly…is its acknowledgement and deep
respect for people's capacity to create
knowledge about, and solutions to, their
own experiences. (American Evaluation
Association, 2005)
Principles of Participatory Research

Equitable partnerships

Process of exchanging expert knowledge and skills of
the partners to develop and implement all stages of
the research

Researchers acknowledge that they are not experts in
the community culture while the communities hold
the expertise in identifying and addressing the issues
   Participatory Evaluation: What
            It Is and Isn’t

Not a method or set of methods

Applied approach--Goal to influence change in
community health through changes in norms,
systems, programs, policies
                 OUTCOMES OF
Empowering             Empowerment                  Health
Intervention            Outcomes:                  Outcomes:
 Strategies             (Proximal)

   CBPR                Psychological               Decreased
 Strategies            Organizational               Health
                        Community/                 Disparities

  Wallerstein, N., (2006), Evidence that Empowerment Reduces Disparities
      and Improves Health, Health Evidence Network, World Health
Evolution of Evaluation
Evolution of Evaluation
    Why Engage Stakeholders
       and Participants?
The evaluator is an educator; success is
judged by what others learn.

Those who shape policy should reach
decisions with their eyes open; It is the
evaluator's task to illuminate the situation,
not to dictate the decision. (Cronbach, 1980)
          When Would You
     Use Participatory Evaluation?
When you're already committed to a participatory process
for your project.

When you have the time, or when results are more
important than time.

When you can convince funders that it's a good idea.

When there may be issues in the community or population
that outside evaluators (or program providers) aren't
likely to be aware of.
         When Would You
    Use Participatory Evaluation?

When you need information that will be difficult
for anyone outside the community or population
to get.

When part of the goal of the project is to empower
participants and help them develop transferable

When you want to bring the community together.
          Why Would You
    Use Participatory Evaluation?

To address health disparities issues, which result
from community inequities

Community participatory approaches foster
understanding of the cultural context of

Those who have been overlooked from being of
certain ethnicities, gender, and even age, become
key and active individuals in the research process
with their perspectives providing a critical
contribution to the project
         How Do You Conduct
      a Participatory Evaluation?

Recruit stakeholders as participant evaluators.

Train evaluators.

Identify evaluation issues.

Determine the evaluation questions and outcomes
with community.
         How Do You Conduct
      a Participatory Evaluation?

Analyze the information with involvement of
community and stakeholders.

Use results to build community ownership and
celebrate what worked, and adjust the rest to
improve the project.

Stick with it indefinitely.
      Pacific Center of
Excellence in the Elimination
   Of Disparities (CEED)

 Department of Family Medicine
  and Community Health, UH
  Neal Palafox, MD, MPH--PI
               Pacific CEED
Department of Family Medicine and
Community Health (DFMCH), John A. Burns
School of Medicine collaborates with the US
Affiliated Pacific Island (USAPI) jurisdictions to
coordinate and implement 3 Pacific cancer
projects funded by CDC

In 10/1/07, DFMCH was awarded funding as
one of 18 CDC Centers of Excellence in the
Elimination of Disparities (CEED) in targeted
              Pacific CEED
Pacific CEED is developing an evaluation
framework using community-based
participatory approaches (CBPA) that involves
all USAPIs and regional stakeholders.

  Federated States of Micronesia
  Commonwealth of the Northern Mariana Islands
  Republic of Marshall Islands
  American Samoa
Pacific CEED
      Pacific CEED: Background

USAPIs spread over 869,540 sq. miles, 5 time
zones, and international date line

Significant health disparities, including cancer,
between the U.S. and the USAPI

Cancer 2nd most common cause of death
   Pacific CEED: Background

  Lack of culturally appropriate preventive
  services and severe challenges in the health

  Westernized health systems are not responsive
  to the cultural preferences and practices in
  seeking care

Goal to reduce cancer disparities in US Affiliated
Pacific Islands focusing on reduction in breast and
cervical cancer
            Pacific CEED: CBPA
 Pacific CEED evaluation
team is working with the
Comprehensive Cancer
Control (CCC) Coalitions
to develop a framework to
evaluate CEED with the
Tailored site visits to each
Multiple CCC evaluation
workshops and meetings
in Hawai’i
Pacific CEED
Pacific CEED: Lessons Learned So Far

Need to identify culturally tailored approaches
to monitoring and evaluation across USAPI

Limited access to official health surveillance
data sources require alternatives, such as local
assessments to obtain baselines and
documenting “success stories”
Pacific CEED: Lessons Learned So Far

  Trust and relationship building are key in
  every phase of the evaluation as these island
  communities have experienced colonialist
  health development and research practices
  that historically take away local knowledge.

  CBPA may help shift the relationship with an
  outside entity toward a more collaborative
Hui Malama o ke Kai
  Hui Malama o ke Kai

Shannon Hirose-Wong, PhD and Ben
Young, MD
US Department of Education
Asian Pacific Islander Youth Violence
Prevention Center, Department of
Psychiatry, UH
Hui Malama i ke Kai Foundation
         Hui Malama o ke Kai

Community generated--Grassroots effort in

Capitalizing on the strengths of community and
Native Hawaiian culture

Youth Development Program
  cultural pride
  awareness of environment: ocean, land, community
  healthy knowledge & behaviors
         Hui Mālama o ke
     Kai Program Description

5th and 6th Grade “at-risk” youth

40-50 youth attending 2 grade schools in

95% of participants are Native Hawaiian

Students recruited by counselors beginning of
         Hui Mālama o ke
     Kai Program Description
Not Babysitting, after-school care

Youth development program– Values & skills
  cultural pride
  awareness of environment: ocean, land, community
  healthy knowledge & behaviors

Capitalizes on the strengths of Native Hawaiian
culture and Waimanalo community
Hui Mālama o ke Kai Program Description

             Mind ~ Body ~ Spirit


      Hui Mālama o ke Kai
     Evaluation Framework
Program staff and school teachers refined
appropriate, feasible, useful tools and measures

Evaluators acknowledge that they are not
experts in the community culture while the
community holds the expertise in identifying
and addressing the issues

Enhanced utility and feasibility
                Hui Mālama o ke Kai
                 Evaluation Model
Native Hawaiian Cultural       Youth Health & Wellness Knowledge, Attitudes,
Values                         Behaviors

Laulima: cooperation            Positive youth development :
Na`auao: learning              School/Education, Perception of Environment,
Wiwo`ole: confidence           Family Support, Interpersonal Relationships, Conflict
Lokahi: harmony                Resolution/ Anger Management, Self-Perception
Mahalo: gratitude               Physical activity
                                Alcohol, tobacco, other drugs

                       HMK Program--
                       Culturally relevant:
                       oHealth education
                  Rubrics of Native
                  Hawaiian Values

 Lokahi                                      Na`auao
“Calm, harmony”                              “Independent,

 Wiwo`ole                                    Laulima
   “Diligent”                                “Cooperative”

                      “Gratitude, respect”
Rubrics of Native Hawaiian Values

Measure students’ progress in learning and applying
Hawaiian values taught in HMK

Developed jointly by principal investigator, evaluators,
and community representatives beginning 2004-2005
school year

Evaluators proposed criteria of assessing students’
behavior and performance based on measures used in
the educational evaluation field
Rubrics of Native Hawaiian Values

Program director and specialists adapted and
refined characteristics of the ideal HMK
student demonstrating mastery of each of the
core Hawaiian values.

5 point rating scale on each value observed by
the prevention specialists was developed to rate
student’s practice of cultural values.
   Validation of Hui Mālama o ke
    Kai Evaluation Instruments

Internal reliability—quantitative measures

Intraclass correlations—same measures over
numerous instruments

Face validity—qualitative tools

Methods triangulation—measures from all
data sources
             Triangulation: Native Hawaiian Values-
   Intraclass Correlations (ICC) of HMK Rubrics from Prevention Specialists with Other Raters
                      Measure                          Data Source           ICC          P            Df

Interpersonal Relationships                        Student survey         0.53        0.005***    47

Self-Perception                                    Student survey         0.55        0.005***    44

School/Education                                   Student survey         0.34        0.076       47

Conflict Resolution/ Anger Management              Student survey         0.69        0.000****   47

Child’s Ability to Deal with Varying Situations    Parent survey          0.54        0.005***    45

Dealing with Child’s Attitudes                     Parent survey          0.63        0.001****   42

Child’s Hawaiian Values                            Parent survey          0.47        0.019*      44

General Learner Objectives                         Schoolteachers         0.47        0.006**     27
  Rubrics of Native Hawaiian Values
 3 evaluation periods: Dec. 2006, Mar. 2007, May 2007
 Significant increase in ratings between each period,
 p=0.000 (1-5: almost never-almost always)

  Value         Definition         Q1 (N=35)    Q3 (N=28)
                                   X     SD      X     SD
Lokahi     Calm, harmonious        3.2   1.01    4.6   0.57
Wiwo`ole   Work hard               3.1   0.77    4.8   0.42
Na`auao    Work on own,            3.0   0.95    4.4   0.63
Laulima    Reliable, cooperative   3.3   0.97    4.6   0.57
Mahalo     Thankfulness, respect   3.6   0.84    4.8   0.52
                          Parent Pre & Post Surveys

          Measure effect of HMK program from parents’ perspective
          Increase in parents’ mean ratings pre-post
          Significant increase in ratings on child’s practice of Hawaiian
      Scale: 1-5: poor-very good
     Parent Survey                      Pre 10/06                               Post 6/07                  Mann-Whitney U
                           N      #      Alpha      Mean    SD     N      #      Alpha      Mean    SD      Z        p
                                Items                                   Items

Child’s ability to deal    51      16     0.948       3.8   0.74   24     14     0.948        4.3   0.59   -1.1      0.263
w/ varying situations

Child’s attitude           48      13     0.956       3.8   0.80   24     12     0.903        4.3   0.58   -1.6      0.105

Child’s Hawaiian values    50       3     0.930       3.8   1.02   24       3    0.865        4.6   0.48   -2.3     *0.022

Parenting skills           51       7      0.917      3.9   0.73   24       4    0.035        4.3   0.77   -0.4   0.723
               Triangulation: Native Hawaiian Values:
                            Data Sources
     Data Source                                 Results                                 Psychometrics-Validity


1) Student survey     Significant increase (p=0.000) understanding/practice of alpha=0.675, 3 items
                      Hawaiian values/culture, 3.5-3.8, 4 pt. scale

2) Parent survey      Significant increase (p=0.022) child’s Hawaiian values 3.8-4.6,   alpha=0.898, 2 items
                      5 pt. scale

3) HMK Rubrics of     Significant increase (p=0.000) in ratings over school year:       0.939
Hawaiian values       3.2-4.6, 5 pt. scale                                              Face validity—HMK staff


1) Retreat            Parents also wanted to further learn about Hawaiian culture Face validity—HMK staff
evaluations           and values

2) Parent interview   Child learned Hawaiian values at HMK                              Face validity—HMK staff
3) Student journal    Wrote, defined, explained Hawaiian values and how they            Face validity—HMK staff
                      have been applied
      Qualitative Results:
     Native Hawaiian Values
Student journals
  “Today, I learned the value ‘laulima.’ I used
  laulima by helping everyone. I scraped all the
  black stuff off the kalo. I will use this in my
  life by helping everyone out to get the job
          Summary and Results

Rubrics of Hawaiian values correlated with:
  Youth development outcomes on the student survey
  (p=0.000 to 0.005, ICC=0.69 to 0.53)
  All of the parent attitude outcomes on their child on
  the parent survey (p=0.001 to 0.019, ICC=0.63 to
  School teachers’ grading of students on General
  Learner Objectives (p=0.006, ICC=0.47): e.g., self
  directed learners, community contributors, complex

Triangulation also indicated reliability across
different evaluation tools
     Summary & Conclusion

Results from the qualitative and quantitative
data sources corroborated an increase of
understanding, knowledge, and practice of
Hawaiian values among the youth.

Remarkably, measures most reliable and valid
addressed the key program outcome—
understanding and practice of Hawaiian
cultural values.
Corroboration of results from the different
data sources indicate reliability of measures
and validity of HMK Rubrics to evaluate youth
understanding and practice of Hawaiian

Rubrics of Native Hawaiian Values may serve
as a tool to evaluate culturally based youth
development programs for Hawaiians

Community- driven program evaluation:
  May identify appropriate measures and outcomes
  Address reliability and validity of instrumentation
 to evaluate unique outcomes
Community Based Participatory
  Inquiry to Reduce Smoking
 Among Native Hawaiian and
    Filipina Girls in Hawaii

Maile Goo, PsyD: Co PI

National Cancer Institute Pilot Project to
Reduce Cancer Health Disparities

`Imi Hale Native Hawaiian Cancer Network

Native Hawaiian and Filipina females have
high and rising smoking rates

Females smoke at lower rates than males
in their respective ethnic groups
            Specific Aims

Goal: To investigate knowledge, attitudes,
   perceptions, and sociocultural factors
   that may be related to the high tobacco
   use among Filipino American and Native
   Hawaiian girls in Hawai`i, using CBPA, to
   develop future innovative tailored youth
   delivered smoking prevention
              Specific Aims

1)   To identify barriers and facilitators for
     Native Hawaiian and Filipina American
     girls toward smoking.

2)   To determine which barriers and
     facilitators among Native Hawaiian and
     Filipina American girls toward smoking
     are related to their respective ethnicities.
             Specific Aims

3)   To identify anti-smoking messages
     relevant to Native Hawaiian and Filipina
     American girls, and define components
     for a future novel youth-led anti-smoking
     program using a participatory research
     framework and adapting evidence-based
     strategies on youth smoking prevention
     to reduce smoking among Native
     Hawaiian and Filipina girls.
  Community Partners

UH Student Excellence in Equity and
Diversity (SEEd)
REAL, Cancer Research Center
Coalition for a Tobacco Free Hawaii
Kauai District Health Office
`Imi Hale Native Hawaiian Cancer
Asian American Network of Cancer
Awareness, Research, and Training
15-21 focus groups, 6-10 participants
  Recruitment: SEED program

Peer facilitators—REAL-conduct groups
  Comprehensive focus group training
  curriculum developed by `Imi Hale

Qualitative analysis, deductive

Summaries and results presented to
`Imi Hale and AANCART
          Future Plans
Community partners will guide Native
Hawaiian and Filipina girls to create and
implement their tailored youth led,
program(s) based on this study’s results

Build ownership among the involved youth
with opportunity to develop and
implement their own programs in their
own schools and communities
     Community Benefits

Strengthen partnerships with continued
community guided efforts to address key
health issue

Partnerships may raise awareness in the
Native Hawaiian and Filipino communities
on smoking prevalence among girls to
potentially ignite further community
  Community Approaches to Address
Health Disparities: Concluding Remarks

 Mutually agreed priority issues from all sectors
 facilitated successful mobilization around the

 Community- driven program evaluation helps to
 identify most relevant outcomes for the
 community and their appropriate measures