"Health Disparities and"
Community Participatory Approaches to Evaluation: Addressing Health Disparities in Hawaii and the Pacific Public Health Colloquium Series August 28, 2008 Angela Sy, DrPH Department of Public Health Sciences University of Hawaii What Is Participatory Evaluation? Evaluation that involves all the stakeholders in a project - those directly affected by it or by carrying it out - in every phase of evaluating it, and in applying the results of that evaluation to the improvement of the work. Empowerment Evaluation The use of evaluation concepts, techniques, and findings to foster improvement and self determination (Fetterman et al. 1996) An evaluation approach that aims to increase the probability of achieving program success by Providing program stakeholders with tools for assessing the planning, implementation, and self-evaluation of their program Mainstreaming evaluation as part of the planning and management of the program/organization. Empowerment Evaluation Perhaps what distinguishes EE most clearly…is its acknowledgement and deep respect for people's capacity to create knowledge about, and solutions to, their own experiences. (American Evaluation Association, 2005) Principles of Participatory Research Equitable partnerships Process of exchanging expert knowledge and skills of the partners to develop and implement all stages of the research Researchers acknowledge that they are not experts in the community culture while the communities hold the expertise in identifying and addressing the issues Participatory Evaluation: What It Is and Isn’t Not a method or set of methods Applied approach--Goal to influence change in community health through changes in norms, systems, programs, policies OUTCOMES OF PARTICIPATION Empowering Empowerment Health Intervention Outcomes: Outcomes: Strategies (Proximal) CBPR Psychological Decreased Strategies Organizational Health Community/ Disparities Political Wallerstein, N., (2006), Evidence that Empowerment Reduces Disparities and Improves Health, Health Evidence Network, World Health Organization. http://www.euro.who.int/HEN/Syntheses/empowerment/20060119_10 Evolution of Evaluation Evolution of Evaluation Why Engage Stakeholders and Participants? The evaluator is an educator; success is judged by what others learn. Those who shape policy should reach decisions with their eyes open; It is the evaluator's task to illuminate the situation, not to dictate the decision. (Cronbach, 1980) When Would You Use Participatory Evaluation? When you're already committed to a participatory process for your project. When you have the time, or when results are more important than time. When you can convince funders that it's a good idea. When there may be issues in the community or population that outside evaluators (or program providers) aren't likely to be aware of. When Would You Use Participatory Evaluation? When you need information that will be difficult for anyone outside the community or population to get. When part of the goal of the project is to empower participants and help them develop transferable skills. When you want to bring the community together. Why Would You Use Participatory Evaluation? To address health disparities issues, which result from community inequities Community participatory approaches foster understanding of the cultural context of communities Those who have been overlooked from being of certain ethnicities, gender, and even age, become key and active individuals in the research process with their perspectives providing a critical contribution to the project How Do You Conduct a Participatory Evaluation? Recruit stakeholders as participant evaluators. Train evaluators. Identify evaluation issues. Determine the evaluation questions and outcomes with community. How Do You Conduct a Participatory Evaluation? Analyze the information with involvement of community and stakeholders. Use results to build community ownership and celebrate what worked, and adjust the rest to improve the project. Stick with it indefinitely. Pacific Center of Excellence in the Elimination Of Disparities (CEED) Department of Family Medicine and Community Health, UH Neal Palafox, MD, MPH--PI Pacific CEED Department of Family Medicine and Community Health (DFMCH), John A. Burns School of Medicine collaborates with the US Affiliated Pacific Island (USAPI) jurisdictions to coordinate and implement 3 Pacific cancer projects funded by CDC In 10/1/07, DFMCH was awarded funding as one of 18 CDC Centers of Excellence in the Elimination of Disparities (CEED) in targeted populations. Pacific CEED Pacific CEED is developing an evaluation framework using community-based participatory approaches (CBPA) that involves all USAPIs and regional stakeholders. USAPI: Federated States of Micronesia Commonwealth of the Northern Mariana Islands Republic of Marshall Islands Palau Guam American Samoa Pacific CEED Pacific CEED: Background USAPIs spread over 869,540 sq. miles, 5 time zones, and international date line Significant health disparities, including cancer, between the U.S. and the USAPI Cancer 2nd most common cause of death Pacific CEED: Background Lack of culturally appropriate preventive services and severe challenges in the health infrastructure Westernized health systems are not responsive to the cultural preferences and practices in seeking care Goal to reduce cancer disparities in US Affiliated Pacific Islands focusing on reduction in breast and cervical cancer Pacific CEED: CBPA Pacific CEED evaluation team is working with the USAPI and Comprehensive Cancer Control (CCC) Coalitions to develop a framework to evaluate CEED with the CCC Tailored site visits to each USAPI Multiple CCC evaluation workshops and meetings in Hawai’i Pacific CEED Pacific CEED: Lessons Learned So Far Need to identify culturally tailored approaches to monitoring and evaluation across USAPI Limited access to official health surveillance data sources require alternatives, such as local assessments to obtain baselines and documenting “success stories” Pacific CEED: Lessons Learned So Far Trust and relationship building are key in every phase of the evaluation as these island communities have experienced colonialist health development and research practices that historically take away local knowledge. CBPA may help shift the relationship with an outside entity toward a more collaborative approach. Hui Malama o ke Kai Hui Malama o ke Kai Acknowledgements Shannon Hirose-Wong, PhD and Ben Young, MD US Department of Education Asian Pacific Islander Youth Violence Prevention Center, Department of Psychiatry, UH Hui Malama i ke Kai Foundation Hui Malama o ke Kai Community generated--Grassroots effort in Waimanalo Capitalizing on the strengths of community and Native Hawaiian culture Youth Development Program cultural pride awareness of environment: ocean, land, community healthy knowledge & behaviors Hui Mālama o ke Kai Program Description 5th and 6th Grade “at-risk” youth 40-50 youth attending 2 grade schools in Waimanalo 95% of participants are Native Hawaiian Students recruited by counselors beginning of year Hui Mālama o ke Kai Program Description Not Babysitting, after-school care Youth development program– Values & skills cultural pride awareness of environment: ocean, land, community healthy knowledge & behaviors Capitalizes on the strengths of Native Hawaiian culture and Waimanalo community Hui Mālama o ke Kai Program Description Child Mind ~ Body ~ Spirit Family Community/Society Hui Mālama o ke Kai Evaluation Framework Program staff and school teachers refined appropriate, feasible, useful tools and measures Evaluators acknowledge that they are not experts in the community culture while the community holds the expertise in identifying and addressing the issues Enhanced utility and feasibility Hui Mālama o ke Kai Evaluation Model Native Hawaiian Cultural Youth Health & Wellness Knowledge, Attitudes, Values Behaviors Laulima: cooperation Positive youth development : Na`auao: learning School/Education, Perception of Environment, Wiwo`ole: confidence Family Support, Interpersonal Relationships, Conflict Lokahi: harmony Resolution/ Anger Management, Self-Perception Mahalo: gratitude Physical activity Nutrition Violence Alcohol, tobacco, other drugs HMK Program-- Culturally relevant: oHealth education oActivities Rubrics of Native Hawaiian Values Lokahi Na`auao “Calm, harmony” “Independent, Responsible” Wiwo`ole Laulima “Diligent” “Cooperative” Mahalo “Gratitude, respect” Rubrics of Native Hawaiian Values Measure students’ progress in learning and applying Hawaiian values taught in HMK Developed jointly by principal investigator, evaluators, and community representatives beginning 2004-2005 school year Evaluators proposed criteria of assessing students’ behavior and performance based on measures used in the educational evaluation field Rubrics of Native Hawaiian Values Program director and specialists adapted and refined characteristics of the ideal HMK student demonstrating mastery of each of the core Hawaiian values. 5 point rating scale on each value observed by the prevention specialists was developed to rate student’s practice of cultural values. Validation of Hui Mālama o ke Kai Evaluation Instruments Internal reliability—quantitative measures Intraclass correlations—same measures over numerous instruments Face validity—qualitative tools Methods triangulation—measures from all data sources Triangulation: Native Hawaiian Values- Raters Intraclass Correlations (ICC) of HMK Rubrics from Prevention Specialists with Other Raters Measure Data Source ICC P Df Interpersonal Relationships Student survey 0.53 0.005*** 47 Self-Perception Student survey 0.55 0.005*** 44 School/Education Student survey 0.34 0.076 47 Conflict Resolution/ Anger Management Student survey 0.69 0.000**** 47 Child’s Ability to Deal with Varying Situations Parent survey 0.54 0.005*** 45 Dealing with Child’s Attitudes Parent survey 0.63 0.001**** 42 Child’s Hawaiian Values Parent survey 0.47 0.019* 44 General Learner Objectives Schoolteachers 0.47 0.006** 27 Rubrics of Native Hawaiian Values 3 evaluation periods: Dec. 2006, Mar. 2007, May 2007 Significant increase in ratings between each period, p=0.000 (1-5: almost never-almost always) Value Definition Q1 (N=35) Q3 (N=28) X SD X SD Lokahi Calm, harmonious 3.2 1.01 4.6 0.57 Wiwo`ole Work hard 3.1 0.77 4.8 0.42 Na`auao Work on own, 3.0 0.95 4.4 0.63 responsible Laulima Reliable, cooperative 3.3 0.97 4.6 0.57 Mahalo Thankfulness, respect 3.6 0.84 4.8 0.52 Parent Pre & Post Surveys Measure effect of HMK program from parents’ perspective Increase in parents’ mean ratings pre-post Significant increase in ratings on child’s practice of Hawaiian values Scale: 1-5: poor-very good Parent Survey Pre 10/06 Post 6/07 Mann-Whitney U Measures N # Alpha Mean SD N # Alpha Mean SD Z p Items Items Child’s ability to deal 51 16 0.948 3.8 0.74 24 14 0.948 4.3 0.59 -1.1 0.263 w/ varying situations Child’s attitude 48 13 0.956 3.8 0.80 24 12 0.903 4.3 0.58 -1.6 0.105 Child’s Hawaiian values 50 3 0.930 3.8 1.02 24 3 0.865 4.6 0.48 -2.3 *0.022 Parenting skills 51 7 0.917 3.9 0.73 24 4 0.035 4.3 0.77 -0.4 0.723 Triangulation: Native Hawaiian Values: Data Sources Data Source Results Psychometrics-Validity Quantitative 1) Student survey Significant increase (p=0.000) understanding/practice of alpha=0.675, 3 items Hawaiian values/culture, 3.5-3.8, 4 pt. scale 2) Parent survey Significant increase (p=0.022) child’s Hawaiian values 3.8-4.6, alpha=0.898, 2 items 5 pt. scale 3) HMK Rubrics of Significant increase (p=0.000) in ratings over school year: 0.939 Hawaiian values 3.2-4.6, 5 pt. scale Face validity—HMK staff Qualitative 1) Retreat Parents also wanted to further learn about Hawaiian culture Face validity—HMK staff evaluations and values 2) Parent interview Child learned Hawaiian values at HMK Face validity—HMK staff 3) Student journal Wrote, defined, explained Hawaiian values and how they Face validity—HMK staff have been applied Qualitative Results: Native Hawaiian Values Student journals “Today, I learned the value ‘laulima.’ I used laulima by helping everyone. I scraped all the black stuff off the kalo. I will use this in my life by helping everyone out to get the job done.” Summary and Results Rubrics of Hawaiian values correlated with: Youth development outcomes on the student survey (p=0.000 to 0.005, ICC=0.69 to 0.53) All of the parent attitude outcomes on their child on the parent survey (p=0.001 to 0.019, ICC=0.63 to 0.47). School teachers’ grading of students on General Learner Objectives (p=0.006, ICC=0.47): e.g., self directed learners, community contributors, complex thinkers Triangulation also indicated reliability across different evaluation tools Summary & Conclusion Results from the qualitative and quantitative data sources corroborated an increase of understanding, knowledge, and practice of Hawaiian values among the youth. Remarkably, measures most reliable and valid addressed the key program outcome— understanding and practice of Hawaiian cultural values. Conclusions Corroboration of results from the different data sources indicate reliability of measures and validity of HMK Rubrics to evaluate youth understanding and practice of Hawaiian values. Rubrics of Native Hawaiian Values may serve as a tool to evaluate culturally based youth development programs for Hawaiians Conclusions Community- driven program evaluation: May identify appropriate measures and outcomes Address reliability and validity of instrumentation to evaluate unique outcomes Community Based Participatory Inquiry to Reduce Smoking Among Native Hawaiian and Filipina Girls in Hawaii Acknowledgements Maile Goo, PsyD: Co PI National Cancer Institute Pilot Project to Reduce Cancer Health Disparities `Imi Hale Native Hawaiian Cancer Network Background Native Hawaiian and Filipina females have high and rising smoking rates Females smoke at lower rates than males in their respective ethnic groups Background Specific Aims Goal: To investigate knowledge, attitudes, perceptions, and sociocultural factors that may be related to the high tobacco use among Filipino American and Native Hawaiian girls in Hawai`i, using CBPA, to develop future innovative tailored youth delivered smoking prevention interventions. Specific Aims 1) To identify barriers and facilitators for Native Hawaiian and Filipina American girls toward smoking. 2) To determine which barriers and facilitators among Native Hawaiian and Filipina American girls toward smoking are related to their respective ethnicities. Specific Aims 3) To identify anti-smoking messages relevant to Native Hawaiian and Filipina American girls, and define components for a future novel youth-led anti-smoking program using a participatory research framework and adapting evidence-based strategies on youth smoking prevention to reduce smoking among Native Hawaiian and Filipina girls. Community Partners UH Student Excellence in Equity and Diversity (SEEd) REAL, Cancer Research Center Coalition for a Tobacco Free Hawaii Kauai District Health Office `Imi Hale Native Hawaiian Cancer Network Asian American Network of Cancer Awareness, Research, and Training (AANCART) Methods 15-21 focus groups, 6-10 participants Recruitment: SEED program Peer facilitators—REAL-conduct groups Comprehensive focus group training curriculum developed by `Imi Hale Qualitative analysis, deductive approach Summaries and results presented to `Imi Hale and AANCART Future Plans Community partners will guide Native Hawaiian and Filipina girls to create and implement their tailored youth led, program(s) based on this study’s results Build ownership among the involved youth with opportunity to develop and implement their own programs in their own schools and communities Community Benefits Strengthen partnerships with continued community guided efforts to address key health issue Partnerships may raise awareness in the Native Hawaiian and Filipino communities on smoking prevalence among girls to potentially ignite further community action Community Approaches to Address Health Disparities: Concluding Remarks Mutually agreed priority issues from all sectors facilitated successful mobilization around the issues Community- driven program evaluation helps to identify most relevant outcomes for the community and their appropriate measures