Qualitative Methods and How to Dovetail With Quantitative Methods
Melanie Percy, RN,PhD,CPNP The University of Texas at Austin School of Nursing
Dawn Nixon, MA Red Cliff Birth to Six Mental Health Program Red Cliff Ojibwe Reservation Wisconsin
Young Wings
What is Young Wings?
Initial Definition: A pediatric mental health program, based upon Dr. T. Berry Brazelton Touchpoints model, located within a tribal early childhood center. It will provide early identification of, and intervention for, signs of developmental derailment in children aged birth to 6. (Goals 4.1, 5.3, 5.4)
Initial Plan for Services
Developmental screening. Identification of children with early signs of difficulty. Interventions:
Psychological assessment. Parent education. Individual play therapy. Referral to community partners (e.g. Evenstart Literacy program).
Barriers to Receipt of Mental Health Services Addressed
Lack of pediatric mental health availability.
60 mile round-trip. 3-6 month wait. Lack of reliable transportation. Limited understanding of hx., Native American values, range of acculturalization and significance of spirituality and ritual. (Goals 1.1, 3.1, 3.2)
Lack of cultural competency
Stigma for receipt of mental health services.
Initial Evaluation Plan
Pre- and post- administration of the Parenting Stress Index. Client satisfaction survey. Child/family identification rate. Longitudinal statistical analysis of county child abuse and neglect statistics. Analysis of community utilization rates.
What is Young Wings?
All that was in the initial plan, plus:
A valued member of the ECC management team who effects and writes policy from a mental health perspective. An advocate for parents with the local school district. Liaison between ECC, school district, and community. A resource for the ECC and community during times of stress. And more……
Questions…..
How would the families/staff/community define Young Wings? How is it perceived? How has Young Wings been accepted by families/staff/community? What factors make Young Wings successful and accessible?
Core Principles of Qualitative Evaluation
Each person/community is unique. Each person/community deserves respect. People and communities should be understood in context and holistically. The process (how things are done) is as important as the outcomes (what is achieved). Information should be openly shared and honestly communicated. Patton,
1990
Why Qualitative?
“The personal nature of qualitative inquiry derives from its openness, the evaluator‟s close contact with the program, and the procedures of observation and in-depth interviewing, particularly the latter, which communicate respect to respondents by making their ideas and opinions (stated in their own terms) the important data source for the evaluation.”
Patton, 1990 pg.124
Cornerstones of Evaluation
Why? To validate your program goals. To guide future program development. To maintain or increase funding.
Basic questions to consider
What are your goals? Who is your population? Who are the stakeholders? Where does your funding come from and what will you have to do to continue it?
Quantitative vs Qualitative
Quantitative - statistical analysis of an objective number. That number is usually created by counting the number of clients served, or statistical analysis of data received.
Qualitative - using written or spoken interviews to determine impact of program on families. Then analyzed for themes and reported as a narrative.
Why quantitative?
Use standardized measures that fit diverse opinions and experiences into predetermined categories. Statistically significant findings. Broad, generalizable set of findings.
Why Qualitative ?
Can study selected issues in depth. Provides detailed data Direct quotation, careful descriptions are used in an openended narrative. No quantitative tools may exist for variables of interest.
Comparing qualitative and quantitative data
Please rate your satisfaction with the healthcare you child has received in this clinic. a. very satisfied b. somewhat satisfied c. satisfied d. not satisfied e. very dissatisfied
Please tell me about your experience of receiving healthcare for your child in this clinic.
Kay Gibbons is wonderful! Very kind, thoughtful. I hope that this clinic is not taken away because we need it badly, and it‟s helped us a lot already. I am happy because they speak Spanish.
Philosophy behind qualitative evaluation
What is reality? What is knowable? What is the nature of human experience? Value seeing the world as understood by the client. Desire to capture accurately and thoroughly the experiences, and perspectives of the staff, administration, and participants of the program being evaluated.
Key Elements
Observing
Participant observation of program Observing during interviews (nonverbal behavior, relationship with interviewee) Formal interviews Informal, naturally occurring conversations.
Interviewing
Doing Interviews
Qualitative evaluation means the evaluator must quickly attempt to develop an “insider‟s view” of what is happening While maintaining an outsider‟s ability to describe the experience.
Gaining entry
One of most difficult parts of qualitative evaluation is knowing whom to interview or observe. Need “key informants”. People who have insight and respect in the community. Key informants can explain and interpret what the evaluator is experiencing.
Doing Qualitative Evaluation
Consider: what was said? what was happening? what was the context? what is the history of the program? the larger community?
Inductive Analysis
Audiotapes will be transcribed and transcripts of tapes will be analyzed. Attempt to make sense of program without imposing pre-existing expectations on the program setting. Begin with specific observations and builds towards generalized patterns
Analyzing Data
Multiple computer programs have been developed to analyze qualitative data:
Ethnograph, Hyperesearch, Nudist, Hermann, etc Transcripts are entered into program and themes are developed
Statements are placed on cards and sorted according to their “theme”. Themes are collapsed and reorganized as major patterns are seen and categorized. Then rival or competing themes and explanations should be examined and rejected or explained.
Presenting Data
Detailed descriptions and in-depth quotations are the data that are reported to validate the categories decided upon. Presenting qualitative data can be very lengthy and time-consuming, but also interesting and insightful.
Objectivity and Truth vs. Reliability and Validity
Important that the evaluator is neutral - has no theory to prove, and no predetermined results to support. Finally results need to be brought back to the program and discussed to determine the “goodness” of the data. Does it fit the program? Does it seem factual and confirmable?
Summary
Qualitative evaluation can be a useful tool for understanding the impact you are having on a community. Conducting a qualitative evaluation can be difficult and time-consuming but ultimately rewarding as you gain new understandings about your program and the people you serve.
Now – it‟s your turn
In groups of 3-4 write an evaluation question from your clinic as a qualitative question and as a quantitative question. Discuss in your groups what kind of information would you expect from each question. Be prepared to have a spokesperson to present this to the larger group.
Group Discussion
A spokesperson from each group will present the two questions from the group. For group discussion
What makes one qualitative, and one quantitaive? How will the information be different based on these questions? How is each question valuable to the evaluation?
Mock Interviews
Who would your key informant be? Why did you choose that person? Let the group become a focus group, pick one person to be the “interviewer”. Ask one of your questions, and see what kind of information you receive. Report to group.
Analysis
Next, take tapes to the data transcriber, or transcribe them yourself. Once tapes have returned, go ahead and start analyzing - look for similar subject lines.
What does it mean?
Collapse categories What does it all mean? May want to go back to your key informants, to discuss what you think the interviews meant.
4. Have you ever brought your child to the school based health center? 5. What was that experience like? 6. Is there anything else that you would like to tell us about your or your child‟s experience with healthcare?
How to present this to stakeholders
Numbers (statistics) and then talk about themes discovered in qualitative method. Then present theme, quotes that support that theme, and then the final analysis of what it all means.
Example:
Qualitative Evaluation of School Based Health Centers
City of Austin asked to have an evaluation of their school-based health centers. After meeting with city officials, health department administrators, school officials, and school-based health center personnel it was determined to conduct several focus groups with parents of children seen at the center.
Interviews
PTA and community leaders were asked to identify people who would like to participate in the focus group. Also a notice went out in the school newsletter asking people to participate. Participants were given a $20 grocery store coupon to thank them for their time; food and child care were provided.
Focus groups
Focus groups were formed with 6-8 people each. English language and Spanish language focus groups were conducted separately. Focus groups were audiotaped and note takers (undergraduate students) took notes as people talked.
Questions
Questions began as very generalized and became more specific. 1. Tell me about a time when your child was sick and you took your child to a health care provider. 2. Tell me about a good experience with a healthcare provider. 3. Tell me about a bad experience with a healthcare provider.
Themes
Mother's descriptions of their experiences with the health care system were organized into the following thematic categories: "doing what I have to", "caring for my child", "no respect".
“Doing what I have to do”
They won‟t see her if she doesn‟t have a fever, so I always say she has a fever when I go to take her in. It‟s a long way to the health dept clinic. Before the SBHC was here, I would take three buses to get to the clinic. Sometimes it would be pouring rain, and I‟m standing out at the bus stop waiting with my three small children, and one of them is already sick. But you have to take them in when they get sick.
“Caring for my child”
When my child is sick, I or my husband stay home from work. We won‟t get no pay that day but that‟s what you have to do. I‟ve gotten fired before. The medicine is so expensive, sometimes it takes all the money we have for the week.
“No Respect”
I go to the clinic and it takes all morning. They don‟t care that you are just sitting and sitting there. You can see the nurses and doctors just drinking coffee, and here we are just waiting. Even the doctors and nurses talk down to you. They act like you don‟t know nothing. I‟m a good mother „n I love my kids. I‟m taking care of them the best way I know how.
Mothers described feeling angry and frustrated when they were forced to wait for long periods of time with sick children. The attitudes of professional and non-professional staff strongly influenced their satisfaction with the health care experience. Mothers were most satisfied when their child was seen quickly by someone who was caring and attentive to their needs.
Conclusions
This study emphasizes the importance of valuing and respecting the knowledge that mother's bring to the health care environment and clearly demonstrating that respect throughout the health care setting.
Questions? Further discussions? Resources:
Patton, M. (1996). Utilization Focused Evaluation: The New Century Text. Sage Patton, M. (2001). Qualitative Research and Evaluation Methods. Sage. Patton, M. (1995). How to Use Qualitative Methods in Evaluation, Vol. 4. Sage.
Shaw, I. (1999). Qualitative evaluation. Sage Denzin, N., Lincoln, Y., (1998). Collecting & Interpreting Qualitative Materials. Sage Babiss, F. (2004). An Ethnographic Study of Mental Health Treatment & Outcomes: Doing What Works. Haworth Press.