Assessment in Psychiatric Rehabilitation for Planning

							Assessment in Psychiatric
Rehabilitation for Planning
Interventions and Tracking
                 Outcomes

  Kim MacDonald-Wilson, Sc.D.
           Alexis Henry, Sc.D.
        Sandra Wilkniss, Ph.D.
      Joanne Nicholson, Ph.D.
Agenda
   Introduction to Assessment
   Brief Primer on Issues in Measurement
   Person Level Assessment for Intervention
    Planning
   Program Level Assessment for Outcome
    Tracking
   Integrating Assessments in PR Programs
Introduction to
   Assessment
Kim MacDonald-Wilson
Review of Assessment
Experiences
   Questions
       How do you use assessments?
       What assessments do you use?
       What questions do you have about assessment
        (problems, issues)?
What is it?
   Assessment is appraising or estimating the
    attributes or characteristics of a person, a group, or
    programs. The tools of assessment can include:
       Checklists
       Inventories
       Observational Schedules
       Needs Assessments
       Rating Scales
       Tests
   In PR, we focus most often on:
       Assessment of skills, resources, supports
       Measurement of outcomes of our interventions
Why Do It?
   Intervention planning
       Understanding client’s strengths, resources,
        challenges, and needs
       Understanding the client’s values and goals
       Setting intervention goals
   Outcome tracking
       Recording change in key areas
       Demonstrating impact of the intervention
Principles of Assessment in
PR
   Focus on functioning
   Centered on and driven by person in
    recovery
   Environment is important
   Useful for practitioners and managers
   Easily administered
   Issues in
Measurement
     Alexis Henry
Assessment Methods
   Standardized or informal
   May assess skills, knowledge, attitudes
   Methods
       Observational – simulated or situational
         Generally used to assess skills

       Paper-and-pencil self-ratings or checklists
         Generally used to assess knowledge and attitudes

       Interviews
         Can be used to assess skills, knowledge and attitudes

         Best for developing rapport and setting intervention goals
General Considerations in
Conducting Assessment
   When
       Intervention planning
         At intake (of course), but also periodically

       Outcome tracking
         When it occurs (of course)

         Short, medium or longer term outcomes

   Where
       Program/clinic; in “real-world” context
   Relevance
       Are assessments relevant to interventions/services
         Important for both intervention planning and outcome tracking

   Reliability, validity, cultural sensitivity
Reliability of Assessment
   Stability, consistency, dependability
   Internal consistency
       how well do items on an instrument measure the
        same concept/construct
   test-retest reliability
       stability over time
   interrater reliability
       stability/agreement across raters
Validity of Assessment
   Does assessment measure what it purports to
    measure?
       meaningfulness, usefulness
   Face validity
       Does the assessment “look valid” to the person being
        assessed
   Content validity
       does the assessment adequately cover the domain being
        assessed
   Construct validity
       does the assessment correspond to the theoretical
        concepts (“constructs”) underlying the measure
         e.g. self-esteem scale
Validity
   Construct Validity
       Convergent and divergent (discriminant) validity
       Criterion-related validity
           concurrent - uses an existing criterion
           predictive - uses a future criterion
   Responsiveness
       Is the assessment sensitive enough to detect small
        changes
Cultural Sensitivity
   Considerations
       Gender
       Age
       Race
       Ethnicity and language
       Cultural norms
       Socioeconomic status
       Context/environment
        Person Level
      Assessment for
Intervention Planning
     Kim MacDonald-Wilson
Why conduct assessments for
intervention planning?
   Involve the person in rehabilitation and
    recovery
   Provide direction
   Efficiency
   Mobilize needed resources
   Organize the interventions
Domains for Person Level
Assessment
   Work
   School
   Housing/Community Living
   Social/Recreational/Community Participation
   Parenting and other Social Roles
   Quality of Life
   Empowerment and Recovery
What to Measure in Person
Level Assessment
   What is the goal environment or domain?
   Skills strengths and deficits
       What skills are required to be successful and
        satisfied?
       What can the person do or not do?
   Environmental Resources and Supports
       What supports are required to be successful and
        satisfied?
       What does the person have or not have?
Assessment of Functioning
   Assessment of Functioning does not = Skills
    assessment
   Global Functioning – Focus on Outcome of
    Functioning
       Performance in Roles (spouse, worker)
       Performance in Domains (social, psychological)
       Status (employment, educational, residential)
   Tools – GAF, BASIS-32, Multnomah
    Community Ability Scale, Life Skills Profile
Skill Assessment Examples
   Skill Functioning = Focus on Behaviors
   Living
       Test of Grocery Shopping
       Independent Living Skills Survey – ILSS
       Client Assessment of Skills, Interests, and Goals –
        CASIG
   Work
       Work Behavior Inventory – WBI
       Vocational Cognitive Rating Scale - VCRS
   Parenting
       Parenting Well Strengths and Goals
Resource Assessment
Examples
   Social Support
       MOS Social Support Scale
   Parenting
       Family Resources Scale
Examples of Assessments for
Intervention Planning
   Observational assessments
       Test of Grocery Shopping Skills
       Vocational Cognitive Rating Scale
       Work Behavior Inventory
   Self-report measures
       ParentingWell Strengths and Goals
       Family Resource Scale
       MOS Social Support Scale
   Interviews
       Independent Living Skills Survey
       Client Assessment of Strengths, Interests and Goals
        (CASIG)
Test of Grocery Shopping Skills
   Developed to be used in conjunction with a grocery
    shopping skills training curriculum for people with
    serious mental illness
   Assesses person’s ability to locate 10 specific
    common food items in a regular grocery store
   Can be used as a baseline assessment and a follow-
    up assessment after training – 2 versions
   Developed by Hamera and Brown at University of
    Kansas Medical Center
       http://www2.kumc.edu/pdrp/interventionandoutcome.htm
Test of Grocery Shopping Skills
   Examines the person’s ability to find:
       Correct item
          Correct size
          Lowest price
       Takes speed into account
   Sample items:
       15 oz. can red kidney beans
       1 lb. bag salad mix
       16 oz. frozen corn
       8 oz. low fat/nonfat strawberry yogurt
       16 oz. spaghetti
Vocational Cognitive Rating
Scale
   A 16-item instrument designed to measure cognitive
    skills in the workplace
   Ratings are accomplished by a clinician who has
    observed the individual in the work setting and has
    interviewed the onsite supervisor
   Behavioral anchors are provided for each rating on
    each item
   Developed by Greig, Nicholls, Bryson, & Bell (2004)
    at Yale University School of Medicine
VCRS – Sample Item
   Stays focused when performing a simple task
       Consistently Inferior performance. Is consistently off task.
        Requires nearly continuous help from job coach/supervisor
        to stay focused.
       Occasionally inferior performance. Can remain on task for
        brief intervals, but requires frequent monitoring by job
        coach to stay focused.
       Adequate performance. Generally is able to stay on task
        and keep focused when doing simple tasks with minimal or
        infrequent assistance.
       Occasionally superior performance. Stays on task, and
        only rarely requires assistance with focusing.
       Consistently superior performance. Consistently completes
        simple tasks in a focused, organized manner.
Work Behavior Inventory
   A 36-item work performance assessment
       Social Skills
       Cooperativeness
       Work Habits
       Work Quality
       Personal Presentation
   Ratings on a 5-point scale are accomplished by a
    clinician who has observed the individual in the work
    setting and has interviewed the onsite supervisor
   Indicates both strengths and deficits, Behavioral
    anchors are provided for each rating on each item
WBI – Sample Items
   Each item rated on a 5-point scale from 1-
    Consistently inferior performance to 5-
    Consistently superior performance
       Expresses positive feelings appropriately
       Accepts constructive criticism without becoming
        upset
       Begins work tasks promptly
       Corrects own mistakes
       Personal hygiene is satisfactory
       General Rating of Work Behavior
ParentingWell™ Strengths and
Goals
   A parenting self-assessment and
    collaborative goal setting tool
   Developed for use with parents with mental
    illness and their families
   26 items covering everyday demands of
    parenting
   Developed by Nicholson & Henry
   Available at: www.parentingwell.org
ParentingWell™ Strengths and
Goals: Sample Items
                                   ParentingWellTM STRENGTHS & GOALS
                                                                                                    I’d             Check
                                                                                                 like to            items
                                                                             This is a    I do      do     Does       to
Here is a list of things you may need to do as a parent. For each one that   strength     this     this     not     work
applies to you, circle the answer that describes you best.                   of mine.    okay.   better.   apply.     on.
1. Manage everyday household tasks                                           Strength    Okay    Better    DNA

2. Plan and make healthy meals                                               Strength    Okay    Better    DNA

3. Understand the relationship between my feelings                           Strength    Okay    Better    DNA
   and my actions
4. Manage my family’s money                                                  Strength    Okay    Better    DNA

5. Set limits with my child                                                  Strength    Okay    Better    DNA

6. Have positive interactions/visits with my child                           Strength    Okay    Better    DNA

7. Have a pleasant routine with my child                                     Strength    Okay    Better    DNA

8. Find fun things to do with my child                                       Strength    Okay    Better    DNA

9. Get adequate child care for my child                                      Strength    Okay    Better    DNA

10. Balance work or school, and parenting                                    Strength    Okay    Better    DNA
Family Resource Scale
   Assesses family self-sufficiency by determining the
    adequacy of different types of resources in
    households of young children.
   30 item self-report tool, rated using a 5-point Likert-
    type scale anchored by "not at all adequate" to
    "almost always adequate.“
   Covers access to food, shelter, financial resources,
    transportation, health care, time to be with family,
    child care, and time for self.
   Developed by Leet & Dunst
   Published in Dunst, C. J., Trivette, C. M., & Deal, A. G. (Eds.) (1994).
    Supporting and strengthening families. Methods, strategies and
    practices. Cambridge, MA: Brookline Books. http://brooklinebooks.com/
   Family Resource Scale:
   Sample items

                                                                                      Almost
                                      Not at All    Seldom    Sometimes    Usually              N   R   D
To what extent are the following                                                      Always
                                      Adequate     Adequate   Adequate    Adequate              A   F   K
resources adequate for this family?                                                  Adequate
Food for 2 meals a day                    1           2          3           4          5       7   8   9
House or apartment                        1           2          3           4          5       7   8   9
Money to buy necessities                  1           2          3           4          5       7   8   9
Enough clothes for your family            1           2          3           4          5       7   8   9
Heat for your house or apartment          1           2          3           4          5       7   8   9
Indoor plumbing/water                     1           2          3           4          5       7   8   9
Money to pay bills                        1           2          3           4          5       7   8   9
Good job for yourself or spouse/
partner
                                          1           2          3           4          5       7   8   9
Medical care for your family              1           2          3           4          5       7   8   9
Public assistance (SSI, AFDC,
                                          1           2          3           4          5       7   8   9
Medicaid, etc.)
Dependable transportation
(own car or provided by others)
                                          1           2          3           4          5       7   8   9
Time to get enough sleep                  1           2          3           4          5       7   8   9
MOS Social Support Survey
   19 item self-report tool measuring multiple
    dimensions of social support
       Emotional/informational
       Tangible
       Affectionate
       Positive social interaction
   Designed for use with individuals with long-term
    conditions, including serious mental illness
   Developed by Sherbourne & Stewart
   Available from:
    http://www.rand.org/health/surveys_tools/mos/mos_socialsupport.ht
    ml
MOS Social Support Survey:
Sample items
People sometimes look to others for companionship, assistance, or other
types of support. How often is each of the following kinds of support
available to you if you need it? Circle on number on each line.

                                                          None     A little   Some     Most of   All of
                                                          of the   of the     of the     the      the
                                                           time     time       time     time     time
Emotional/informational support
Someone you can count on to listen to you when you          1         2         3        4         5
need to talk
Someone to give you information to help you                 1         2         3        4         5
understand a situation
Someone to give you good advice about a crisis              1         2         3        4         5
Someone to confide in or talk to about yourself or your     1         2         3        4         5
problems
Someone whose advice you really want                        1         2         3        4         5
Someone to share your most private worries and fears        1         2         3        4         5
with
Someone to turn to for suggestions about how to deal        1         2         3        4         5
with a personal problem
Independent Living Skills Survey
   Designed to assess ADLs and IADLs among people with
    schizophrenia and other serious mental illnesses
   Two versions
        Self-Report Interview: 76 items
              Personal appearance; care of possessions; food prep and
               storage; health maintenance; money management;
               transportation; leisure and community participation; job seeking
               and job maintenance
        Informant Interview: 104 items
              Same as above; also includes eating behaviors; social
               relationships
Both instruments are available in appendix of published article: Wallace, C. J., Liberman, R. P.,
    Tauber, R., & Wallace, J. (2000). The Independent Living Skills Survey: A comprehensive
    measure of the community functioning of severely and persistently mentally ill individuals.
    Schizophrenia Bulletin, 26(3), 631-658.
   Independent Living Skills Survey-
   I: Sample items

How often did the person perform each behavior                                                          No
                                                      Always   Usually   Often   Sometimes   Never
in the last 30 days?                                                                                 Opportunity
Washed clothes by hand or machine using proper
amount of detergent (without prompting)
Bathed/showered with soap at least twice/week
Vacuumed or mopped as needed
Prepared foods that require some cooking such as
eggs, TV dinners
Self-administered medication (if not on medication,
when the person last took medication)
Paid bills such as rent, utilities, phone, and
transportation
Used public buses, trains, or subway
Worked regularly on a hobby
Arrived at work on time and followed daily break
and work schedule
Ate at reasonable pace
Maintained conversations for more than brief
greetings
    Client’s Assessment of Strengths,
    Interests and Goals (CASIG)
   A lengthy, structured interview designed for use with people with
    serious mental illness, covering multiple domains of functioning
   Assesses client’s goals related to:
      Housing/living; financial/vocational; relationships; spiritual/religious;
        health
   Assesses client’s skills/strengths/needs related to:
      Lifestyle supports; money management; health management;
        nutrition; vocational; transportation; friends; leisure; personal
        hygiene; care of possession; medications and side effects; personal
        rights; cognition; symptoms; risk behaviors
   Assesses satisfaction with:
      Quality of life; quality of treatment
   Developed by Wallace, Lecomte, Wilde, & Liberman (2001) at UCLA
   Available at: http://www.ct.gov/dmhas/LIB/dmhas/MRO/CASIG.pdf
CASIG: Sample items
   Goals
       Housing/Living Goals
           “One year from now, what would you like your living
            arrangements to be?”
       Financial/Vocational Goals
           “Would you like to improve your money/finances in the
            next year? How might you be able to improve them?
            Do you want to get a job or go to school in the coming
            year?”
CASIG: Sample items
   Skills/Strengths
       Health management
           “…at any time in the last 3 months did you:
               “Make most of your own appointments with your doctor,
                case manager, etc?”
               “Administer your own medication?”
       Food preparation
           “…at any time in the last 3 months did you:
               “Plan your meals with a health balance of foods?”
               “Use a stove to prepare meals?”
      Individualized
assessment and goal
             setting
     Goal Attainment Scaling
                Alexis Henry
Goal Attainment Scaling
   Framework for developing service goals
    (outcomes) that are measurable, relevant,
    attainable
   Individualized goals
   Quantitative index of client’s progress
       Can compare client to self over time
       Can compare attainment across clients
Advantages of GAS
   Not bound by any theoretical orientation,
    treatment or outcome measure
   Allows for individualization of goals
   Allows for prioritizing goals
   Numeric index of performance over time
   Easily computed
   Useful in informal evaluations or in formal
    experimental designs
Disadvantages of GAS
   Reliability issue needs to be considered
   Outcomes best assessed by blind judge
   Possibility of floor effect
   Possibility of misuse if goals are “too easy”
Steps in GAS
   Identify overall objective or domain
   Identify specific problem area
   Identify behavior that indicates improvement
   Determine data collection method
   Specify expected outcome
   Specify most and least favorable outcome
   Determine current status and timeframe for
    evaluation
Goal Attainment Scale
   -2 – most unfavorable outcome
   -1 – less than expected outcome
   0 – expected outcome
   +1 – greater than expected outcome
   +2 – most favorable outcome

   Goals can be weighted to reflect priorities
GAS Example:
25 y.o. male wants to increase work
hours and earnings


   +2 – works 20 hrs/week at $10/hour
   +1 – works 10 hrs/week at $10/hour
    0 – works 10 hrs/week at minimum wage
   -1 – works less than 10 hours/week
   -2 – no longer working
GAS Example:
35 y.o. mother wants to have more fun
with kids age 8 and 10

   +2 – family participates in mutually enjoyable
    leisure/recreational activity 1x per week
   +1 – family identifies mutually enjoyable
    leisure/recreational activity; participate 2x per month
    0 – family engages in leisure together about once
    per month
   -1 – family leisure/recreational activities happen less
    than once per month
   -2 – no regular family leisure/recreational activities
   Program Level
  Assessment for
Outcome Tracking
    Sandra Wilkniss, PhD
Program Level Assessment for
Outcome Tracking
   Why track outcomes on a program level?
   Domains of outcome tracking
     Clinical Status
     Recovery Status
     Quality of Life
     Functional Status
     Safety and Health
   Context and Methods for collecting and interpreting outcomes
   Types of Services and related outcome tracking issues
   Sample outcome tracking tools
   Exercise: Selecting assessment tools for outcome tracking
Why track outcomes on a program
level?
   Access to services
   Quality of services
   Cost/benefit
Domains of outcome tracking
 - Demographics & Clinical Status
    Recovery Status/Quality of Life
    Functional Status
        Roles/Community Integration
        Skills
        Resources
    Safety and Health
Demographics & Clinical
Status
   Demographics
   Diagnostic and Historical information
   Service use (baseline and change)
   Physical health related outcomes
   Cognition
   Hospitalization
   Jail and prison contact
Recovery/Quality of Life Status
   Domains
       Recovery
       Empowerment
       Self Efficacy
       Hope
       Satisfaction
       Quality of Life
   Methods
       Consumer Self-report tools
       Provider Evaluation tools
       Family/significant other input
       Interviews
         Quantitative
         qualitative
       Proximal markers
Functional Status
   Living situation
     Housed or homeless
     Level of independence
     In community of choice
   Education
     Type of institution
     Number of credit hours attempted/completed
     Degree or voc/tech certificate
   Work
     In a job of consumer’s preference
     Competitive
     Salary and/or benefits
     How much
     For how long
     Advancement
     Follow along support variables **
Functional Status (continued)
   Community Integration
   Instrumental Roles
   Skills and Personal Resources
       Social
       Vocational
       Financial
       Recreational
       Cognitive
       Others
   Environmental resources and supports available
       Community
       Family
       Social capital
       Others
Safety and Health


   Personal and other safety
   Physical Health and Wellness
Context and Methods for
Collecting and Interpreting Outcomes
   Context
       Cultural Sensitivity
           Philosophy and interpretation
           Specific measures
   Cognition
   Methods
       Baseline tracking
       Interval tracking
       sources
Services and Interventions Offered:
Type, quantity, and quality
   Outreach
   Linkage
   PSR
   Individual and group
   Peer Support
   Integrated Health care (primary care & mental health)
   Pharmacotherapy
   Adjunct Interventions (e.g., cognitive support)
   Evidence-Based or emerging best practices
       GOI/penetration
       Fidelity
Sample Tracking Tools
   Comprehensive outcomes packages
   Specific Measures
       Cultural diversity measures
       Recovery/Satisfaction Measures
       Empowerment
       Cognitive Functioning
       Skills
       Environment
Exercise: Selecting Tools for
Outcome Tracking
   Selection Strategies
       Determine program emphasis/focus
       Evaluate needs and preferences of individual consumers
       Evaluate resources and means for data collection,
        description/analysis, interpretation and dissemination
       Review sample tools in each domain of interest
         evaluate pros and cons
             Psychometric properties vs. uniqueness
             Thorough vs. brief and more accessible
             Language geared toward consumers
             Issues of cultural diversity
             Etc.
Integrating Assessments in
  Psychiatric Rehabilitation
                 Programs:
Where the rubber meets the road…

               Joanne Nicholson
            So far we’ve covered…
         Principles for assessment

           Issues in measurement

         Person-level assessment

        Program-level assessment
Integrating assessments in
programs requires consideration of:

    Relevant domains, e.g., work, education, etc.
        Skills
        Resources and supports
    Individual’s goals
    Program’s goals
    Additional outcome tracking needs, e.g., data
     required for program funders?
    Organizational resources and supports
Providers’ Motivations for Using
Evidence-Based Practices*
   Enhanced professional competence
   Perceived utility
   Fit with usual tasks and duties
   Fit with one’s own experience
   Fit with mission of organization, structure
    of the intervention, & organizing
    conceptual model
                          *(Aarons & Palinkas, 2007)
Potential Barriers to Use
   Lack of training, motivation, connection
   Measures, methods viewed as irrelevant
   Measures, methods viewed as
    burdensome
   Measures, methods not viewed as useful
   Lack of organizational support→resources
   Program measures and research
    measures don’t “mix” well
Strategies for Overcoming
Barriers to Assessment
   Educate re: rationale
   Engage program staff & participants in
    process
   Anticipate & address logistical challenges
   Create opportunities for feedback &
    refinement
   Use data in meaningful, productive ways
Learning Activity
   Break into groups ↔ Select a domain
   Discuss assessment issues:
       Intervention planning? Outcomes?
       Existing measures? Adaptation? Refinement?
       Logistics – Why? Who? How?
       Mechanisms for feedback to staff, participants?
   Identify implementation challenges &
    suggest strategies for addressing
   Report back