Docstoc

Designing _ Implementing Randomi

Document Sample
Designing _ Implementing Randomi Powered By Docstoc
					   Designing & Implementing
Randomized Controlled Trials For
Community-Based Psychosocial
         Interventions

           Phyllis Solomon, Ph.D.
                  Professor
       School of Social Policy & Practice
          University of Pennsylvania

                March 17, 2010
         Overview of Workshop
•   Introduction
•   So you think you want to do an RCT?
•   RCT Ethical Considerations
•   Planning an RCT
•   NIH Exploratory Research Grants
•   Developing Conceptual Foundation
•   Designing an RCT
•   Implementing an RCT
•   Generalizing RCT Outcomes

                                          2
Introduction
           What is an RCT?
• True experimental design. Participants assigned
  by chance, following consent, to one of at least
  two conditions
• Key features of classic experimental design:
  – Random assignment
     • determines who assigned to which group
  – Pre & post tests
     • outcome measured before & after intervention
  – Control group
     • same experiences as experimental group except
       no exposure to experimental stimulus


                                                       4
         What is an RCT?

• Can have more than two groups
• Sometimes no pre-test measures
• Chance does not necessarily mean equal,
  but known probability




                                            5
        Community-Based
     Psychosocial Interventions
• Psychosocial Intervention – any service,
  program, educational curriculum, or
  workshop whose goal is to produce
  positive outcomes for individuals
  confronted with social &/or behavioral
  issues & challenges
• Community-Based -Conducted in agency
  & social work settings

                                             6
        Community-Based
     Psychosocial Interventions
• Community-based psychosocial
  intervention – reflects impact of
  environmental context in which
  interventions are imbedded, on clients and
  providers & interactions between both/all
  systems
• Less control, more complex environmental
  context with participants with multiple
  problems

                                           7
         RCT vs. Evaluation
• Research uses scientific methodology to
  generate generalizable knowledge
• Evaluation uses same methodology but
  primary goal is not for generalizable
  knowledge
• For NIH grants do not use term evaluation



                                              8
          RCT vs. Evaluation
• In evaluation RCT known as experimental
  study or a randomized field experiment
• Both examine a program or policy
• Both addresses effectiveness & cost
  effectiveness
• Evaluation experimental studies closely
  resemble community-based psychosocial
  RCTs
  – literature in this area may be helpful
                                             9
         RCT vs. Evaluation
• Purpose of RCTs & field experiments may
  differ
• RCT – research –generalized knowledge
• Field experiments – evaluation – answer
  local questions – but also policy questions
  of broader application
• Semantic difference

                                            10
   Psychosocial Community-Based
Interventions = Effectiveness Studies

• Efficacy studies occur under ideal or
  optimum conditions
• Effectiveness studies occur in “real world”
• Efficacy studies greater internal validity
• Effectiveness studies greater external
  validity

                                                11
So You Think You Want
   To Do An RCT?
     Appraising Whether to Move
        Forward with an RCT
Preliminary questions to be addressed before
  moving forward:
• Is the question well justified?
• Is the question an important one to answer?
• Is the question addressing a gap in the
  literature?
• Is the question an ethical one?
• Is the question posing the correct question?
• Would you fund this RCT?
                                                 13
      Appraising Whether to Move
        Forward with Your RCT
Case Example 1
• Is a 90 day Advanced Practice Nurse-Transitional Care
  Model more effective than usual discharge in improving
  adherence to treatment & quality of life for persons with
  SMI being released from a psychiatric hospital?
   – “hand-off” from hospital to home of SMI linked to
     gaps in delivery of MH services
   – Consequently high rates of rehosp & poor outcomes
   – EBP – Advanced Practice Nurse-Transitional Care
     Model improves outcomes following acute medical
     care discharge for elderly adults with complex
     medical problems

                                                          14
     Appraising Whether to Move
         Forward with RCT
Case Example 1 (continued)
 - Intervention hybrid of case management,
 disease management, & home health care
 - Nurse works with hospital team to develop
 discharge plan & then implement in the
 community
 - Believe adapting this intervention potential to
 be equally successful with adults with SMI being
 discharge from acute hospital

                                                 15
     Appraising Whether to Move
         Forward with RCT
Case Example 2
• Is Multidimensional Treatment Foster Care (MTFC)
  Program more effective in reduction of disruptive
  behaviors than traditional Therapeutic Foster Care (TFC)
  among children in foster care?
  - Instability in foster care placement ranges from 22%-
  56%
  - Instability in placement due to child‟s disruptive
  behaviors
  - TFC typically used for children with more demanding
  emotional & behavior needs & has more intensive
  structure & MH services


                                                        16
      Appraising Whether to Move
          Forward with RCT
Case Example 2 (continued)
  - Data on disruptions for TFC sparse but estimated 38%-
  70%
  - Limited evidence on TFC effectiveness – most studies
  descriptive, methodologically flawed
  - Lack of clear standards & specification of actual
  implementation of TFC
  -MTFC – manualized intervention with goals to improve
  well-being & reduce disruptions
  - MTFC placement augmented with coordinating an
  array of clinical interventions in family, school, & peer
  group
                                                          17
      Appraising Whether to Move
          Forward with RCT
Case Example 3
Is CBT for adolescents with sickle cell disease (SCD) more
   effective than medical management of the disease in
   increasing coping strategies?
   - adolescents with SCD have a number of adjustment
   difficulties that have received little attention
   - some psychosocial difficulties include stress-
   processing
         e.g. decreased coping strategies, lack of knowledge
         of SCD
   - need to promote biological & psychosocial adjustment

                                                           18
    Appraising Whether to Move
        Forward with RCT
Case Example 4
• Is Forensic Assertive Community Treatment
  (FACT) more effective than forensic intensive
  case management (FICM) in a variety of
  psychosocial and clinical outcomes for homeless
  adults with SMI leaving jail?

  - Pop. has multiplicity of needs due to mental
  illness, homelessness, & criminal justice
  involvement


                                                   19
     Appraising Whether to Move
         Forward with RCT
Case Example 4 (continued)
- cognitive deficits & poor social skills complicate
  ability to coordinate efforts to meet needs
  - FICM single point of planning, monitoring &
  accountability considered beneficial for this pop.
      - FICM specialized ICM
  - FACT –team approach (shared caseload), self
  contained intervention to meet all needs of client
  – includes psychiatrist, case managers, etc.
  - Based on ACT for criminally involved
                                                   20
RCT Ethical Considerations
     RCT Ethical Considerations
•   Appropriate question to ask
•   Who ethically eligible to randomize
•   What ethical comparison
•   How & when to randomize
•   When are providers human subjects
•   What is ethical responsibility at termination


                                                22
 Justifying the RCT to Doubters
• Want to provide most effective services to clients
• Expectation when treated by a doctor
• RCTs best means to making causal inference
  with high degree of confidence
• Unethical to offer untested intervention
• Not denying better treatment to controls
  – if answer known, there would be no need for study
• Frequently those who receive services
  determined on a haphazard or a biased basis
                                                        23
        Ethical Justification For
            Randomization
• Lack of adequate evidence of
  effectiveness of exp. intervention
  understudy
• Experimental intervention theoretically
  justified to potentially benefit target pop.
• Uncertainty of effectiveness (equipoise) –
  otherwise no scientific basis for RCT


                                                 24
       Principle of Equipoise

• Substantial degree of uncertainty /
  ambiguity necessary
  – Specific population
  – Setting




                                        25
       Integration of Practice &
           Research Ethics
• Practice – interventions designed solely to
  enhance well-being of client & has
  reasonable chance of success (Belmont
 Report, 1979)
• Research – activities designed to test
  hypothesis, permit conclusions to be
  drawn, thereby contribute to generalized
  knowledge (Belmont Report, 1979)
• RCTs = Practice & Research

                                             26
       Integration of Practice &
           Research Ethics
• Practice ethics = human subject
  protections
   – may conflict w/ scientific rigor
• Participant deterioration in experimental
  condition results in biased attrition
• Exclusion criteria for clinical reasons –
  reduce external validity

                                              27
         Ethics of Scientific
        Untested Interventions

• Experimental intervention at least as
  effective as TAU
• Do no harm - even if voluntarily consents
• Risks assessment for participant
  – Extends to others & community-at-large



                                              28
 Ethics of Selecting Control Group
• Justify no service comparison
  – Gas to no gas
• Waitlist may be justified if agency normally
  has waiting list, or no service offered
• Inert intervention may be justified
• TAU may be most justifiable comparison



                                             29
               Consent Forms
• Must inform potential participant that will
  receive experimental intervention by
  chance
     • i.e., like flipping a coin
• Indicate chance of receiving experimental
  intervention
  – equal chance or 1 out of 3 chance
• People grasp natural frequencies rather
  than probabilities
                                                30
           Consent Forms
• Describe all interventions
• Merely saying „standard care‟ not helpful
• Remember need to provide reasonable
  information to make a decision
• Dishonest to promise benefit – uncertainty
  justification for study
• Need to ensure non-participation will not
  jeopardize usual services to which entitled
                                            31
      When to Gain Consent
• Gain consent prior to random assignment
• Unethical to indicate allocate by chance
  when already assigned
• If assigned prior to consent, require two
  separate consent forms
• Allocation prior to consent - result in
  biased attrition

                                              32
      Multiple Consent Forms
• Screening for eligibility may require
  consent form
• Children require assent & possible multiple
  consents
• Process assessments may require
  consents from family members, providers
  etc


                                           33
           RCT Providers

• Consents for providers – When are
  consent forms needed?

• Need for Federal-Wide Assurance




                                      34
Incentive Payments to Participants


• Negotiate payments with agencies
  – Clients
  – Providers


• Types of payments



                                     35
           Responsibilities at
          Termination of RCT

• Provision for ongoing care of participants
• Experimental service to control condition
• Feedback & dissemination to agency




                                               36
    Data Safety & Monitoring
• NIH require Board for RCT oversight
• Often 3-4 members – meet quarterly in
  person or via phone
• Report adverse events – also to IRB
• Review of adverse events




                                          37
Considerations for Internet RCT
• Consents handled either by mail or via
  Internet
• Monitored or unmonitored interventions
• Are internet communities public or private
  spaces?
• Consent forms – need to specify potential
  risks due to internet

                                               38
Planning An RCT
        Determining Whether to
          Undertake an RCT
• Selecting a site
  – Pipeline of available & willing eligible
    participants
  – Setting prepared & willing to commit &
    support RCT
     • Financially, space, & supervision
     • Others willing to financially support
     • Sustainability of effective intervention


                                                  40
   Negotiating with the Setting

• Top down & bottom up approach

• Honesty in negotiating
  – “You‟ll hardly know we are here”
  – Collaborative partnership



                                       41
               REAL SCORE
•   Respect for providers & clients
•   Establish credibility
•   Acknowledge strengths
•   Low burden
•   Shared ownership – reciprocity
•   Collaborative relationship
•   Offer incentives – be responsive & appreciative
•   Recognize environmental strengths
•   Ensure trust – be sure providers feel heard

                                                      42
     Feasibility & Pilot Studies
• Worthiness, practicality, feasibility &
  acceptability of intervention
• Modification of intervention for new
  population
• Pilot testing recruitment, retention, & data
  collection
• Estimate required sample size

                                                 43
             Defining Treatment /
              Program Manuals
• Specifies:
  – Intervention
  – Standards for evaluating adherence
  – Guidance for training
  – Quality assurance & monitoring standards
  – Facilitation or replication
  – Stimulates dissemination & replication
  (Carroll & Rounsaville, 2008)


                                               44
 Treatment / Program Manuals
• Brief literature review
• Guidelines for establishing therapeutic
  relationship
• Defining & specifying intervention
• Contrast to other approaches
• Specific techniques & content
• Suggestions for sequencing activities
  (Carroll & Rounsville, 2008)


                                            45
 Treatment / Program Manuals
• Suggestions for dealing with specific
  problems
• Implementation issues
• Termination issues
• Qualifications of providers
• Training providers
• Supervising of providers
  (Carroll & Rounsville, 2008)

                                          46
 Treatment / Program Manuals
Deal with structural aspects
 - Caseload
 - Staff qualifications
 - Location/setting
      e.g., space
 - Integration into service setting
     (Carroll & Rounsville, 2008)


                                      47
Criticisms of Treatment Manuals
• Limited application to diversified
  population with complex problems
• Overemphasis on specific techniques –
  rather than competencies
• Focus on technique rather than theory
• Reduction of provider competence
• Lack of applicability to diverse providers
• Designed for highly motivated & single
  problem clients
                                               48
   Adapting Existing Manuals
• Use of qualitative methods
  – Focus groups
  – In-depth interviews
  – Group processes – nominal group process,
    Delphi method, & concept mapping
  – Ethnographic methods




                                               49
         Fidelity Assessment
• Determining whether the intervention was
  conducted as planned and is consistent with
  service or program elements delineated in
  manual, including structures & goals
• Fidelity measure
     – scale or tool assessing adequacy of
  implementation of service or program
     - means to quantify degree to which program
  service elements or services are implemented

                                               50
       Leakage Assessment

• Assesses degree of contamination

• Captures degree to which participants in
  control condition receive services planned
  only for experimental intervention



                                           51
           Developing & Piloting
           Fidelity Assessment
•   Self report measures
•   Chart reviews
•   Observations
•   Data extraction from billing forms
•   Service logs
•   Video tapping
•   Administrative data

                                         52
          Steps in Developing
           a Fidelity Measure
• Define purpose of fidelity scale
• Assess degree of model development
• Identify model dimensions
• Determine if appropriate fidelity scales
  already exist
• Formulate fidelity scale plan
• Develop items
• Develop response scale points
                                             53
            Steps in Developing
             a Fidelity Measure
•   Choose data collection sources & methods
•   Determine item order
•   Develop data collection protocol
•   Train interviewers / raters
•   Pilot Scale
•   Assess psychometric properties
•   Determine scoring & weighting of items
(Bond et al, Nov 2000)
                                           54
NIH Exploratory Research Grants
             (R34)
    R34 Research Mechanism
• Purpose
   – to evaluate feasibility, tolerability, acceptability
  & safety of novel approaches to improving
  mental health & modifying health risk behavior
  - to obtain preliminary data needed as
  prerequisite to efficacy or effectiveness
  intervention or service study

Key purpose - data for larger scale (R01) study
                                                       56
      R34 Research Objectives
         Relevant to RCTs
• Development & pilot testing new or
  adapted intervention
  – Examples
    • Develop, adapt, or revise intervention for different
      target population
    • Testing & refining intervention manual
    • Development or adaptation of measures,
        – e.g., provider competency, adherence to protocol,
          implementation fidelity measures
    • Pilot test of efficacy trial
                                                              57
      R34 Research Objectives
         Relevant to RCTs
• Adaptation & pilot testing for effectiveness
  – process of moving from efficacy research to
   effectiveness research
     • Feasibility studies to assess parameters for
       conducting efficacy intervention in “real world
       service environment”
     • Standardization of research instruments
     • Studies to develop & standardize training
       protocols, supervisory standards, or
       implementation of fidelity procedures

                                                         58
    Example of Process of Adapting
        Effective Intervention
• Use of qualitative interviews with participants &
  social supports to assess needs & role of mental
  illness for specific cultural group
• Use Advisory Board
    – Logic Model Process
    – Identify & prioritize determinants based on qualitative
      data
•   Review of past & current existing programs
•   Develop intervention plan & theory
•   Focus group assessment of intervention plan
•   Develop process & outcome plan

                                                            59
        Example of Developing
         Intervention for RCT
• Adding criminogenic component to
  multifaceted biopsychosocial treatment
  model for mental ill offenders in prison

• Criminogenic component based on CBT –
  cognitive restructuring

• Need to assess criminal thoughts &
  attitudes of mentally ill offenders

                                             60
       Example of Developing
        Intervention for RCT
• Use 2 existing measures to assess these
  factors that have been used with non-
  mentally ill offenders
• Determine if factor structure for these
  measures same as for non-mentally ill
• Cluster analyses of these two measures
  and DSM disorders for implications for
  structuring criminogenic component
                                            61
              Example of Refining
              Existing Intervention
• New conceptual model with measuring service context
  variables & moderator variable to determining effects on
  outcomes
• Quantitatively assessing conceptual model
   – Test utility of model
   – Estimate effect sizes of predictor variables & outcomes
• Qualitative component – examine experiences of
  implementing intervention & identify factors that promote
  or inhibit effectiveness of intervention
• Refine model based on results & more definitively
  operationalize service context variable & implementation
  of intervention

                                                               62
Developing Conceptual
Foundations for RCTs
 Conceptual Foundations for RCTs
• Theories for RCTs support explanatory
  models of process & outcomes
• Frameworks that delineate role of
  intervention in affecting change
• Empirical base justifies change over time –
  expected timeframe for specific levels of
  change


                                           64
Common Theories for Interventions
• Cognitive Behavioral Theory
• Social learning theory
• Stress, Coping, & Adaptation
• Social Support
• Social Capital
• Health Beliefs
• Theory of Planned Behavior/Theory of
  Reasoned Action
• Transtheoretical Model of Change

                                         65
  Stronger Theoretical Models
• Mediators
  – variable that is hypothesized to help make
    change happen
  – Conceptual link in the middle of cause &
    effect argument
  – Sometimes referred to as intervening or
    process variable
  – Mechanisms of change in outcomes
    associated with the intervention & precede
    outcome
                                                 66
Mediation Diagram


       MED




 X            Y




                    67
                Mediation
• Step 1 Show intervention variable is
  correlated with outcome
• Step 2 Show intervention variable is
  correlated with mediator
• Step 3 Show mediator affects outcome
• Step 4 To establish mediation, effect of
  intervention on outcome, controlling for
  mediator should equal 0 or greatly
  reduced (partial mediation)
                                             68
  Stronger Theoretical Models
• Moderators
  – Variable that interacts with intervention in
    such a way that interaction variable has a
    different effect or strength of the effect on the
    outcome
  – Moderators alter strength of causal
    relationship
     • e.g., psychotherapy may reduce depression more
       for men than women or high risk youths do better
       on outcomes
                                                          69
  Stronger Theoretical Models
• Moderators associated with service
  context &/or service population e.g., Police
  intervention program for persons with
  mental illness (Crisis Intervention Team)
  moderated by available MH treatment
  programs in community
• Moderator analysis assess external
  validity – answers question of how
  universal is causal effect

                                             70
Moderator Diagram


       MOD




 X            Y




                    71
       Experimental Intervention
         Compared to What?
• Essence of RCT question is “Compared to
  What?”
• Need to consider what usual care is – TAU
• If no usual care, nothing or waitlist appropriate
  comparison
• Benign intervention, such as supportive or
  educational interventions, not expected to have
  deep or lasting impact on outcome measure
• Control condition used to control for attention or
  placebo effect as could affect outcome

                                                       72
   Examples of Comparisons
• Consumer Case Management Teams
  compared to Non-Consumer Case
  Management Teams
  – Outcomes essentially same for both teams
  – Limitation –Could be both team equally
    ineffective – with no control condition this
    alternative hypothesis could not be ruled out



                                                    73
    Examples of Comparison
• Problem–Solving Educational intervention
  compared to depression education
  materials & referral for antidepressant
  medication among elderly with depressive
  symptoms receiving home health care for
  their medical problems
  – Standard care alone not felt to be strong
    comparison to determine effectiveness
  – Limits external validity of study results
                                                74
Designing RCTs
     Be Sure Design Matches The
       Policy Relevant Question
Is Mental Health Treatment Court (MHTC) more
   effective than usual adversarial court processing in
   reducing criminal activity and improving their
   psychosocial functioning for adults with mental
   illness involved in the criminal justice system?
   - MHTC part of large movement of “therapeutic
   jurisprudence”
   _ designed to reduce arrests & jail time by addressing
   psychosocial needs of indiv.
   - MHTC involves cooperative agreements between
   criminal justice & MH treatment providers


                                                            76
    Be Sure Design Matches The
      Policy Relevant Question
- designed to reduce arrests & jail time by addressing
  psychosocial needs of indiv.
  - MHTC involves cooperative agreements between
  criminal justice & MH treatment providers
  - Indivs. served poor tx compliance lead to erratic
  behaviors, but safely be diverted from criminal justice
  system
  - ACT is EBP for helping persons with SMI
  - MHTC incorporated an ACT approach
  - Adversarial court processing received usual MH
  services
                                                            77
 What Design Captures the Relevant
 Policy Question For Case Example?

• Design employed:

         R: MH Court +ACT
         R: TAU Court + TAU MH services




                                      78
  Problems with Design Employed
• Study provided most positive evidence of
  MH Courts
• However, was it MH Court or ACT?
• Or, interaction of the two?
• Do not know




                                             79
Design Required to Answer Policy
       Relevant Question


   R: MHC + TAU MH services
   R: MHC + ACT
   R: TAU court + ACT
   R: TAU court + TAU MH services
            (Control Condition)


                                    80
 Design Required to Answer Policy
        Relevant Question
• Policy relevant design provides attribution
  to outcome of
     • Court
     • ACT
     • Interaction of the two
• However require sufficient sample size of
  eligible & willing participants


                                                81
  Controlling for Contamination
• Referred to as blurring of conditions, drift,
  or treatment dilution
• Ways contamination may occur:
  – Control condition participants gain benefit
    from experimental condition
  – Experimental condition drifts toward control
    condition
  – Control condition drifts toward experimental

                                                   82
           Examples of Drift
• Caused by either provider or client behavior
• Drift between ACT & individual intensive case
  managers
  – Individual case managers from same agency began
    functioning as a team
  – Resulted in blurring of conditions
• Clients sharing same waiting room
• Behavioral anger management intervention with
  homework assignments taking place in a
  residential treatment setting

                                                      83
          Potential Solutions
• Different locations
• Different times of operation
• Different providers delivering the exp. &
  control interventions
But these solutions raise additional
  confounds
      - result in different types of clients
      - providers with different qualifications
           & experience
                                                  84
         Design Consideration to
         Address Contamination

•   Provider qualifications
•   Training providers
•   Ongoing support to providers
•   Monitoring of interventions




                                   85
             Changes in
      Intervention Environment
• History internal threat
• Policy change may affect one or both
  conditions
• Becomes a confound when interacts with
  one condition differentially to outcome
• One proposed strategy is nested RCT in a
  longitudinal quasi experimental design
• Another is conducting continuing ongoing
  process assessment
                                         86
            Biased Attrition
• Biased attrition to one condition or the
  other is real threat to internal validity

• E.g., Concern of biased attrition in control
  condition of ACT homeless jail study

• Loss also reduces power

                                                 87
     Potential Design Solutions to
                Attrition
• Protocol designed to engage & keep participants
  engaged
• Pre-randomization introductory phase absorbing early
  stage attrition
   – Trade off – reduced external validity
• Increased incentive payment at points expect greater
  loss
   – e.g., Exit from prison
• More participants assigned to condition with greater
  anticipated loss
• Statistical procedures – require anticipation to obtain
  necessary data

                                                            88
            Randomization
• Usually equal assignment to all conditions

• Unequal assignment requires justification

• Computer randomization preferred to
  physical manipulation


                                              89
        When to Use Stratified
          Randomization?
• Randomization may not ensure equal
  proportions across conditions
• When sample size small
  – e.g., less than 100
• Subpopulation small
  – e.g., less than 20%
• Bigger problem of small samples – low
  power
• Increases complexity
                                          90
           When to Use Cluster
            Randomization?
• Control for contamination
  – e.g., same providers delivering two interventions
• Efficiency – everyone in intervention served in
  one location
• Cost & time-efficient – can‟t feasibly gain
  consent from everyone & change in policy or
  guideline
• Limitation – requires larger sample size to
  maintain power

                                                        91
         When to Use Blocked
          Randomization?

• When employing group interventions

• Control flow into different conditions

• Assignments made for smaller units, such
  as in blocks of 4, 6, etc.

                                           92
                  Blinding
• Controls for potential bias, specifically
  reactivity of client &/or provider

• Difficult to do in community–based
  psychosocial interventions

• Possibly blind data collectors

                                              93
    Randomization in Practice
• Assignment occur after consent & baseline
  assessment completed
• Random assignment not in hands of
  providers or even research workers
• Procedures for random assignment
  centrally controlled to protect against
  subversion


                                          94
   How to Design Recruitment &
        Sampling Strategy

• Need to demonstrate can consent &
  maintain sufficient sample size for analysis

• Need to determine at what point in pipeline
  feasible & conceptually justified to recruit



                                             95
       What Inclusion Criteria to
              Consider?
• Need to operationally define inclusion &
  exclusion criteria
• Consideration & implications of criteria,
  e.g., new intakes or current clients, i.e.,
  current testing TAU + exp. int.
• Consideration of age, diagnoses,
  language, & geography


                                                96
       What Exclusion Criteria
           to Consider?
• Vulnerable populations

• Co-morbid or specific disorders

• Specific system status levels

• Frequently, no exclusion criteria

                                      97
   Considering Sample Method &
       Recruitment Process
• Frequently use consecutive samples

• Combination of purposive, snowball, &
  quota samples

• Agency staff vs. research staff doing
  recruitment

                                          98
     Determining Sample Size
• Determining effect size
  – Prior research – literature
  – Pilot studies
• Estimating attrition
  – Prior studies
  – Pilot studies




                                  99
  Operationalizing Experimental &
       Control Interventions
• Need to clearly specify all conditions
  – Experimental interventions – manualized/tool
    kit – clearly specified intervention
  – Justify that exp. & control conditions truly
    differ
  – Need to operationalize TAU & benign
    interventions
  – If exp. longer or more intense (dosage) than
    control – time &/or amount may be variables
    effecting outcome
                                               100
Outcome Measures & Data Points
• Need psychometrically sound measures –
  unreliable measures reduce power
• Valid measures for sample
• Sensitive to capture change in short time
  frame
• Some concepts unlikely to change in short
  time frame
• Justify time period for data points
                                         101
  Approaches to Data Analysis
• Expected to use Intent-to-treat analysis
• Avoid temptation of eliminating participants
  receiving limited service
• Dosage effect variables – fidelity, compliance,
  adherence, & engagement
• Carefully conceptualize dosage effect so do not
  substitute for main independent variable
• With enough data points can estimate missing
  data
• Statistical consultant early in design process
                                                102
        Concerns with RCTs
• Highly selective samples
• Preferences interacting with actual service
  delivered
• Complex interventions not accounting for
  accumulative effects of all service
  components



                                           103
  Alternative Designs to RCTs
• Fixed adaptive designs
  – randomly assigned to condition, but progress
   through intervention determined by intensity
   of treatment need
• Randomized adaptive designs
  – changes in service condition are done by
   randomization to choices of participant or
   provider


                                                104
  Alternative Designs to RCTs
• Encouragement or randomized consent
  trials
  – Encouraged to participate in one service
    option or other, but constricted to the selected
    option
• Randomized preferences
  – Participants decide whether they will be
    randomized or choose their service option

                                                  105
Implementing Randomized
        Designs
     Preparing Setting for RCT
• Inform setting with time for preparation, but not
  so far in advance that forgotten
• Research & Agency jointly decide on how &
  when to inform personnel
• Jointly present RCT with administrators & staff
• Need to sell RCT on benefits to setting,
  providers, & clients
• Don‟t oversell what can‟t be delivered
• Understand provider‟s perspective
                                                      107
     Preparing Setting for RCT
• Sensitivity to language & examples employ
• Turn lack of clarity into an advantage
• Anticipate questions & issues & raise them first
• Address random assignment in straightforward
  manner
• Try to counter negative momentum
• Positive frame of mind critical – “you need them
  more than they need you”

                                                 108
        Tracking Participants
• At enrollment participant complete locator
  form – working document of all info to help
  find someone including:
  – Demographic & identifying info
  – Relatives, info from multiple people at
    different locations
  – Professional contacts for contact info
  – Incidental contacts
     • e.g., where one goes when out of money, or
       hungry, or where one sleeps when homeless
                                                    109
        Tracking Participants
• Working document
  – update every time contact participant
   - indicate helpful & unhelpful info
   - offer incentive for participant to contact
       researchers with change of info
• Computerized system to generate timely
  lists for follow-up data points

                                              110
          Monitor Recruitment
• Use track system, monitor recruitment to ensure
  accruing sample to meet timely projections
• Big push late in study resulting in non-
  completers of intervention &/or outcomes
• Relying totally on providers is usually ineffective
• Creative means to control recruitment within
  confidentiality & legal policies
• Remember providers usually do not want
  responsibility for recruitment

                                                    111
          Referral Process
• Providers make referrals, but best they not
  do eligibility determinations or consents
• Providers obtain Release of Information
  form from potential eligible participants
• Provider referrals based on easily
  observable or obtainable criteria (using
  system categorization), & casting wide net
  – Lessens burden on providers
  – Providers more likely to do
                                           112
   Ensuring Participant Retention
            in Research
• Collect complete locator info at study
  entrance
• Inform participants when they will be
  followed up
• Review locator information at subsequent
  data collection points
• Offer adequate incentives
• Employ effective research data collectors
                                          113
   Ensuring Participant Retention
            in Research
• Document all follow-up activities in detail
• Exploit contact information obtained
• Reasonably accommodate participant for
  follow up data collection
• Allocate enough resources for travel
• Allow ample time for tracking down
  participants

                                                114
        Ensuring Participant
     Engagement in Intervention
• Communicating importance of intervention
• Outlining benefits & expectations of participants
• Making minor modifications
  – e.g., reducing # of sessions if too many dropping out
• Training providers (both conditions) in
  engagement, retention, & relationship building
• Building trust
• Incorporating outreach efforts as part of
  intervention
• Novel thinking
  – e.g., giving up professional offices
                                                        115
          Qualifications & Training
                 of Providers
• Equality of qualifications for all conditions –
  otherwise confounding
• Training of Experimental Providers
   –   Human subject protections
   –   Overview & purpose of RCT
   –   Conceptual basis of RCT
   –   Design of RCT
   –   Appealing argument for need for random assignment
   –   Operation & implementation of RCT

                                                       116
Training Experimental Providers
•   Introduction to intervention
•   Program philosophy
•   Program goals & principles
•   Practice experience delivering intervention
•   Role modeling with target population
•   Review manual/toolkit, etc.
•   Using existing training material if available
•   Consider hiring trainer – control for potential bias
•   Consider on-going support, coaching, booster
    sessions

                                                      117
Training Experimental Providers
• Training involves engaging, teaching, &
  supporting in performance of intervention
• Training ensures fidelity of intervention
• Provision for new hires
• Supervisors need to be trained
• Supervision/monitoring of exper. providers
  best done by those with investment in
  RCT- e.g., research staff
                                          118
         Training & Monitoring
           Control Condition
• Less involved than experimental condition
• Training in eligibility determination
• Training in completion of fidelity / leakage
  forms
• Researcher monitor fidelity / leakage
  forms to take corrective action



                                             119
           Training & Supervising
               Research Staff
•   Rationale for RCT
•   Overview of RCT
•   Human Subject Protection
•   Recruitment procedures
•   Randomization process
•   Review of all data collection forms
•   Training in experimental intervention if providing
    ongoing support & technical assistance

                                                    120
            Fidelity Assessment
• Time points – developmental & mature phases
  of intervention
• Provider & client perspective
• Data sources
  –   Billing data
  –   Treatment/activity logs
  –   Attendance records
  –   Site visits
  –   Ethnographic methods
       • e.g., shadowing


                                                121
 Assessing Environmental Context
• Systematically tracking organizational
  changes
  – e.g., policy, eligibility requirements by dates
• Use of quantitative & qualitative methods
• Importance as participants will be served
  over time – not all served at same point in
  time


                                                      122
      Implementation Disaster
• To test effectiveness of self-help for persons
  with severe mental illness
• Roster of 1185 clients from an urban CMHC who
  received tx in past 2 years; 853 met eligibility;
  decreased to 241 due to hosp, participation in
  self help, etc, but 90 consented, completed data,
  & were randomly assigned
• Inclusion criteria:
  – Dx. schiz., schizaffective, or major mood
  – Normal intelligence
  – Not participated in self help

                                                 123
         Implementation Disaster
• Both groups monitored for self help attendance to
  assess contamination
    – reviewed daily sign in sheets of self help group
•   17% of both conditions participated in self help
•   Self help has selective rather than universal appeal
•   Outreach efforts minimally affected participation
•   Self-selection tremendous impact on sample size in self
    help research – not likely to recruit adequate randomized
    sample with no prior exposure to self help from a single
    geographical area
        (Kaufman, Schulberg, & Schooler, 1994)




                                                          124
Final Note
                Final Note
• Generalizability to other service settings
• Sustainability of intervention in research
  setting
• Transparency of reporting RCTs
  – CONSORT: checklist & flow diagram
• If experimental intervention effective, cost
  effectiveness important – but need to
  design at beginning of study, not as an
  after thought
                                               126
127

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:13
posted:4/28/2010
language:English
pages:127