Learning Center
Plans & pricing Sign in
Sign Out

Designing _ Implementing Randomi

VIEWS: 13 PAGES: 127

  • pg 1
									   Designing & Implementing
Randomized Controlled Trials For
Community-Based Psychosocial

           Phyllis Solomon, Ph.D.
       School of Social Policy & Practice
          University of Pennsylvania

                March 17, 2010
         Overview of Workshop
•   Introduction
•   So you think you want to do an RCT?
•   RCT Ethical Considerations
•   Planning an RCT
•   NIH Exploratory Research Grants
•   Developing Conceptual Foundation
•   Designing an RCT
•   Implementing an RCT
•   Generalizing RCT Outcomes

           What is an RCT?
• True experimental design. Participants assigned
  by chance, following consent, to one of at least
  two conditions
• Key features of classic experimental design:
  – Random assignment
     • determines who assigned to which group
  – Pre & post tests
     • outcome measured before & after intervention
  – Control group
     • same experiences as experimental group except
       no exposure to experimental stimulus

         What is an RCT?

• Can have more than two groups
• Sometimes no pre-test measures
• Chance does not necessarily mean equal,
  but known probability

     Psychosocial Interventions
• Psychosocial Intervention – any service,
  program, educational curriculum, or
  workshop whose goal is to produce
  positive outcomes for individuals
  confronted with social &/or behavioral
  issues & challenges
• Community-Based -Conducted in agency
  & social work settings

     Psychosocial Interventions
• Community-based psychosocial
  intervention – reflects impact of
  environmental context in which
  interventions are imbedded, on clients and
  providers & interactions between both/all
• Less control, more complex environmental
  context with participants with multiple

         RCT vs. Evaluation
• Research uses scientific methodology to
  generate generalizable knowledge
• Evaluation uses same methodology but
  primary goal is not for generalizable
• For NIH grants do not use term evaluation

          RCT vs. Evaluation
• In evaluation RCT known as experimental
  study or a randomized field experiment
• Both examine a program or policy
• Both addresses effectiveness & cost
• Evaluation experimental studies closely
  resemble community-based psychosocial
  – literature in this area may be helpful
         RCT vs. Evaluation
• Purpose of RCTs & field experiments may
• RCT – research –generalized knowledge
• Field experiments – evaluation – answer
  local questions – but also policy questions
  of broader application
• Semantic difference

   Psychosocial Community-Based
Interventions = Effectiveness Studies

• Efficacy studies occur under ideal or
  optimum conditions
• Effectiveness studies occur in “real world”
• Efficacy studies greater internal validity
• Effectiveness studies greater external

So You Think You Want
   To Do An RCT?
     Appraising Whether to Move
        Forward with an RCT
Preliminary questions to be addressed before
  moving forward:
• Is the question well justified?
• Is the question an important one to answer?
• Is the question addressing a gap in the
• Is the question an ethical one?
• Is the question posing the correct question?
• Would you fund this RCT?
      Appraising Whether to Move
        Forward with Your RCT
Case Example 1
• Is a 90 day Advanced Practice Nurse-Transitional Care
  Model more effective than usual discharge in improving
  adherence to treatment & quality of life for persons with
  SMI being released from a psychiatric hospital?
   – “hand-off” from hospital to home of SMI linked to
     gaps in delivery of MH services
   – Consequently high rates of rehosp & poor outcomes
   – EBP – Advanced Practice Nurse-Transitional Care
     Model improves outcomes following acute medical
     care discharge for elderly adults with complex
     medical problems

     Appraising Whether to Move
         Forward with RCT
Case Example 1 (continued)
 - Intervention hybrid of case management,
 disease management, & home health care
 - Nurse works with hospital team to develop
 discharge plan & then implement in the
 - Believe adapting this intervention potential to
 be equally successful with adults with SMI being
 discharge from acute hospital

     Appraising Whether to Move
         Forward with RCT
Case Example 2
• Is Multidimensional Treatment Foster Care (MTFC)
  Program more effective in reduction of disruptive
  behaviors than traditional Therapeutic Foster Care (TFC)
  among children in foster care?
  - Instability in foster care placement ranges from 22%-
  - Instability in placement due to child‟s disruptive
  - TFC typically used for children with more demanding
  emotional & behavior needs & has more intensive
  structure & MH services

      Appraising Whether to Move
          Forward with RCT
Case Example 2 (continued)
  - Data on disruptions for TFC sparse but estimated 38%-
  - Limited evidence on TFC effectiveness – most studies
  descriptive, methodologically flawed
  - Lack of clear standards & specification of actual
  implementation of TFC
  -MTFC – manualized intervention with goals to improve
  well-being & reduce disruptions
  - MTFC placement augmented with coordinating an
  array of clinical interventions in family, school, & peer
      Appraising Whether to Move
          Forward with RCT
Case Example 3
Is CBT for adolescents with sickle cell disease (SCD) more
   effective than medical management of the disease in
   increasing coping strategies?
   - adolescents with SCD have a number of adjustment
   difficulties that have received little attention
   - some psychosocial difficulties include stress-
         e.g. decreased coping strategies, lack of knowledge
         of SCD
   - need to promote biological & psychosocial adjustment

    Appraising Whether to Move
        Forward with RCT
Case Example 4
• Is Forensic Assertive Community Treatment
  (FACT) more effective than forensic intensive
  case management (FICM) in a variety of
  psychosocial and clinical outcomes for homeless
  adults with SMI leaving jail?

  - Pop. has multiplicity of needs due to mental
  illness, homelessness, & criminal justice

     Appraising Whether to Move
         Forward with RCT
Case Example 4 (continued)
- cognitive deficits & poor social skills complicate
  ability to coordinate efforts to meet needs
  - FICM single point of planning, monitoring &
  accountability considered beneficial for this pop.
      - FICM specialized ICM
  - FACT –team approach (shared caseload), self
  contained intervention to meet all needs of client
  – includes psychiatrist, case managers, etc.
  - Based on ACT for criminally involved
RCT Ethical Considerations
     RCT Ethical Considerations
•   Appropriate question to ask
•   Who ethically eligible to randomize
•   What ethical comparison
•   How & when to randomize
•   When are providers human subjects
•   What is ethical responsibility at termination

 Justifying the RCT to Doubters
• Want to provide most effective services to clients
• Expectation when treated by a doctor
• RCTs best means to making causal inference
  with high degree of confidence
• Unethical to offer untested intervention
• Not denying better treatment to controls
  – if answer known, there would be no need for study
• Frequently those who receive services
  determined on a haphazard or a biased basis
        Ethical Justification For
• Lack of adequate evidence of
  effectiveness of exp. intervention
• Experimental intervention theoretically
  justified to potentially benefit target pop.
• Uncertainty of effectiveness (equipoise) –
  otherwise no scientific basis for RCT

       Principle of Equipoise

• Substantial degree of uncertainty /
  ambiguity necessary
  – Specific population
  – Setting

       Integration of Practice &
           Research Ethics
• Practice – interventions designed solely to
  enhance well-being of client & has
  reasonable chance of success (Belmont
 Report, 1979)
• Research – activities designed to test
  hypothesis, permit conclusions to be
  drawn, thereby contribute to generalized
  knowledge (Belmont Report, 1979)
• RCTs = Practice & Research

       Integration of Practice &
           Research Ethics
• Practice ethics = human subject
   – may conflict w/ scientific rigor
• Participant deterioration in experimental
  condition results in biased attrition
• Exclusion criteria for clinical reasons –
  reduce external validity

         Ethics of Scientific
        Untested Interventions

• Experimental intervention at least as
  effective as TAU
• Do no harm - even if voluntarily consents
• Risks assessment for participant
  – Extends to others & community-at-large

 Ethics of Selecting Control Group
• Justify no service comparison
  – Gas to no gas
• Waitlist may be justified if agency normally
  has waiting list, or no service offered
• Inert intervention may be justified
• TAU may be most justifiable comparison

               Consent Forms
• Must inform potential participant that will
  receive experimental intervention by
     • i.e., like flipping a coin
• Indicate chance of receiving experimental
  – equal chance or 1 out of 3 chance
• People grasp natural frequencies rather
  than probabilities
           Consent Forms
• Describe all interventions
• Merely saying „standard care‟ not helpful
• Remember need to provide reasonable
  information to make a decision
• Dishonest to promise benefit – uncertainty
  justification for study
• Need to ensure non-participation will not
  jeopardize usual services to which entitled
      When to Gain Consent
• Gain consent prior to random assignment
• Unethical to indicate allocate by chance
  when already assigned
• If assigned prior to consent, require two
  separate consent forms
• Allocation prior to consent - result in
  biased attrition

      Multiple Consent Forms
• Screening for eligibility may require
  consent form
• Children require assent & possible multiple
• Process assessments may require
  consents from family members, providers

           RCT Providers

• Consents for providers – When are
  consent forms needed?

• Need for Federal-Wide Assurance

Incentive Payments to Participants

• Negotiate payments with agencies
  – Clients
  – Providers

• Types of payments

           Responsibilities at
          Termination of RCT

• Provision for ongoing care of participants
• Experimental service to control condition
• Feedback & dissemination to agency

    Data Safety & Monitoring
• NIH require Board for RCT oversight
• Often 3-4 members – meet quarterly in
  person or via phone
• Report adverse events – also to IRB
• Review of adverse events

Considerations for Internet RCT
• Consents handled either by mail or via
• Monitored or unmonitored interventions
• Are internet communities public or private
• Consent forms – need to specify potential
  risks due to internet

Planning An RCT
        Determining Whether to
          Undertake an RCT
• Selecting a site
  – Pipeline of available & willing eligible
  – Setting prepared & willing to commit &
    support RCT
     • Financially, space, & supervision
     • Others willing to financially support
     • Sustainability of effective intervention

   Negotiating with the Setting

• Top down & bottom up approach

• Honesty in negotiating
  – “You‟ll hardly know we are here”
  – Collaborative partnership

               REAL SCORE
•   Respect for providers & clients
•   Establish credibility
•   Acknowledge strengths
•   Low burden
•   Shared ownership – reciprocity
•   Collaborative relationship
•   Offer incentives – be responsive & appreciative
•   Recognize environmental strengths
•   Ensure trust – be sure providers feel heard

     Feasibility & Pilot Studies
• Worthiness, practicality, feasibility &
  acceptability of intervention
• Modification of intervention for new
• Pilot testing recruitment, retention, & data
• Estimate required sample size

             Defining Treatment /
              Program Manuals
• Specifies:
  – Intervention
  – Standards for evaluating adherence
  – Guidance for training
  – Quality assurance & monitoring standards
  – Facilitation or replication
  – Stimulates dissemination & replication
  (Carroll & Rounsaville, 2008)

 Treatment / Program Manuals
• Brief literature review
• Guidelines for establishing therapeutic
• Defining & specifying intervention
• Contrast to other approaches
• Specific techniques & content
• Suggestions for sequencing activities
  (Carroll & Rounsville, 2008)

 Treatment / Program Manuals
• Suggestions for dealing with specific
• Implementation issues
• Termination issues
• Qualifications of providers
• Training providers
• Supervising of providers
  (Carroll & Rounsville, 2008)

 Treatment / Program Manuals
Deal with structural aspects
 - Caseload
 - Staff qualifications
 - Location/setting
      e.g., space
 - Integration into service setting
     (Carroll & Rounsville, 2008)

Criticisms of Treatment Manuals
• Limited application to diversified
  population with complex problems
• Overemphasis on specific techniques –
  rather than competencies
• Focus on technique rather than theory
• Reduction of provider competence
• Lack of applicability to diverse providers
• Designed for highly motivated & single
  problem clients
   Adapting Existing Manuals
• Use of qualitative methods
  – Focus groups
  – In-depth interviews
  – Group processes – nominal group process,
    Delphi method, & concept mapping
  – Ethnographic methods

         Fidelity Assessment
• Determining whether the intervention was
  conducted as planned and is consistent with
  service or program elements delineated in
  manual, including structures & goals
• Fidelity measure
     – scale or tool assessing adequacy of
  implementation of service or program
     - means to quantify degree to which program
  service elements or services are implemented

       Leakage Assessment

• Assesses degree of contamination

• Captures degree to which participants in
  control condition receive services planned
  only for experimental intervention

           Developing & Piloting
           Fidelity Assessment
•   Self report measures
•   Chart reviews
•   Observations
•   Data extraction from billing forms
•   Service logs
•   Video tapping
•   Administrative data

          Steps in Developing
           a Fidelity Measure
• Define purpose of fidelity scale
• Assess degree of model development
• Identify model dimensions
• Determine if appropriate fidelity scales
  already exist
• Formulate fidelity scale plan
• Develop items
• Develop response scale points
            Steps in Developing
             a Fidelity Measure
•   Choose data collection sources & methods
•   Determine item order
•   Develop data collection protocol
•   Train interviewers / raters
•   Pilot Scale
•   Assess psychometric properties
•   Determine scoring & weighting of items
(Bond et al, Nov 2000)
NIH Exploratory Research Grants
    R34 Research Mechanism
• Purpose
   – to evaluate feasibility, tolerability, acceptability
  & safety of novel approaches to improving
  mental health & modifying health risk behavior
  - to obtain preliminary data needed as
  prerequisite to efficacy or effectiveness
  intervention or service study

Key purpose - data for larger scale (R01) study
      R34 Research Objectives
         Relevant to RCTs
• Development & pilot testing new or
  adapted intervention
  – Examples
    • Develop, adapt, or revise intervention for different
      target population
    • Testing & refining intervention manual
    • Development or adaptation of measures,
        – e.g., provider competency, adherence to protocol,
          implementation fidelity measures
    • Pilot test of efficacy trial
      R34 Research Objectives
         Relevant to RCTs
• Adaptation & pilot testing for effectiveness
  – process of moving from efficacy research to
   effectiveness research
     • Feasibility studies to assess parameters for
       conducting efficacy intervention in “real world
       service environment”
     • Standardization of research instruments
     • Studies to develop & standardize training
       protocols, supervisory standards, or
       implementation of fidelity procedures

    Example of Process of Adapting
        Effective Intervention
• Use of qualitative interviews with participants &
  social supports to assess needs & role of mental
  illness for specific cultural group
• Use Advisory Board
    – Logic Model Process
    – Identify & prioritize determinants based on qualitative
•   Review of past & current existing programs
•   Develop intervention plan & theory
•   Focus group assessment of intervention plan
•   Develop process & outcome plan

        Example of Developing
         Intervention for RCT
• Adding criminogenic component to
  multifaceted biopsychosocial treatment
  model for mental ill offenders in prison

• Criminogenic component based on CBT –
  cognitive restructuring

• Need to assess criminal thoughts &
  attitudes of mentally ill offenders

       Example of Developing
        Intervention for RCT
• Use 2 existing measures to assess these
  factors that have been used with non-
  mentally ill offenders
• Determine if factor structure for these
  measures same as for non-mentally ill
• Cluster analyses of these two measures
  and DSM disorders for implications for
  structuring criminogenic component
              Example of Refining
              Existing Intervention
• New conceptual model with measuring service context
  variables & moderator variable to determining effects on
• Quantitatively assessing conceptual model
   – Test utility of model
   – Estimate effect sizes of predictor variables & outcomes
• Qualitative component – examine experiences of
  implementing intervention & identify factors that promote
  or inhibit effectiveness of intervention
• Refine model based on results & more definitively
  operationalize service context variable & implementation
  of intervention

Developing Conceptual
Foundations for RCTs
 Conceptual Foundations for RCTs
• Theories for RCTs support explanatory
  models of process & outcomes
• Frameworks that delineate role of
  intervention in affecting change
• Empirical base justifies change over time –
  expected timeframe for specific levels of

Common Theories for Interventions
• Cognitive Behavioral Theory
• Social learning theory
• Stress, Coping, & Adaptation
• Social Support
• Social Capital
• Health Beliefs
• Theory of Planned Behavior/Theory of
  Reasoned Action
• Transtheoretical Model of Change

  Stronger Theoretical Models
• Mediators
  – variable that is hypothesized to help make
    change happen
  – Conceptual link in the middle of cause &
    effect argument
  – Sometimes referred to as intervening or
    process variable
  – Mechanisms of change in outcomes
    associated with the intervention & precede
Mediation Diagram


 X            Y

• Step 1 Show intervention variable is
  correlated with outcome
• Step 2 Show intervention variable is
  correlated with mediator
• Step 3 Show mediator affects outcome
• Step 4 To establish mediation, effect of
  intervention on outcome, controlling for
  mediator should equal 0 or greatly
  reduced (partial mediation)
  Stronger Theoretical Models
• Moderators
  – Variable that interacts with intervention in
    such a way that interaction variable has a
    different effect or strength of the effect on the
  – Moderators alter strength of causal
     • e.g., psychotherapy may reduce depression more
       for men than women or high risk youths do better
       on outcomes
  Stronger Theoretical Models
• Moderators associated with service
  context &/or service population e.g., Police
  intervention program for persons with
  mental illness (Crisis Intervention Team)
  moderated by available MH treatment
  programs in community
• Moderator analysis assess external
  validity – answers question of how
  universal is causal effect

Moderator Diagram


 X            Y

       Experimental Intervention
         Compared to What?
• Essence of RCT question is “Compared to
• Need to consider what usual care is – TAU
• If no usual care, nothing or waitlist appropriate
• Benign intervention, such as supportive or
  educational interventions, not expected to have
  deep or lasting impact on outcome measure
• Control condition used to control for attention or
  placebo effect as could affect outcome

   Examples of Comparisons
• Consumer Case Management Teams
  compared to Non-Consumer Case
  Management Teams
  – Outcomes essentially same for both teams
  – Limitation –Could be both team equally
    ineffective – with no control condition this
    alternative hypothesis could not be ruled out

    Examples of Comparison
• Problem–Solving Educational intervention
  compared to depression education
  materials & referral for antidepressant
  medication among elderly with depressive
  symptoms receiving home health care for
  their medical problems
  – Standard care alone not felt to be strong
    comparison to determine effectiveness
  – Limits external validity of study results
Designing RCTs
     Be Sure Design Matches The
       Policy Relevant Question
Is Mental Health Treatment Court (MHTC) more
   effective than usual adversarial court processing in
   reducing criminal activity and improving their
   psychosocial functioning for adults with mental
   illness involved in the criminal justice system?
   - MHTC part of large movement of “therapeutic
   _ designed to reduce arrests & jail time by addressing
   psychosocial needs of indiv.
   - MHTC involves cooperative agreements between
   criminal justice & MH treatment providers

    Be Sure Design Matches The
      Policy Relevant Question
- designed to reduce arrests & jail time by addressing
  psychosocial needs of indiv.
  - MHTC involves cooperative agreements between
  criminal justice & MH treatment providers
  - Indivs. served poor tx compliance lead to erratic
  behaviors, but safely be diverted from criminal justice
  - ACT is EBP for helping persons with SMI
  - MHTC incorporated an ACT approach
  - Adversarial court processing received usual MH
 What Design Captures the Relevant
 Policy Question For Case Example?

• Design employed:

         R: MH Court +ACT
         R: TAU Court + TAU MH services

  Problems with Design Employed
• Study provided most positive evidence of
  MH Courts
• However, was it MH Court or ACT?
• Or, interaction of the two?
• Do not know

Design Required to Answer Policy
       Relevant Question

   R: MHC + TAU MH services
   R: MHC + ACT
   R: TAU court + ACT
   R: TAU court + TAU MH services
            (Control Condition)

 Design Required to Answer Policy
        Relevant Question
• Policy relevant design provides attribution
  to outcome of
     • Court
     • ACT
     • Interaction of the two
• However require sufficient sample size of
  eligible & willing participants

  Controlling for Contamination
• Referred to as blurring of conditions, drift,
  or treatment dilution
• Ways contamination may occur:
  – Control condition participants gain benefit
    from experimental condition
  – Experimental condition drifts toward control
  – Control condition drifts toward experimental

           Examples of Drift
• Caused by either provider or client behavior
• Drift between ACT & individual intensive case
  – Individual case managers from same agency began
    functioning as a team
  – Resulted in blurring of conditions
• Clients sharing same waiting room
• Behavioral anger management intervention with
  homework assignments taking place in a
  residential treatment setting

          Potential Solutions
• Different locations
• Different times of operation
• Different providers delivering the exp. &
  control interventions
But these solutions raise additional
      - result in different types of clients
      - providers with different qualifications
           & experience
         Design Consideration to
         Address Contamination

•   Provider qualifications
•   Training providers
•   Ongoing support to providers
•   Monitoring of interventions

             Changes in
      Intervention Environment
• History internal threat
• Policy change may affect one or both
• Becomes a confound when interacts with
  one condition differentially to outcome
• One proposed strategy is nested RCT in a
  longitudinal quasi experimental design
• Another is conducting continuing ongoing
  process assessment
            Biased Attrition
• Biased attrition to one condition or the
  other is real threat to internal validity

• E.g., Concern of biased attrition in control
  condition of ACT homeless jail study

• Loss also reduces power

     Potential Design Solutions to
• Protocol designed to engage & keep participants
• Pre-randomization introductory phase absorbing early
  stage attrition
   – Trade off – reduced external validity
• Increased incentive payment at points expect greater
   – e.g., Exit from prison
• More participants assigned to condition with greater
  anticipated loss
• Statistical procedures – require anticipation to obtain
  necessary data

• Usually equal assignment to all conditions

• Unequal assignment requires justification

• Computer randomization preferred to
  physical manipulation

        When to Use Stratified
• Randomization may not ensure equal
  proportions across conditions
• When sample size small
  – e.g., less than 100
• Subpopulation small
  – e.g., less than 20%
• Bigger problem of small samples – low
• Increases complexity
           When to Use Cluster
• Control for contamination
  – e.g., same providers delivering two interventions
• Efficiency – everyone in intervention served in
  one location
• Cost & time-efficient – can‟t feasibly gain
  consent from everyone & change in policy or
• Limitation – requires larger sample size to
  maintain power

         When to Use Blocked

• When employing group interventions

• Control flow into different conditions

• Assignments made for smaller units, such
  as in blocks of 4, 6, etc.

• Controls for potential bias, specifically
  reactivity of client &/or provider

• Difficult to do in community–based
  psychosocial interventions

• Possibly blind data collectors

    Randomization in Practice
• Assignment occur after consent & baseline
  assessment completed
• Random assignment not in hands of
  providers or even research workers
• Procedures for random assignment
  centrally controlled to protect against

   How to Design Recruitment &
        Sampling Strategy

• Need to demonstrate can consent &
  maintain sufficient sample size for analysis

• Need to determine at what point in pipeline
  feasible & conceptually justified to recruit

       What Inclusion Criteria to
• Need to operationally define inclusion &
  exclusion criteria
• Consideration & implications of criteria,
  e.g., new intakes or current clients, i.e.,
  current testing TAU + exp. int.
• Consideration of age, diagnoses,
  language, & geography

       What Exclusion Criteria
           to Consider?
• Vulnerable populations

• Co-morbid or specific disorders

• Specific system status levels

• Frequently, no exclusion criteria

   Considering Sample Method &
       Recruitment Process
• Frequently use consecutive samples

• Combination of purposive, snowball, &
  quota samples

• Agency staff vs. research staff doing

     Determining Sample Size
• Determining effect size
  – Prior research – literature
  – Pilot studies
• Estimating attrition
  – Prior studies
  – Pilot studies

  Operationalizing Experimental &
       Control Interventions
• Need to clearly specify all conditions
  – Experimental interventions – manualized/tool
    kit – clearly specified intervention
  – Justify that exp. & control conditions truly
  – Need to operationalize TAU & benign
  – If exp. longer or more intense (dosage) than
    control – time &/or amount may be variables
    effecting outcome
Outcome Measures & Data Points
• Need psychometrically sound measures –
  unreliable measures reduce power
• Valid measures for sample
• Sensitive to capture change in short time
• Some concepts unlikely to change in short
  time frame
• Justify time period for data points
  Approaches to Data Analysis
• Expected to use Intent-to-treat analysis
• Avoid temptation of eliminating participants
  receiving limited service
• Dosage effect variables – fidelity, compliance,
  adherence, & engagement
• Carefully conceptualize dosage effect so do not
  substitute for main independent variable
• With enough data points can estimate missing
• Statistical consultant early in design process
        Concerns with RCTs
• Highly selective samples
• Preferences interacting with actual service
• Complex interventions not accounting for
  accumulative effects of all service

  Alternative Designs to RCTs
• Fixed adaptive designs
  – randomly assigned to condition, but progress
   through intervention determined by intensity
   of treatment need
• Randomized adaptive designs
  – changes in service condition are done by
   randomization to choices of participant or

  Alternative Designs to RCTs
• Encouragement or randomized consent
  – Encouraged to participate in one service
    option or other, but constricted to the selected
• Randomized preferences
  – Participants decide whether they will be
    randomized or choose their service option

Implementing Randomized
     Preparing Setting for RCT
• Inform setting with time for preparation, but not
  so far in advance that forgotten
• Research & Agency jointly decide on how &
  when to inform personnel
• Jointly present RCT with administrators & staff
• Need to sell RCT on benefits to setting,
  providers, & clients
• Don‟t oversell what can‟t be delivered
• Understand provider‟s perspective
     Preparing Setting for RCT
• Sensitivity to language & examples employ
• Turn lack of clarity into an advantage
• Anticipate questions & issues & raise them first
• Address random assignment in straightforward
• Try to counter negative momentum
• Positive frame of mind critical – “you need them
  more than they need you”

        Tracking Participants
• At enrollment participant complete locator
  form – working document of all info to help
  find someone including:
  – Demographic & identifying info
  – Relatives, info from multiple people at
    different locations
  – Professional contacts for contact info
  – Incidental contacts
     • e.g., where one goes when out of money, or
       hungry, or where one sleeps when homeless
        Tracking Participants
• Working document
  – update every time contact participant
   - indicate helpful & unhelpful info
   - offer incentive for participant to contact
       researchers with change of info
• Computerized system to generate timely
  lists for follow-up data points

          Monitor Recruitment
• Use track system, monitor recruitment to ensure
  accruing sample to meet timely projections
• Big push late in study resulting in non-
  completers of intervention &/or outcomes
• Relying totally on providers is usually ineffective
• Creative means to control recruitment within
  confidentiality & legal policies
• Remember providers usually do not want
  responsibility for recruitment

          Referral Process
• Providers make referrals, but best they not
  do eligibility determinations or consents
• Providers obtain Release of Information
  form from potential eligible participants
• Provider referrals based on easily
  observable or obtainable criteria (using
  system categorization), & casting wide net
  – Lessens burden on providers
  – Providers more likely to do
   Ensuring Participant Retention
            in Research
• Collect complete locator info at study
• Inform participants when they will be
  followed up
• Review locator information at subsequent
  data collection points
• Offer adequate incentives
• Employ effective research data collectors
   Ensuring Participant Retention
            in Research
• Document all follow-up activities in detail
• Exploit contact information obtained
• Reasonably accommodate participant for
  follow up data collection
• Allocate enough resources for travel
• Allow ample time for tracking down

        Ensuring Participant
     Engagement in Intervention
• Communicating importance of intervention
• Outlining benefits & expectations of participants
• Making minor modifications
  – e.g., reducing # of sessions if too many dropping out
• Training providers (both conditions) in
  engagement, retention, & relationship building
• Building trust
• Incorporating outreach efforts as part of
• Novel thinking
  – e.g., giving up professional offices
          Qualifications & Training
                 of Providers
• Equality of qualifications for all conditions –
  otherwise confounding
• Training of Experimental Providers
   –   Human subject protections
   –   Overview & purpose of RCT
   –   Conceptual basis of RCT
   –   Design of RCT
   –   Appealing argument for need for random assignment
   –   Operation & implementation of RCT

Training Experimental Providers
•   Introduction to intervention
•   Program philosophy
•   Program goals & principles
•   Practice experience delivering intervention
•   Role modeling with target population
•   Review manual/toolkit, etc.
•   Using existing training material if available
•   Consider hiring trainer – control for potential bias
•   Consider on-going support, coaching, booster

Training Experimental Providers
• Training involves engaging, teaching, &
  supporting in performance of intervention
• Training ensures fidelity of intervention
• Provision for new hires
• Supervisors need to be trained
• Supervision/monitoring of exper. providers
  best done by those with investment in
  RCT- e.g., research staff
         Training & Monitoring
           Control Condition
• Less involved than experimental condition
• Training in eligibility determination
• Training in completion of fidelity / leakage
• Researcher monitor fidelity / leakage
  forms to take corrective action

           Training & Supervising
               Research Staff
•   Rationale for RCT
•   Overview of RCT
•   Human Subject Protection
•   Recruitment procedures
•   Randomization process
•   Review of all data collection forms
•   Training in experimental intervention if providing
    ongoing support & technical assistance

            Fidelity Assessment
• Time points – developmental & mature phases
  of intervention
• Provider & client perspective
• Data sources
  –   Billing data
  –   Treatment/activity logs
  –   Attendance records
  –   Site visits
  –   Ethnographic methods
       • e.g., shadowing

 Assessing Environmental Context
• Systematically tracking organizational
  – e.g., policy, eligibility requirements by dates
• Use of quantitative & qualitative methods
• Importance as participants will be served
  over time – not all served at same point in

      Implementation Disaster
• To test effectiveness of self-help for persons
  with severe mental illness
• Roster of 1185 clients from an urban CMHC who
  received tx in past 2 years; 853 met eligibility;
  decreased to 241 due to hosp, participation in
  self help, etc, but 90 consented, completed data,
  & were randomly assigned
• Inclusion criteria:
  – Dx. schiz., schizaffective, or major mood
  – Normal intelligence
  – Not participated in self help

         Implementation Disaster
• Both groups monitored for self help attendance to
  assess contamination
    – reviewed daily sign in sheets of self help group
•   17% of both conditions participated in self help
•   Self help has selective rather than universal appeal
•   Outreach efforts minimally affected participation
•   Self-selection tremendous impact on sample size in self
    help research – not likely to recruit adequate randomized
    sample with no prior exposure to self help from a single
    geographical area
        (Kaufman, Schulberg, & Schooler, 1994)

Final Note
                Final Note
• Generalizability to other service settings
• Sustainability of intervention in research
• Transparency of reporting RCTs
  – CONSORT: checklist & flow diagram
• If experimental intervention effective, cost
  effectiveness important – but need to
  design at beginning of study, not as an
  after thought

To top