Get documents about "Designing _ Implementing Randomi
Document Sample
Designing & Implementing
Randomized Controlled Trials For
Community-Based Psychosocial
Interventions
Phyllis Solomon, Ph.D.
Professor
School of Social Policy & Practice
University of Pennsylvania
March 17, 2010
Overview of Workshop
• Introduction
• So you think you want to do an RCT?
• RCT Ethical Considerations
• Planning an RCT
• NIH Exploratory Research Grants
• Developing Conceptual Foundation
• Designing an RCT
• Implementing an RCT
• Generalizing RCT Outcomes
2
Introduction
What is an RCT?
• True experimental design. Participants assigned
by chance, following consent, to one of at least
two conditions
• Key features of classic experimental design:
– Random assignment
• determines who assigned to which group
– Pre & post tests
• outcome measured before & after intervention
– Control group
• same experiences as experimental group except
no exposure to experimental stimulus
4
What is an RCT?
• Can have more than two groups
• Sometimes no pre-test measures
• Chance does not necessarily mean equal,
but known probability
5
Community-Based
Psychosocial Interventions
• Psychosocial Intervention – any service,
program, educational curriculum, or
workshop whose goal is to produce
positive outcomes for individuals
confronted with social &/or behavioral
issues & challenges
• Community-Based -Conducted in agency
& social work settings
6
Community-Based
Psychosocial Interventions
• Community-based psychosocial
intervention – reflects impact of
environmental context in which
interventions are imbedded, on clients and
providers & interactions between both/all
systems
• Less control, more complex environmental
context with participants with multiple
problems
7
RCT vs. Evaluation
• Research uses scientific methodology to
generate generalizable knowledge
• Evaluation uses same methodology but
primary goal is not for generalizable
knowledge
• For NIH grants do not use term evaluation
8
RCT vs. Evaluation
• In evaluation RCT known as experimental
study or a randomized field experiment
• Both examine a program or policy
• Both addresses effectiveness & cost
effectiveness
• Evaluation experimental studies closely
resemble community-based psychosocial
RCTs
– literature in this area may be helpful
9
RCT vs. Evaluation
• Purpose of RCTs & field experiments may
differ
• RCT – research –generalized knowledge
• Field experiments – evaluation – answer
local questions – but also policy questions
of broader application
• Semantic difference
10
Psychosocial Community-Based
Interventions = Effectiveness Studies
• Efficacy studies occur under ideal or
optimum conditions
• Effectiveness studies occur in “real world”
• Efficacy studies greater internal validity
• Effectiveness studies greater external
validity
11
So You Think You Want
To Do An RCT?
Appraising Whether to Move
Forward with an RCT
Preliminary questions to be addressed before
moving forward:
• Is the question well justified?
• Is the question an important one to answer?
• Is the question addressing a gap in the
literature?
• Is the question an ethical one?
• Is the question posing the correct question?
• Would you fund this RCT?
13
Appraising Whether to Move
Forward with Your RCT
Case Example 1
• Is a 90 day Advanced Practice Nurse-Transitional Care
Model more effective than usual discharge in improving
adherence to treatment & quality of life for persons with
SMI being released from a psychiatric hospital?
– “hand-off” from hospital to home of SMI linked to
gaps in delivery of MH services
– Consequently high rates of rehosp & poor outcomes
– EBP – Advanced Practice Nurse-Transitional Care
Model improves outcomes following acute medical
care discharge for elderly adults with complex
medical problems
14
Appraising Whether to Move
Forward with RCT
Case Example 1 (continued)
- Intervention hybrid of case management,
disease management, & home health care
- Nurse works with hospital team to develop
discharge plan & then implement in the
community
- Believe adapting this intervention potential to
be equally successful with adults with SMI being
discharge from acute hospital
15
Appraising Whether to Move
Forward with RCT
Case Example 2
• Is Multidimensional Treatment Foster Care (MTFC)
Program more effective in reduction of disruptive
behaviors than traditional Therapeutic Foster Care (TFC)
among children in foster care?
- Instability in foster care placement ranges from 22%-
56%
- Instability in placement due to child‟s disruptive
behaviors
- TFC typically used for children with more demanding
emotional & behavior needs & has more intensive
structure & MH services
16
Appraising Whether to Move
Forward with RCT
Case Example 2 (continued)
- Data on disruptions for TFC sparse but estimated 38%-
70%
- Limited evidence on TFC effectiveness – most studies
descriptive, methodologically flawed
- Lack of clear standards & specification of actual
implementation of TFC
-MTFC – manualized intervention with goals to improve
well-being & reduce disruptions
- MTFC placement augmented with coordinating an
array of clinical interventions in family, school, & peer
group
17
Appraising Whether to Move
Forward with RCT
Case Example 3
Is CBT for adolescents with sickle cell disease (SCD) more
effective than medical management of the disease in
increasing coping strategies?
- adolescents with SCD have a number of adjustment
difficulties that have received little attention
- some psychosocial difficulties include stress-
processing
e.g. decreased coping strategies, lack of knowledge
of SCD
- need to promote biological & psychosocial adjustment
18
Appraising Whether to Move
Forward with RCT
Case Example 4
• Is Forensic Assertive Community Treatment
(FACT) more effective than forensic intensive
case management (FICM) in a variety of
psychosocial and clinical outcomes for homeless
adults with SMI leaving jail?
- Pop. has multiplicity of needs due to mental
illness, homelessness, & criminal justice
involvement
19
Appraising Whether to Move
Forward with RCT
Case Example 4 (continued)
- cognitive deficits & poor social skills complicate
ability to coordinate efforts to meet needs
- FICM single point of planning, monitoring &
accountability considered beneficial for this pop.
- FICM specialized ICM
- FACT –team approach (shared caseload), self
contained intervention to meet all needs of client
– includes psychiatrist, case managers, etc.
- Based on ACT for criminally involved
20
RCT Ethical Considerations
RCT Ethical Considerations
• Appropriate question to ask
• Who ethically eligible to randomize
• What ethical comparison
• How & when to randomize
• When are providers human subjects
• What is ethical responsibility at termination
22
Justifying the RCT to Doubters
• Want to provide most effective services to clients
• Expectation when treated by a doctor
• RCTs best means to making causal inference
with high degree of confidence
• Unethical to offer untested intervention
• Not denying better treatment to controls
– if answer known, there would be no need for study
• Frequently those who receive services
determined on a haphazard or a biased basis
23
Ethical Justification For
Randomization
• Lack of adequate evidence of
effectiveness of exp. intervention
understudy
• Experimental intervention theoretically
justified to potentially benefit target pop.
• Uncertainty of effectiveness (equipoise) –
otherwise no scientific basis for RCT
24
Principle of Equipoise
• Substantial degree of uncertainty /
ambiguity necessary
– Specific population
– Setting
25
Integration of Practice &
Research Ethics
• Practice – interventions designed solely to
enhance well-being of client & has
reasonable chance of success (Belmont
Report, 1979)
• Research – activities designed to test
hypothesis, permit conclusions to be
drawn, thereby contribute to generalized
knowledge (Belmont Report, 1979)
• RCTs = Practice & Research
26
Integration of Practice &
Research Ethics
• Practice ethics = human subject
protections
– may conflict w/ scientific rigor
• Participant deterioration in experimental
condition results in biased attrition
• Exclusion criteria for clinical reasons –
reduce external validity
27
Ethics of Scientific
Untested Interventions
• Experimental intervention at least as
effective as TAU
• Do no harm - even if voluntarily consents
• Risks assessment for participant
– Extends to others & community-at-large
28
Ethics of Selecting Control Group
• Justify no service comparison
– Gas to no gas
• Waitlist may be justified if agency normally
has waiting list, or no service offered
• Inert intervention may be justified
• TAU may be most justifiable comparison
29
Consent Forms
• Must inform potential participant that will
receive experimental intervention by
chance
• i.e., like flipping a coin
• Indicate chance of receiving experimental
intervention
– equal chance or 1 out of 3 chance
• People grasp natural frequencies rather
than probabilities
30
Consent Forms
• Describe all interventions
• Merely saying „standard care‟ not helpful
• Remember need to provide reasonable
information to make a decision
• Dishonest to promise benefit – uncertainty
justification for study
• Need to ensure non-participation will not
jeopardize usual services to which entitled
31
When to Gain Consent
• Gain consent prior to random assignment
• Unethical to indicate allocate by chance
when already assigned
• If assigned prior to consent, require two
separate consent forms
• Allocation prior to consent - result in
biased attrition
32
Multiple Consent Forms
• Screening for eligibility may require
consent form
• Children require assent & possible multiple
consents
• Process assessments may require
consents from family members, providers
etc
33
RCT Providers
• Consents for providers – When are
consent forms needed?
• Need for Federal-Wide Assurance
34
Incentive Payments to Participants
• Negotiate payments with agencies
– Clients
– Providers
• Types of payments
35
Responsibilities at
Termination of RCT
• Provision for ongoing care of participants
• Experimental service to control condition
• Feedback & dissemination to agency
36
Data Safety & Monitoring
• NIH require Board for RCT oversight
• Often 3-4 members – meet quarterly in
person or via phone
• Report adverse events – also to IRB
• Review of adverse events
37
Considerations for Internet RCT
• Consents handled either by mail or via
Internet
• Monitored or unmonitored interventions
• Are internet communities public or private
spaces?
• Consent forms – need to specify potential
risks due to internet
38
Planning An RCT
Determining Whether to
Undertake an RCT
• Selecting a site
– Pipeline of available & willing eligible
participants
– Setting prepared & willing to commit &
support RCT
• Financially, space, & supervision
• Others willing to financially support
• Sustainability of effective intervention
40
Negotiating with the Setting
• Top down & bottom up approach
• Honesty in negotiating
– “You‟ll hardly know we are here”
– Collaborative partnership
41
REAL SCORE
• Respect for providers & clients
• Establish credibility
• Acknowledge strengths
• Low burden
• Shared ownership – reciprocity
• Collaborative relationship
• Offer incentives – be responsive & appreciative
• Recognize environmental strengths
• Ensure trust – be sure providers feel heard
42
Feasibility & Pilot Studies
• Worthiness, practicality, feasibility &
acceptability of intervention
• Modification of intervention for new
population
• Pilot testing recruitment, retention, & data
collection
• Estimate required sample size
43
Defining Treatment /
Program Manuals
• Specifies:
– Intervention
– Standards for evaluating adherence
– Guidance for training
– Quality assurance & monitoring standards
– Facilitation or replication
– Stimulates dissemination & replication
(Carroll & Rounsaville, 2008)
44
Treatment / Program Manuals
• Brief literature review
• Guidelines for establishing therapeutic
relationship
• Defining & specifying intervention
• Contrast to other approaches
• Specific techniques & content
• Suggestions for sequencing activities
(Carroll & Rounsville, 2008)
45
Treatment / Program Manuals
• Suggestions for dealing with specific
problems
• Implementation issues
• Termination issues
• Qualifications of providers
• Training providers
• Supervising of providers
(Carroll & Rounsville, 2008)
46
Treatment / Program Manuals
Deal with structural aspects
- Caseload
- Staff qualifications
- Location/setting
e.g., space
- Integration into service setting
(Carroll & Rounsville, 2008)
47
Criticisms of Treatment Manuals
• Limited application to diversified
population with complex problems
• Overemphasis on specific techniques –
rather than competencies
• Focus on technique rather than theory
• Reduction of provider competence
• Lack of applicability to diverse providers
• Designed for highly motivated & single
problem clients
48
Adapting Existing Manuals
• Use of qualitative methods
– Focus groups
– In-depth interviews
– Group processes – nominal group process,
Delphi method, & concept mapping
– Ethnographic methods
49
Fidelity Assessment
• Determining whether the intervention was
conducted as planned and is consistent with
service or program elements delineated in
manual, including structures & goals
• Fidelity measure
– scale or tool assessing adequacy of
implementation of service or program
- means to quantify degree to which program
service elements or services are implemented
50
Leakage Assessment
• Assesses degree of contamination
• Captures degree to which participants in
control condition receive services planned
only for experimental intervention
51
Developing & Piloting
Fidelity Assessment
• Self report measures
• Chart reviews
• Observations
• Data extraction from billing forms
• Service logs
• Video tapping
• Administrative data
52
Steps in Developing
a Fidelity Measure
• Define purpose of fidelity scale
• Assess degree of model development
• Identify model dimensions
• Determine if appropriate fidelity scales
already exist
• Formulate fidelity scale plan
• Develop items
• Develop response scale points
53
Steps in Developing
a Fidelity Measure
• Choose data collection sources & methods
• Determine item order
• Develop data collection protocol
• Train interviewers / raters
• Pilot Scale
• Assess psychometric properties
• Determine scoring & weighting of items
(Bond et al, Nov 2000)
54
NIH Exploratory Research Grants
(R34)
R34 Research Mechanism
• Purpose
– to evaluate feasibility, tolerability, acceptability
& safety of novel approaches to improving
mental health & modifying health risk behavior
- to obtain preliminary data needed as
prerequisite to efficacy or effectiveness
intervention or service study
Key purpose - data for larger scale (R01) study
56
R34 Research Objectives
Relevant to RCTs
• Development & pilot testing new or
adapted intervention
– Examples
• Develop, adapt, or revise intervention for different
target population
• Testing & refining intervention manual
• Development or adaptation of measures,
– e.g., provider competency, adherence to protocol,
implementation fidelity measures
• Pilot test of efficacy trial
57
R34 Research Objectives
Relevant to RCTs
• Adaptation & pilot testing for effectiveness
– process of moving from efficacy research to
effectiveness research
• Feasibility studies to assess parameters for
conducting efficacy intervention in “real world
service environment”
• Standardization of research instruments
• Studies to develop & standardize training
protocols, supervisory standards, or
implementation of fidelity procedures
58
Example of Process of Adapting
Effective Intervention
• Use of qualitative interviews with participants &
social supports to assess needs & role of mental
illness for specific cultural group
• Use Advisory Board
– Logic Model Process
– Identify & prioritize determinants based on qualitative
data
• Review of past & current existing programs
• Develop intervention plan & theory
• Focus group assessment of intervention plan
• Develop process & outcome plan
59
Example of Developing
Intervention for RCT
• Adding criminogenic component to
multifaceted biopsychosocial treatment
model for mental ill offenders in prison
• Criminogenic component based on CBT –
cognitive restructuring
• Need to assess criminal thoughts &
attitudes of mentally ill offenders
60
Example of Developing
Intervention for RCT
• Use 2 existing measures to assess these
factors that have been used with non-
mentally ill offenders
• Determine if factor structure for these
measures same as for non-mentally ill
• Cluster analyses of these two measures
and DSM disorders for implications for
structuring criminogenic component
61
Example of Refining
Existing Intervention
• New conceptual model with measuring service context
variables & moderator variable to determining effects on
outcomes
• Quantitatively assessing conceptual model
– Test utility of model
– Estimate effect sizes of predictor variables & outcomes
• Qualitative component – examine experiences of
implementing intervention & identify factors that promote
or inhibit effectiveness of intervention
• Refine model based on results & more definitively
operationalize service context variable & implementation
of intervention
62
Developing Conceptual
Foundations for RCTs
Conceptual Foundations for RCTs
• Theories for RCTs support explanatory
models of process & outcomes
• Frameworks that delineate role of
intervention in affecting change
• Empirical base justifies change over time –
expected timeframe for specific levels of
change
64
Common Theories for Interventions
• Cognitive Behavioral Theory
• Social learning theory
• Stress, Coping, & Adaptation
• Social Support
• Social Capital
• Health Beliefs
• Theory of Planned Behavior/Theory of
Reasoned Action
• Transtheoretical Model of Change
65
Stronger Theoretical Models
• Mediators
– variable that is hypothesized to help make
change happen
– Conceptual link in the middle of cause &
effect argument
– Sometimes referred to as intervening or
process variable
– Mechanisms of change in outcomes
associated with the intervention & precede
outcome
66
Mediation Diagram
MED
X Y
67
Mediation
• Step 1 Show intervention variable is
correlated with outcome
• Step 2 Show intervention variable is
correlated with mediator
• Step 3 Show mediator affects outcome
• Step 4 To establish mediation, effect of
intervention on outcome, controlling for
mediator should equal 0 or greatly
reduced (partial mediation)
68
Stronger Theoretical Models
• Moderators
– Variable that interacts with intervention in
such a way that interaction variable has a
different effect or strength of the effect on the
outcome
– Moderators alter strength of causal
relationship
• e.g., psychotherapy may reduce depression more
for men than women or high risk youths do better
on outcomes
69
Stronger Theoretical Models
• Moderators associated with service
context &/or service population e.g., Police
intervention program for persons with
mental illness (Crisis Intervention Team)
moderated by available MH treatment
programs in community
• Moderator analysis assess external
validity – answers question of how
universal is causal effect
70
Moderator Diagram
MOD
X Y
71
Experimental Intervention
Compared to What?
• Essence of RCT question is “Compared to
What?”
• Need to consider what usual care is – TAU
• If no usual care, nothing or waitlist appropriate
comparison
• Benign intervention, such as supportive or
educational interventions, not expected to have
deep or lasting impact on outcome measure
• Control condition used to control for attention or
placebo effect as could affect outcome
72
Examples of Comparisons
• Consumer Case Management Teams
compared to Non-Consumer Case
Management Teams
– Outcomes essentially same for both teams
– Limitation –Could be both team equally
ineffective – with no control condition this
alternative hypothesis could not be ruled out
73
Examples of Comparison
• Problem–Solving Educational intervention
compared to depression education
materials & referral for antidepressant
medication among elderly with depressive
symptoms receiving home health care for
their medical problems
– Standard care alone not felt to be strong
comparison to determine effectiveness
– Limits external validity of study results
74
Designing RCTs
Be Sure Design Matches The
Policy Relevant Question
Is Mental Health Treatment Court (MHTC) more
effective than usual adversarial court processing in
reducing criminal activity and improving their
psychosocial functioning for adults with mental
illness involved in the criminal justice system?
- MHTC part of large movement of “therapeutic
jurisprudence”
_ designed to reduce arrests & jail time by addressing
psychosocial needs of indiv.
- MHTC involves cooperative agreements between
criminal justice & MH treatment providers
76
Be Sure Design Matches The
Policy Relevant Question
- designed to reduce arrests & jail time by addressing
psychosocial needs of indiv.
- MHTC involves cooperative agreements between
criminal justice & MH treatment providers
- Indivs. served poor tx compliance lead to erratic
behaviors, but safely be diverted from criminal justice
system
- ACT is EBP for helping persons with SMI
- MHTC incorporated an ACT approach
- Adversarial court processing received usual MH
services
77
What Design Captures the Relevant
Policy Question For Case Example?
• Design employed:
R: MH Court +ACT
R: TAU Court + TAU MH services
78
Problems with Design Employed
• Study provided most positive evidence of
MH Courts
• However, was it MH Court or ACT?
• Or, interaction of the two?
• Do not know
79
Design Required to Answer Policy
Relevant Question
R: MHC + TAU MH services
R: MHC + ACT
R: TAU court + ACT
R: TAU court + TAU MH services
(Control Condition)
80
Design Required to Answer Policy
Relevant Question
• Policy relevant design provides attribution
to outcome of
• Court
• ACT
• Interaction of the two
• However require sufficient sample size of
eligible & willing participants
81
Controlling for Contamination
• Referred to as blurring of conditions, drift,
or treatment dilution
• Ways contamination may occur:
– Control condition participants gain benefit
from experimental condition
– Experimental condition drifts toward control
condition
– Control condition drifts toward experimental
82
Examples of Drift
• Caused by either provider or client behavior
• Drift between ACT & individual intensive case
managers
– Individual case managers from same agency began
functioning as a team
– Resulted in blurring of conditions
• Clients sharing same waiting room
• Behavioral anger management intervention with
homework assignments taking place in a
residential treatment setting
83
Potential Solutions
• Different locations
• Different times of operation
• Different providers delivering the exp. &
control interventions
But these solutions raise additional
confounds
- result in different types of clients
- providers with different qualifications
& experience
84
Design Consideration to
Address Contamination
• Provider qualifications
• Training providers
• Ongoing support to providers
• Monitoring of interventions
85
Changes in
Intervention Environment
• History internal threat
• Policy change may affect one or both
conditions
• Becomes a confound when interacts with
one condition differentially to outcome
• One proposed strategy is nested RCT in a
longitudinal quasi experimental design
• Another is conducting continuing ongoing
process assessment
86
Biased Attrition
• Biased attrition to one condition or the
other is real threat to internal validity
• E.g., Concern of biased attrition in control
condition of ACT homeless jail study
• Loss also reduces power
87
Potential Design Solutions to
Attrition
• Protocol designed to engage & keep participants
engaged
• Pre-randomization introductory phase absorbing early
stage attrition
– Trade off – reduced external validity
• Increased incentive payment at points expect greater
loss
– e.g., Exit from prison
• More participants assigned to condition with greater
anticipated loss
• Statistical procedures – require anticipation to obtain
necessary data
88
Randomization
• Usually equal assignment to all conditions
• Unequal assignment requires justification
• Computer randomization preferred to
physical manipulation
89
When to Use Stratified
Randomization?
• Randomization may not ensure equal
proportions across conditions
• When sample size small
– e.g., less than 100
• Subpopulation small
– e.g., less than 20%
• Bigger problem of small samples – low
power
• Increases complexity
90
When to Use Cluster
Randomization?
• Control for contamination
– e.g., same providers delivering two interventions
• Efficiency – everyone in intervention served in
one location
• Cost & time-efficient – can‟t feasibly gain
consent from everyone & change in policy or
guideline
• Limitation – requires larger sample size to
maintain power
91
When to Use Blocked
Randomization?
• When employing group interventions
• Control flow into different conditions
• Assignments made for smaller units, such
as in blocks of 4, 6, etc.
92
Blinding
• Controls for potential bias, specifically
reactivity of client &/or provider
• Difficult to do in community–based
psychosocial interventions
• Possibly blind data collectors
93
Randomization in Practice
• Assignment occur after consent & baseline
assessment completed
• Random assignment not in hands of
providers or even research workers
• Procedures for random assignment
centrally controlled to protect against
subversion
94
How to Design Recruitment &
Sampling Strategy
• Need to demonstrate can consent &
maintain sufficient sample size for analysis
• Need to determine at what point in pipeline
feasible & conceptually justified to recruit
95
What Inclusion Criteria to
Consider?
• Need to operationally define inclusion &
exclusion criteria
• Consideration & implications of criteria,
e.g., new intakes or current clients, i.e.,
current testing TAU + exp. int.
• Consideration of age, diagnoses,
language, & geography
96
What Exclusion Criteria
to Consider?
• Vulnerable populations
• Co-morbid or specific disorders
• Specific system status levels
• Frequently, no exclusion criteria
97
Considering Sample Method &
Recruitment Process
• Frequently use consecutive samples
• Combination of purposive, snowball, &
quota samples
• Agency staff vs. research staff doing
recruitment
98
Determining Sample Size
• Determining effect size
– Prior research – literature
– Pilot studies
• Estimating attrition
– Prior studies
– Pilot studies
99
Operationalizing Experimental &
Control Interventions
• Need to clearly specify all conditions
– Experimental interventions – manualized/tool
kit – clearly specified intervention
– Justify that exp. & control conditions truly
differ
– Need to operationalize TAU & benign
interventions
– If exp. longer or more intense (dosage) than
control – time &/or amount may be variables
effecting outcome
100
Outcome Measures & Data Points
• Need psychometrically sound measures –
unreliable measures reduce power
• Valid measures for sample
• Sensitive to capture change in short time
frame
• Some concepts unlikely to change in short
time frame
• Justify time period for data points
101
Approaches to Data Analysis
• Expected to use Intent-to-treat analysis
• Avoid temptation of eliminating participants
receiving limited service
• Dosage effect variables – fidelity, compliance,
adherence, & engagement
• Carefully conceptualize dosage effect so do not
substitute for main independent variable
• With enough data points can estimate missing
data
• Statistical consultant early in design process
102
Concerns with RCTs
• Highly selective samples
• Preferences interacting with actual service
delivered
• Complex interventions not accounting for
accumulative effects of all service
components
103
Alternative Designs to RCTs
• Fixed adaptive designs
– randomly assigned to condition, but progress
through intervention determined by intensity
of treatment need
• Randomized adaptive designs
– changes in service condition are done by
randomization to choices of participant or
provider
104
Alternative Designs to RCTs
• Encouragement or randomized consent
trials
– Encouraged to participate in one service
option or other, but constricted to the selected
option
• Randomized preferences
– Participants decide whether they will be
randomized or choose their service option
105
Implementing Randomized
Designs
Preparing Setting for RCT
• Inform setting with time for preparation, but not
so far in advance that forgotten
• Research & Agency jointly decide on how &
when to inform personnel
• Jointly present RCT with administrators & staff
• Need to sell RCT on benefits to setting,
providers, & clients
• Don‟t oversell what can‟t be delivered
• Understand provider‟s perspective
107
Preparing Setting for RCT
• Sensitivity to language & examples employ
• Turn lack of clarity into an advantage
• Anticipate questions & issues & raise them first
• Address random assignment in straightforward
manner
• Try to counter negative momentum
• Positive frame of mind critical – “you need them
more than they need you”
108
Tracking Participants
• At enrollment participant complete locator
form – working document of all info to help
find someone including:
– Demographic & identifying info
– Relatives, info from multiple people at
different locations
– Professional contacts for contact info
– Incidental contacts
• e.g., where one goes when out of money, or
hungry, or where one sleeps when homeless
109
Tracking Participants
• Working document
– update every time contact participant
- indicate helpful & unhelpful info
- offer incentive for participant to contact
researchers with change of info
• Computerized system to generate timely
lists for follow-up data points
110
Monitor Recruitment
• Use track system, monitor recruitment to ensure
accruing sample to meet timely projections
• Big push late in study resulting in non-
completers of intervention &/or outcomes
• Relying totally on providers is usually ineffective
• Creative means to control recruitment within
confidentiality & legal policies
• Remember providers usually do not want
responsibility for recruitment
111
Referral Process
• Providers make referrals, but best they not
do eligibility determinations or consents
• Providers obtain Release of Information
form from potential eligible participants
• Provider referrals based on easily
observable or obtainable criteria (using
system categorization), & casting wide net
– Lessens burden on providers
– Providers more likely to do
112
Ensuring Participant Retention
in Research
• Collect complete locator info at study
entrance
• Inform participants when they will be
followed up
• Review locator information at subsequent
data collection points
• Offer adequate incentives
• Employ effective research data collectors
113
Ensuring Participant Retention
in Research
• Document all follow-up activities in detail
• Exploit contact information obtained
• Reasonably accommodate participant for
follow up data collection
• Allocate enough resources for travel
• Allow ample time for tracking down
participants
114
Ensuring Participant
Engagement in Intervention
• Communicating importance of intervention
• Outlining benefits & expectations of participants
• Making minor modifications
– e.g., reducing # of sessions if too many dropping out
• Training providers (both conditions) in
engagement, retention, & relationship building
• Building trust
• Incorporating outreach efforts as part of
intervention
• Novel thinking
– e.g., giving up professional offices
115
Qualifications & Training
of Providers
• Equality of qualifications for all conditions –
otherwise confounding
• Training of Experimental Providers
– Human subject protections
– Overview & purpose of RCT
– Conceptual basis of RCT
– Design of RCT
– Appealing argument for need for random assignment
– Operation & implementation of RCT
116
Training Experimental Providers
• Introduction to intervention
• Program philosophy
• Program goals & principles
• Practice experience delivering intervention
• Role modeling with target population
• Review manual/toolkit, etc.
• Using existing training material if available
• Consider hiring trainer – control for potential bias
• Consider on-going support, coaching, booster
sessions
117
Training Experimental Providers
• Training involves engaging, teaching, &
supporting in performance of intervention
• Training ensures fidelity of intervention
• Provision for new hires
• Supervisors need to be trained
• Supervision/monitoring of exper. providers
best done by those with investment in
RCT- e.g., research staff
118
Training & Monitoring
Control Condition
• Less involved than experimental condition
• Training in eligibility determination
• Training in completion of fidelity / leakage
forms
• Researcher monitor fidelity / leakage
forms to take corrective action
119
Training & Supervising
Research Staff
• Rationale for RCT
• Overview of RCT
• Human Subject Protection
• Recruitment procedures
• Randomization process
• Review of all data collection forms
• Training in experimental intervention if providing
ongoing support & technical assistance
120
Fidelity Assessment
• Time points – developmental & mature phases
of intervention
• Provider & client perspective
• Data sources
– Billing data
– Treatment/activity logs
– Attendance records
– Site visits
– Ethnographic methods
• e.g., shadowing
121
Assessing Environmental Context
• Systematically tracking organizational
changes
– e.g., policy, eligibility requirements by dates
• Use of quantitative & qualitative methods
• Importance as participants will be served
over time – not all served at same point in
time
122
Implementation Disaster
• To test effectiveness of self-help for persons
with severe mental illness
• Roster of 1185 clients from an urban CMHC who
received tx in past 2 years; 853 met eligibility;
decreased to 241 due to hosp, participation in
self help, etc, but 90 consented, completed data,
& were randomly assigned
• Inclusion criteria:
– Dx. schiz., schizaffective, or major mood
– Normal intelligence
– Not participated in self help
123
Implementation Disaster
• Both groups monitored for self help attendance to
assess contamination
– reviewed daily sign in sheets of self help group
• 17% of both conditions participated in self help
• Self help has selective rather than universal appeal
• Outreach efforts minimally affected participation
• Self-selection tremendous impact on sample size in self
help research – not likely to recruit adequate randomized
sample with no prior exposure to self help from a single
geographical area
(Kaufman, Schulberg, & Schooler, 1994)
124
Final Note
Final Note
• Generalizability to other service settings
• Sustainability of intervention in research
setting
• Transparency of reporting RCTs
– CONSORT: checklist & flow diagram
• If experimental intervention effective, cost
effectiveness important – but need to
design at beginning of study, not as an
after thought
126
127
