Get documents about "Refinement of the Patient Reported Outcomes Version of the
Document Sample
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
Refinement of the Patient-Reported Outcomes Version of the
Common Terminology Criteria for Adverse Events (PRO-CTCAE):
Usability Testing
NON-THERAPEUTIC/DIAGNOSTIC PROTOCOL
Name Department/Institution
Site Principal Investigator Amy P. Abernethy, MD Department of Medicine, Division of
Medical Oncology, Duke University
Medical Center (Duke)
Investigators
Jane Wheeler, MSPH Medical Oncology, Duke
Please Note: A Consenting Professional must have completed the mandatory Human
Subjects Education and Certification Program.
Collaborating Institutions Memorial Sloan Kettering Cancer Center
Dana-Farber Cancer Institute
M.D. Anderson Cancer Center
Duke University Medical Center
National Cancer Institute
Date prepared: 1 August 2009
Title Page
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
TABLE OF CONTENTS
1.0 PROTOCOL SUMMARY .............................................................................................. 1
2.0 SCIENTIFIC AIM .......................................................................................................... 2
3.0 BACKGROUND ............................................................................................................ 2
4.0 STUDY DESIGN AND IMPLEMENTATION ............................................................. 3
4.1 DESIGN .................................................................................................................... 3
4.1.1 PROJECT LEADERSHIP ...................................................................................... 4
4.1.2 SETTING ........................................................................................................... 4
4.1.3 PARTICIPANTS .................................................................................................. 4
4.2 METHODS ................................................................................................................. 5
4.2.1 RECRUITMENT AND ENROLLMENT ................................................................... 5
4.2.2 TRAINING AND SITE PREPARATION .................................................................. 6
4.2.4 DETAIL OF METHODS ....................................................................................... 7
4.3 DATA COLLECTION ................................................................................................ 9
4.3.1 DATA COLLECTION FORMS ............................................................................... 9
4.3.2 RECORDING ................................................................................................. 10
4.3.3 POSSIBLE FUTURE METHODS OF DATA COLLECTION ...................................... 10
5.0 ELIGIBILITY CRITERIA............................................................................................ 11
5.1 INCLUSION CRITERIA (PATIENTS) ..................................................................... 11
5.2 EXCLUSION CRITERIA (PATIENTS) .................................................................... 11
5.3 INCLUSION CRITERIA (PROVIDERS) .................................................................. 11
5.4 EXCLUSION CRITERIA (PROVIDERS)................................................................. 11
6.0 ANALYSIS PLAN ....................................................................................................... 12
7.0 TOXICITIES/SIDE EFFECTS ..................................................................................... 13
8.0 PRIMARY OUTCOMES ............................................................................................. 13
9.0 CRITERIA FOR REMOVAL FROM STUDY ............................................................ 13
10.0 DATA MANAGEMENT............................................................................................. 13
10.1 QUALITY ASSURANCE .................................................................................... 14
10.2 DATA AND SAFETY MONITORING ............................................................... 14
10.3 SYSTEM SECURITY .......................................................................................... 15
11.0 PROTECTION OF HUMAN SUBJECTS ................................................................... 15
11.1 SERIOUS ADVERSE EVENT AND ADVERSE EVENT REPORTING ................. 15
12.0 INFORMED CONSENT PROCEDURES ................................................................... 16
12.1 RESEARCH AUTHORIZATION ........................................................................ 16
13.0 REFERENCES ............................................................................................................. 17
14.0 APPENDICES .............................................................................................................. 17
APPENDIX A: USABILITY TESTING INTERFACE SCREENSHOTS……….App A
APPENDIX B: USABILITY TESTING INFORMED CONSENT………………App B
APPENDIX C: INFORMATION SYSTEM SECURITY PLAN…………………App C
Date prepared: 1 August 2009
Table of Contents
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
1.0 PROTOCOL SUMMARY
Currently, adverse symptom events occurring in National Cancer Institute (NCI)-sponsored
clinical trials are reported by clinical staff using items from the Common Terminology
Criteria for Adverse Events (CTCAE). A recognition of methodological problems with the
current collection of CTCAE symptom data, the availability of technologies that facilitate
patient self-reporting, and the release of an FDA guidance recommending standards for
patient-reported outcomes (PROs) in the regulatory setting have, in concert, prompted
research on PRO approaches to the CTCAE. As it currently stands, the CTCAE contains
1,059 discrete items, approximately 77 of which contain a subjective component sufficient to
be amenable to patient self-reporting.
This protocol represents one component of a recently funded NCI contract (N02-PC-85002-
29; Principal Investigator, Basch; Co-Investigator, Abernethy). This protocol describes the
usability testing component of the larger PRO-CTCAE initiative, which includes
creating/validating new adverse symptom items for use in NCI-sponsored trials; conducting
cognitive interviews to assure patient understanding of items; and developing a software
system to administer these items. Specifically, this protocol assesses the usability of the
software system.
The specific objective the of usability testing portion of the contract is to evaluate the usability
of the technology interface for collecting the PRO data of the newly developed PRO-CTCAE
system; this protocol follows completion of cognitive interviews, which will examine
comprehension of the newly-developed items as well as the extent to which the items
correspond to the concepts of interest for each PRO.
For purposes of this study, usability of the technology will be defined as its capacity to allow
users to carry out their tasks – in this case, response to PRO-CTCAE items using the software
interface – safely, effectively, efficiently, and enjoyably.1 Usability testing will comprise
three main components: (1) observation of users interacting with the system, guided by “think
aloud” protocols, and the taking of extensive field notes; (2) the gathering of user feedback
through brief semi-structured interviews with users immediately following their use of the
system; and, (3) web analytics including tracking for different web pages, with clickstream
analysis.
Duke will serve as lead site for usability testing, functioning as central coordinating site and
data repository for the Task 6 study. SemanticBits LLC, under a contract from the NCI, is
creating the software; this software will be open source and will comply with NCI security
and privacy specifications.
Date prepared: 1 August 2009
- Page 1 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
The end result of this usability testing protocol will be assurance that target users of the newly
developed PRO-CTCAE system – namely, (a) patients with diverse characteristics, and (b)
clinical research staff – can conduct the expected tasks using this system.
2.0 SCIENTIFIC AIM
The aim of this study is to evaluate the usability of the technology interface developed to
collect the PRO data elements of the PRO-CTCAE system, once it has been built and
modified to accommodate the learning garnered during cognitive interviews (Task 4 of the
overarching project).
3.0 BACKGROUND
Usability of a software product or technology system is broadly defined as the capacity of
that product/system to allow users to carry out their tasks safely, effectively, efficiently, and
enjoyably.1 A “black-box” technique, usability testing aims to observe people, representative
of future users, interacting with a product/system while they are in a realistic environment
that simulates the product/system’s “real world” use. The process seeks to discover
problems, collect quantitative data on users’ performance (e.g., time to task completion, error
rate), determine users’ satisfaction, and identify areas for improvement. Usability testing
generally involves collection of data with respect to four areas of user response: efficiency,
accuracy, recall, and emotional response.2
In a usability test, representative users perform typical tasks with the product/system;
observers (i.e., the testers) watch, listen, and take notes but refrain from instructing or
guiding the user while he/she performs the tasks. Usability testing often follows a “think
aloud” protocol, a method first developed by Clayton Lewis3. In a think aloud protocol,
users verbally describe their processes as they perform the requisite tasks with the
product/system. For example, users talk through what they are seeing, doing, thinking, and
feeling as they undergo the task. The think aloud protocol allows an observer to gather data
on the entire process of product/system use, rather than on end results alone. The observer
objectively records everything that the user says without adding any interpretation.
Additional data is often obtained through audio- or video-taping the session, and
supplementing observer notes with information gained through review of the recording.
Usability testing has been employed to test various new interventions among patients of
diverse disease and demographic characteristics. A large number of usability testing studies
has been published in the medical literature, reporting on the usability of a range of
information systems, user interfaces, web-based applications, and software. Of relevance to
this usability testing protocol are studies that involved minority and low-income individuals,
as it is important that the PRO-CTCAE system under development accommodate these
Date prepared: 1 August 2009
- Page 2 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
potential users. McDaniel et al. performed usability testing of an interactive computer-
mediated smoking cessation program for inner-city women (n=100).4 Sheinfeld Gorin et al.
conducted usability testing of a physician-based prostate cancer education program in an
African American community.5 Also of note is a study by Faulkner et al. which challenged
the general assumption that a sample of five participants is adequate to determine usability
concerns and performance. The investigators included 60 participants in a set of usability
tests, randomly defining subsets of five participants from the total pool. While some
randomly selected sets of five participants identified 99% of problems, other sets found 55%.
Accuracy in identifying problems increased with the number of users included in the set.6
This protocol focuses on the usability of the newly developed PRO-CTCAE system. The
study is designed to ensure that the targeted users can conduct the expected task, namely, to
use the technology interface to report on their symptoms via response to PRO-CTCAE items.
Targeted users include: (a) patients from diverse backgrounds with varying educational
levels, computer experience, and physical illness; and, (b) clinical research personnel, such as
nurses, clinical research coordinators, research assistants, and physicians. Based on the four
standard domains examined in usability testing (efficiency, accuracy, recall, and emotional
response), measures have been defined so as assess five indicators of usability that pertain to
the PRO-CTCAE technology interface.
1. ease of learning – How fast can a user who has never seen the user interface before
learn it sufficiently well to accomplish basic tasks?
2. efficiency of use - Once an experienced user has learned to use the system, how fast
can he or she accomplish tasks required by the system?
3. memorability - If a user has used the system before, can he or she remember enough
to use it effectively the next time, or must the user re-learn the system?
4. error frequency and severity - How often do users make errors while using the
system, how serious are these errors, and how do users recover from these errors?
5. subjective satisfaction - How much do users like using the system?
All five indicators contribute important information to each of the four standard domains of
usability testing.
4.0 STUDY DESIGN AND IMPLEMENTATION
4.1 Design
This usability testing protocol replicates real-world scenarios in which future users will complete
the PRO-CTCAE. Testing will take place in two rounds of one-on-one (user/observer) usability
testing sessions, such that experience gained and lessons learned in the first round can inform
refinement of the system for further testing in the second round.
Date prepared: 1 August 2009
- Page 3 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
Usability testing will collect data from clinical research staff and patients in three ways: (1)
through observation of users interacting with the system, with data recorded on a data collection
form and in extensive field notes; (2) during semi-structured interviews with users immediately
following the observation (#1, preceding), with data collected on a standardized interview form
and supplemented with review of audiotapes of the session; and, (3) through web analytics
including tracking for different web pages, with clickstream analysis. These sources will capture
information from all five critical usability domains: ease of learning, efficiency of use,
memorability, error frequency and severity, and subjective satisfaction. Web analytics will
provide quantitative information on efficiency, memorability, and errors.
Semantic Bits will customize software for this study, including a form builder for creation of the
data collection forms, a user interface for both patients and clinical research staff, reporting
functions, and a study calendar. Examples of the user interface are provided in Appendix A. The
software will be hosted on a secure server at the National Cancer Institute (NCI) during the
development of the software, although no patient data will be stored during the usability testing;
only web analytics information will be collected for usability analysis.
4.1.1 Project leadership
Usability testing will be led by Dr. Abernethy (Task Leader) and Drs. Hay and Mendoza (Deputy
Task Leaders) in conjunction with Dr. Basch (Principal Investigator). The Task 6 team has
communicated frequently with Mr. Chilukuri and SemanticBits during the system design phase,
to understand and inform the platform and system design characteristics, recommend
modifications as needed, and plan for usability testing of the beta model of the PRO-CTCAE
assessment. Located at the central coordinating site for this study (Duke), Dr. Abernethy will
oversee accrual and data collection at the sites, and integration of these data into the centralized
data repository.
4.1.2 Setting
The usability testing plan is carefully designed so as to create scenarios that realistically
represent future uses of the system. The settings selected for testing will enable study personnel
to observe users’ in situ interactions with the technology. Thus, at each site, usability tests will
be conducted in the environment where the system is expected to be most commonly used.
These settings primarily include, but are not limited to, the sites’ outpatient oncology clinics.
The inclusion of inpatient tests will ensure that the system accommodates data collection from
hospitalized patients.
4.1.3 Participants
Participants will be recruited to reflect the composition of the population of future users. Two
groups of users will be included: clinical research staff and patients. In total, usability testing
will involve 60 clinical research staff evaluations and 120 patient evaluations, divided
approximately equally between the two rounds of usability testing (i.e., 30 clinical research staff
Date prepared: 1 August 2009
- Page 4 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
and 60 patients participating in each round). Among clinical research staff, each round will
include 10 nurses, 10 clinical research assistants, and 10 physician investigators. A distinct,
specific definition for each clinical research staff title will be developed and used to categorize
participants; this specificity will avert any potential issues that might arise if different institutions
use the same role terminology to describe different positions.
Among patients, each round will minimally include 60 patients. We will enroll at least 10
individuals with each of the following characteristics: over the age of 75 years, less than a high
school education, non-white race/ethnicity, and inpatient or severely ill (ECOG >2). At least 30
participants will have metastatic disease (vs. non-metastasized cancer). (Note: In certain cases,
the same individual may represent more than one of these characteristics.) After 40 patients have
been involved in each round of usability testing, we will evaluate the composition of the
participant group and enrich it with more patients of a specific characteristic (e.g., low education
level, if this characteristic is under-represented) as warranted. The purpose of this enrichment
will be to ensure that all of the designated characteristics are adequately represented in the
simulated user population.
Participants will be recruited from three institutions: Memorial Sloan Kettering Cancer Center
(MSKCC), M.D. Anderson Cancer Center, and Duke Comprehensive Cancer Institute, and their
respective satellite/network clinics. This inclusion of multiple institutions will ensure a broad
representation of academic and community-based users with diverse demographics, as well as
multiple oncology-based disciplines including medical oncology and radiation oncology. The
anticipated number of patients and clinicians to be enrolled at each site appear in Table 1, below.
Table 1: Number of participants in usability testing by institution, per round
Institution Participants
Patients Clinicians/Researchers
Duke 25 10
MSKCC 10 10
MD Anderson 25 10
4.2 Methods
4.2.1 Recruitment and enrollment
Participants who meet eligibility criteria (Section 5.0, below) will be recruited at each site by
site-based study personnel, in accordance with best clinical research practice. Interviewers will
work closely with clinic staff and physicians to identify potentially eligible and interested
patients. These patients may receive a telephone call from clinic staff in advance of their clinic
appointment to describe the study to them, or they may be approached in the waiting room as
Date prepared: 1 August 2009
- Page 5 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
they wait for a regularly scheduled clinic appointment. In each case, the patient’s primary
physician will be aware of the study and will have given study personnel his/her agreement to
approach the patient. Once in the clinic, the study staff-person will briefly discuss the study, its
purpose and procedures, to ascertain whether or not the patient is interested in participating.
Patient registration will be coordinated by the Clinical Trials Manager with the Duke Cancer
Care Research Program at Duke University Medicine Center. Dr. Amy Abernethy will provide
oversight, and will be in frequent contact with site-based study personnel and data managers.
No participant will be considered enrolled until he/she has provided informed consent. Consent
will be obtained using the form provided in Appendix B.
4.2.2 Training and site preparation
Prior to recruiting participants for the usability testing, Drs. Abernethy, Hay, Mendoza, and
Basch, together with experienced Duke staff trained in usability testing, will work with Mr.
Chilukuri and SemanticBits programmers to finalize the think aloud protocol and to ensure that
the web tracking software is incorporated into the PRO-CTCAE site.
A standard usability testing manual will be used for training site-based study personnel. This
manual will fully describe the conduct of one-on-one usability testing sessions including
observation, data collection/recording, semi-structured interviewing, and data management. The
manual will ensure consistent practices for usability testing across the three sites. Dr. Abernethy
and the Duke team with expertise in usability testing will train research staff at each site in the
protocol, expectations, and data collection processes. A copy of the manual will be left at each
site as a reference guide for site-based study personnel, and the Duke team will remain available
by email and telephone to answer any subsequently arising questions.
Training will explicitly describe data collection procedures under this protocol. The importance
of field notes and extensive written data collection will be emphasized, in addition to proper
recording of quantitative data using the data collection form (Section 4.3, below). Training will
utilize role plays and practice sessions (observation, interview), to ensure a realistic and
pragmatic approach that allows site-based study personnel to develop competence and
confidence in the protocol. Research staff will be allowed to hand-write or dictate field notes,
though they will need to transfer handwritten field notes into typewritten text. Training will
inform site-based personnel about data management procedures.
If site-based study personnel do not feel comfortable conducting the usability tests after the
training, Duke research staff will guide the process through video conferencing and/or will travel
to the site to provide further onsite training and facilitate participant accrual.
4.2.3 Overview of methods
Date prepared: 1 August 2009
- Page 6 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
This usability testing protocol will entail one-on-one interviews between participants/users and a
member of the clinical research staff. Each participant’s involvement in the study will consist of
approximately one hour of time, spent in the clinic or hospital using the system and undergoing a
brief semi-structured interview. Usability tests and interviews will be conducted in a private
space identified by the study team in coordination with administrators and/or clinicians at each
site.
4.2.4 Detail of methods
Procedures for usability testing are as follows:
(1) Written informed consent will be obtained from each participant prior to any data
collection. (See Appendix B: Usability Testing Informed Consent.)
(2) The participant’s will first be engaged in a one-on-one usability testing session, in which
the user/participant navigates the technology interface (software housing the PRO-
CTCAE draft questions, or clinical research interface) and describes his/her processes
according to a think aloud protocol, while the observer/study staff-person records
specified data. Each session will be conducted “in situ,” i.e., in the setting where that
user type is ultimately expected to use the software system; for example, patients will
undergo usability testing in clinic waiting area or hospital room while a clinical research
coordinator will complete the testing in his/her research office.
Usability testing sessions with patient participants/users will entail responding to
approximately 25 draft PROs using an electronic platform (a desktop, laptop, or wireless
tablet personal computer). The software will ask the patient/user to answer a series of
questions, one question per screen; each question will address a particular symptom, the
experience related to a symptom, the intensity of a symptom, or the symptom’s
interference with daily life. The patient will answer each question as instructed by the
system, most likely through touching the screen at an appropriate place. While the user
completes this draft PRO assessment, the observer/study staff-person will record data
onto a data collection form and will take field notes. Following a standardized script, the
study staff-person will encourage the user to think aloud about the software, use of the
system, and attributes of the system as well as to narrate his/her interactions with the
system. The session will be audio-recorded to capture this verbal information; these
qualitative results will be transcribed and collated with the transcribed field notes.
Similarly, the clinical research personnel participant/user will test the software’s research
interface. The usability testing protocol for this group of users will entail similar
procedures as for the patient participant/user group: a standardized think aloud script,
observation and field note taking on the part of study staff, and audio-recording.
However, clinical research personnel will not use the system to respond to the draft
ePROs (as will patient participants). Instead, they will use the software system to:
Date prepared: 1 August 2009
- Page 7 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
nominate symptom items for adverse event reporting; create an adverse event reporting
survey using the symptoms which they nominated; create a schedule for presentation of
surveys to patients; and, create a plan for follow-up and monitoring of symptoms,
including severe symptoms that exceed risk threshold scores.
Immediately after the participant (either patient or clinical research personnel) interacts
directly with the software interface, the study staff-person will conduct a brief semi-
structured interview with him/her to supplement information gained through the think
aloud protocol, observation, and field notes. Semi-structured interviews will consist of
ten open-ended questions related to the five indicators of usability listed in Section 3.0,
above. A different list of questions will be created for each of the two groups (patients,
clinical research personnel). Example questions posed to patient participants/users are:
Is this software easy to use for reporting symptoms that you may be experiencing?
Does the system make instructions clear and easy to follow?
What about the system would you suggest changing in order to make it easier to
use?
How well did you like using the system?
Can you suggest ways that we might make the system more likeable?
Example questions posed to clinical research personnel participants/users are:
Is it obvious how to initiate a new study using this software system?
Is it easy to schedule surveys for patients using this system?
Is the scheduling system convenient and useful?
Are email notifications provided by the system clear and feasible in a busy
clinical practice?
Following the completion of each one-on-one session, the study staff-person will review
his/her data and field notes, before passing them along to the study coordinator.
(3) Quantitative data will be captured through web analysis of software use. Specifically,
with respect to the patient participant/user group, clickstream analysis will capture the
amount of time which the user spends per screen (i.e., PRO-CTCAE item), number of
errors and time taken to correct errors, and total time spent on the PRO-CTCAE survey.
These data will be housed on secure servers at the NCI designated to house the PRO-
CTCAE system. Notably, during this study, only a unique identifier number will be
associated with each patient’s data on these servers, and no identifying patient
information will be stored on the NCI servers. Hence, each participant/user will be
assigned a unique identifier; de-identified datasets will be transferred to Duke for
analysis.
Date prepared: 1 August 2009
- Page 8 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
(4) Web analytics data collected in the one-on-one usability testing sessions will be de-
identified. Each participant/user will be assigned a unique identifier; only the Task
Leader and authorized study personnel will have access to the identifying schema. Data
will be entered into a secure study database. Summary data (de-identified) will be posted
in a password-protected area of the study wiki site. Audiotapes and Data Collection
Forms will remain at the local site, under locked containment, for six years and then
destroyed.
(5) After completion of a first round of one-on-one usability tests (#2 above), engaging
approximately 30 clinical research staff and at least 60 patients, data will be reviewed to
identify major usability issues that can be addressed by modification to the system prior
to the second round. The system will be refined accordingly, and prepared for the second
round of one-on-one usability testing.
(6) A second round of one-on-one usability testing will proceed as described in #2, above,
and will again engage approximately 30 clinical research staff and 60 patients.
4.3 Data collection
4.3.1 Data collection forms
Five forms will be used to collect data for the usability component of the overarching PRO-
CTCAE project. The forms, listed below, will be created specifically for this study. (Note that it
is not possible to develop these forms until the software for the system has been finalized.)
Participant Screening and Registration Log: This form will be used to track participants
screened, contacted regarding participation, reason for refusal (when a person does not
agree to participate), and enrollment in the trial (when a person agrees to participate).
For individuals who do not agree to participate, we will only record a unique screening
number (so that we can track total number screened) and reason for refusal; personal
health information will not be recorded for non-consenting individuals. For individuals
who consent to participate, we will record a unique study linking identification number
(ID), demographics, education level, experience with computers and internet, and basic
illness characteristics (cancer type, stage, performance status, current treatment).
Patient Usability Testing Data Collection Form: This structured form will be used by the
site-based study personnel for collecting data during the one-on-one usability testing
sessions with patient participants. The form will contain blank space for free-form field
notes.
Date prepared: 1 August 2009
- Page 9 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
Clinical Research Personnel Usability Testing Data Collection Form: This structured
form will be used by the site-based study personnel for collecting data during the one-on-
one usability testing sessions with clinical research personnel participants. The form will
contain blank space for free-form field notes.
Patient Usability Testing Interview Form: This semi-structured form will be used by the
site-based study personnel for recording participants’ responses to the ten questions
posed in the brief interview following the one-on-one usability testing session.
Clinical Research Personnel Usability Testing Interview Form: This semi-structured
form will be used by the site-based study personnel for recording participants’ responses
to the ten questions posed in the brief interview following the one-on-one usability
testing session.
4.3.2 Recording
All one-on-one usability testing sessions will be audio-recorded. Audio-recording will allow for
the capture of emotion in participants’ verbal expression, which may add depth to the
information collected. Additionally, study personnel will review the audio-tapes against
information in the field notes and on the data collection forms, to ensure that all relevant data are
captured. Transcription is not planned, as the audiotapes are intended to supplement already
collected data; instead, study personnel will listen to the audiotapes and will add any new data
from that medium to the data collection form and/or field notes. If it becomes apparent that
transcription would contribute to the results of this study, we will then proceed with it.
Recording will be accomplished using digital recording equipment. The audio recordings will be
downloaded into files on protected computers at each site (Duke, MSKCC, MD Anderson), and
de-identified data will be sent to Duke.
4.3.3 Possible future methods of data collection
We do not anticipate needing to incorporate eye tracking protocols or video-based data collection
of screen movements into this study. However, if results of the first round of usability testing
indicate a need for the further information provided by these strategies, we will prepare a
protocol amendment to add either or both of those elements, will modify the study description
(for recruitment purposes) and informed consent accordingly, and will obtain IRB approval prior
to using these strategies in round two of usability testing.
Date prepared: 1 August 2009
- Page 10 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
5.0 ELIGIBILITY CRITERIA
Patients will be eligible for the study if they are currently undergoing chemotherapy and/or
radiotherapy for cancer, whether for primary therapy with curative intent or palliative intent.
The symptoms experienced by patients receiving adjuvant chemotherapy are most likely
attributable to the therapeutic agents, thus the adjuvant setting is optimal in evaluating PROs
for treatment toxicity. However, patients with metastatic disease are also eligible because
these individuals are included in initial drug development and examination of adverse events.
5.1 Inclusion criteria (patients)
diagnosed with cancer
undergoing chemotherapy or radiotherapy with either curative or palliative intent
receiving care at one of the three study sites (Duke, MSKCC, MD Anderson)
aged ≥18 years
fluent in English
able to provide informed consent
5.2 Exclusion criteria (patients)
significant cognitive impairment
Clinician eligibility is intentionally broad, to include a representation of real-world clinical
providers who may potentially use the PRO-CTCAE system in the future.
5.3 Inclusion criteria (providers)
involved in clinical care of cancer patients and/or research enrolling cancer
patients
employed at one of the three study sites (Duke, MSKCC, MD Anderson)
fluent in English
able to provide informed consent
5.4 Exclusion criteria (providers)
involvement in this study protocol or any other aspect of the overarching PRO-
CTCAE project
Date prepared: 1 August 2009
- Page 11 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
6.0 ANALYSIS PLAN
Analytic methods will correspond to the testing round (1 or 2), participant group (patient,
clinical research personnel), and participant characteristics.
Observational data obtained through field testing is the most critical information. Typed
field notes generated through think aloud protocols and observation will be carefully
reviewed and thematically coded using a codebook analysis approach.7 The codebook
structure will include the code, a basic definition, a full definition, guidelines for using the
code, and examples. Codes will be categorized into themes and sub-themes, which will
minimally include the five usability domains: ease of learning, efficiency of use,
memorability, error frequency and severity, and subjective satisfaction. Text will be
reviewed by Duke research staff trained in codebook analysis, and codes applied. A
relational database will be used to facilitate code development and application of code to
text. The codebook analysis will provide (a) a qualitative summary of the types of problems
encountered, and (b) a quantitative summary of the aggregate number of times a particular
problem was encountered.
Web analytics and clickstream analyses will supplement the field notes and user response
data, providing additional information on efficiency, memorability, and errors.
Benchmarking/tracking information will include, but not be limited to: (1) total time
required to use the PRO-CTCAE system for new and repeat users; (2) aspects of the system
that create unnecessary delays; (3) order of pages viewed and number of times the user
needs to “go back;” and, (4) number of missed or skipped items.
Results will be reported in aggregate and by user type (patient, clinical research staff), to
ensure that different categories of users are equally capable of interacting with the system as
intended.
Summaries of the information captured in the first round of usability testing will be
reviewed with Mr. Chilukuri and the programmers from SemanticBits; this review will
inform iterative improvements in the system. The modified PRO-CTCAE system will be
subjected to a second round of usability testing to ensure that issues emerging in the first
round of testing were addressed, and that further development is not needed. Additionally,
information gathered through both rounds of usability testing will guide refinement of the
platform in preparation for the next phase of the overarching PRO-CTCAE initiative, a
validation study and a feasibility study in the cooperative group setting.
Date prepared: 1 August 2009
- Page 12 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
7.0 TOXICITIES/SIDE EFFECTS
There are no anticipated physical risks or side effects associated with participation in this
study. In the unlikely event of extreme psychological distress observed or expressed by
participants, site-based study personnel will refer the patient(s) directly the Principal
Investigator at that site. Each participant will receive contact information for these
investigators, and be informed that he/she may contact one of the investigators to discuss any
concerns related to the study. Additionally, participants will be informed during the
consenting process that they may discontinue participation in the study at any time.
8.0 PRIMARY OUTCOMES
The primary outcomes associated with this usability testing study include documentation of
the ease of learning, efficiency of use, memorability, error frequency and severity, and
satisfaction of representative users of the technology-based system for responding to the
newly developed PRO-CTCAE survey.
9.0 CRITERIA FOR REMOVAL FROM STUDY
In the unlikely event that study personnel observe acute distress in an individual during that
person’s participation in this study, he/she will ask the participant whether he/she wants to
discontinue participation and withdraw from the study. All participants will be given the
option to terminate their participation at any time.
10.0 DATA MANAGEMENT
Usability testing data (collected through the semi-structured interviews) is separate from the
software system being tested (via clickstream and web analytics). The site at Duke
University Medical Center will be responsible for data management and analysis of the
usability testing information. Data forms collected at each site will be sent to Duke for
management. While patient data and user data will be inputted to the PRO-CTCAE system
by study participants, the system will not permanently save responses during the usability
testing phase (see Section 10.3 below). There is one aspect of participant level data that will
be transferred to Duke for analysis – specifically web analytics and clickstream information.
These data will transferred as de-identified datasets to Duke for analysis. A master dataset
linking unique ID with participant personal information will be maintained by Semantic Bits
and transferred to Duke separately by secure transfer.
Date prepared: 1 August 2009
- Page 13 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
At Duke, an ANSI-compliant Microsoft Access 2003 relational database (Seattle, WA),
maintained on a server behind the Duke firewall, will be created for the data collected in the
course of this study. The database will be maintained on a secure password-protected hard
drive. Only study personnel will have access to the database. Each participant will be
assigned a unique identification number (ID). A master file linking ID and personal
identifiers will be maintained in a separate location from the main study database; the master
file will be maintained at the study site and not transferred to Duke. No personal health
information will be entered into the main study database; information within the database
will be linked using the ID.
Data can be viewed or extracted using Microsoft Access 2003 (Seattle, WA), or it can be
extracted using standard statistical packages (e.g., SAS, Cary, NC) and downloaded for
manipulation and analysis. Analyses will be conducted according to the plan outlined in
Section 6.0. Only aggregate reports from usability testing will be transferred to Semantic
Bits and MSKCC in order to protect individual level, potentially personally identifying
results.
10.1 Quality Assurance
Weekly registration reports will be generated to monitor patient accruals and
completeness of registration data. Routine data quality reports will be generated to assess
missing data and inconsistencies. Accrual rates and extent and accuracy of evaluations
and follow-up will be monitored periodically throughout the study period and potential
problems will be brought to the attention of the study team for discussion and action.
10.2 Data and Safety Monitoring
Data and safety monitoring audits will be conducted by the study team on a regular basis
throughout the study period. Standard Data and Safety Monitoring (by the DSM
Committee) will not be required, since this protocol fits the definition of clinical research,
not a clinical trial. However, the Principal Investigator and research team intend to
provide the oversight and monitoring that is required for this study. Confidentiality of
each participant's data will be protected with utmost care with all questionnaire data
identified solely by a code number. Study findings will be presented in aggregate form
only, with no reference made to the individual participant's data. Aggregate data will
never be sent to anyone unless in an encrypted file. The Principal Investigator and
research staff will be responsible to identify, review, and report all necessary adverse
events to the IRB and NCI as appropriate. Adverse events are identified through
standard, routine protocol review and clinical assessment of each participant in the study.
Date prepared: 1 August 2009
- Page 14 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
10.3 System Security
The PRO-CTCAE system will be hosted as a web application, which is accessible via
Internet and intranet using the system’s designated user interface and its web/grid service
interface. During software development, the system will be hosted at the National
Cancer Institute in Bethesda, MD. While patient data and user data will be inputted to
the system by study participants, the system will not permanently save responses during
the usability testing phase. Only web analytics will be saved.
The PRO-CTCAE system is being developed to be 21 CFR Part 11 compliant, with final
certification planned once the beta version of the software is fully tested and updated.
The final PRO-CTCAE system will be implemented and managed in accordance with the
Federal Information Security Management Act of 2002 (FISMA). Safeguards, practices,
training and policies developed to comply with all relevant regulations, laws, and policies
are described in detail in the Information System Security Plan (Appendix C).
11.0 PROTECTION OF HUMAN SUBJECTS
This research poses no more than minimal risk to participants. All potential participants will be
informed as to their rights as volunteers in a research study. The right to refuse or withdraw at
any point during the study, without compromising medical and other care will be explained.
Informed consent will be obtained from each participant in the study. The purpose of the study
and potential risks and benefits associated with it will be stated. All data gathered, whether in
paper or electronic format, will be kept in a secured location. Participants will be assigned
identification numbers, so that names will not be connected to any information.
Due to the sensitivity of the issues discussed during the interview, we will conclude each
interview by asking participants if anything mentioned during the interview triggered feelings of
discomfort or distress. In the unlikely event that participants display acute emotional distress
either during the interview or at the end of the interview, appropriate referrals will be made for
clinical care.
11.1 Serious Adverse Event and Adverse Event reporting
This is a minimal risk study, with no study-related adverse events expected. However,
cancer patients enrolled in the study may experience an adverse event related to their illness,
which overlaps in time with their participation in some aspect of this study. It will be made
clear to patients during informed consent that the PRO-CTCAE is a tool under investigation
Date prepared: 1 August 2009
- Page 15 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
and is not intended within this study to be a mechanism for communicating adverse
symptoms to clinical staff. Therefore, patients will be advised to contact their clinician(s)
via standard channels if they are experiencing an adverse symptom. It will be made clear
that symptom information being entered into the system during usability testing is not being
saved or transmitted to a clinician,
12.0 INFORMED CONSENT PROCEDURES
Study personnel will contact interested participants via telephone or in clinic.
Informed consent will be completed on the day of the one-on-one usability testing
prior to the beginning of the session.
A study staff-person will explain to the potential participant the study's purpose,
procedures, risks and benefits.
Potential participants will be informed that the one-on-one usability testing sessions
will take approximately one hour.
Potential participants will be asked to review the informed consent form.
Potential participants will be asked if they have any questions about the consent form,
the study, or their role in assisting us in our research.
If they agree to participate, potential participants will be asked to sign the consent
form
.
12.1 Research Authorization
Procedures for obtaining Research Authorization: Before any protocol-specific
procedures are carried out, investigators and/or designated staff will fully explain the
details of the protocol, study procedures, and the aspects of patient privacy concerning
research specific information. All participants must sign the IRB-approved informed
consent documents before being involved in any study-related activity. Where applicable
(based upon state- or institutional-based requirements), patients must also sign the
Research Authorization component of the informed consent form. The Research
Authorization requires a separate set of signatures from the patient. The original signed
documents will become part of the patient’s medical record, and each patient will receive
a copy of the signed documents.
Date prepared: 1 August 2009
- Page 16 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
13.0 REFERENCES
1. http://www.usability.gov/basics/whatusa.html, accessed 6/1/08.
2. http://en.wikipedia.org/wiki/Usability_testing.
3. Lewis C, Rieman J. Task-Centered User Interface Design: A Practical Introduction,
available via Shareware at http://hcibib.org/tcuid/; 1993.
4. McDaniel AM, Hutchison S, Casper GR, Ford RT, Stratton R, Rembusch M. Usability
testing and outcomes of an interactive computer program to promote smoking cessation
in low income women. Proceedings / AMIA .. 2002;Annual Symposium.:509-513.
5. Sheinfeld Gorin S, Franco R, Hajiani F, Senathirajah Y. Systematic development and
usability testing of a physician-based prostate cancer education program in an African
American community. AMIA .. 2007;Annual Symposium Proceedings/AMIA
Symposium.:1112.
6. Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in
usability testing. Behavior Research Methods, Instruments, & Computers. Aug
2003;35(3):379-383.
7. MacQueen KM ME, Kay K, and Milstein B. . Codebook Development for Team-Based
Qualitative Analysis. Cultural Anthropology Methods. 1998;10(2):31-36.
14.0 APPENDICES
Appendix A: Usability Testing Interface Screenshots
Appendix B: Usability Testing Informed Consents
Appendix C: Information System Security Plan
Date prepared: 1 August 2009
- Page 17 -
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
Appendix A: Usability Testing Interface Screenshots
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
IRB Protocol – PRO CTCAE Usability Testing
DUHS IRB#: Pro00018699
