Docstoc

Refinement of the Patient Reported Outcomes Version of the

Document Sample
Refinement of the Patient Reported Outcomes Version of the Powered By Docstoc
					            Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                          IRB Protocol – PRO CTCAE Usability Testing

                                                                    DUHS IRB#: Pro00018699



                       Refinement of the Patient-Reported Outcomes Version of the
                     Common Terminology Criteria for Adverse Events (PRO-CTCAE):
                                            Usability Testing



                              NON-THERAPEUTIC/DIAGNOSTIC PROTOCOL

                                     Name                         Department/Institution
Site Principal Investigator          Amy P. Abernethy, MD         Department of Medicine, Division of
                                                                    Medical Oncology, Duke University
                                                                    Medical Center (Duke)


Investigators
                                     Jane Wheeler, MSPH           Medical Oncology, Duke




      Please Note: A Consenting Professional must have completed the mandatory Human
      Subjects Education and Certification Program.

      Collaborating Institutions      Memorial Sloan Kettering Cancer Center
                                      Dana-Farber Cancer Institute
                                      M.D. Anderson Cancer Center
                                      Duke University Medical Center
                                      National Cancer Institute




      Date prepared: 1 August 2009
                                              Title Page
        Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                      IRB Protocol – PRO CTCAE Usability Testing

                                                                                               DUHS IRB#: Pro00018699

TABLE OF CONTENTS
1.0      PROTOCOL SUMMARY .............................................................................................. 1
2.0      SCIENTIFIC AIM .......................................................................................................... 2
3.0      BACKGROUND ............................................................................................................ 2
4.0      STUDY DESIGN AND IMPLEMENTATION ............................................................. 3
     4.1    DESIGN .................................................................................................................... 3
            4.1.1 PROJECT LEADERSHIP ...................................................................................... 4
            4.1.2 SETTING ........................................................................................................... 4
            4.1.3 PARTICIPANTS .................................................................................................. 4
     4.2   METHODS ................................................................................................................. 5
            4.2.1 RECRUITMENT AND ENROLLMENT ................................................................... 5
            4.2.2 TRAINING AND SITE PREPARATION .................................................................. 6
            4.2.4 DETAIL OF METHODS ....................................................................................... 7
     4.3 DATA COLLECTION ................................................................................................ 9
            4.3.1 DATA COLLECTION FORMS ............................................................................... 9
            4.3.2 RECORDING ................................................................................................. 10
            4.3.3 POSSIBLE FUTURE METHODS OF DATA COLLECTION ...................................... 10
5.0      ELIGIBILITY CRITERIA............................................................................................ 11
     5.1 INCLUSION CRITERIA (PATIENTS) ..................................................................... 11
     5.2 EXCLUSION CRITERIA (PATIENTS) .................................................................... 11
     5.3 INCLUSION CRITERIA (PROVIDERS) .................................................................. 11
     5.4 EXCLUSION CRITERIA (PROVIDERS)................................................................. 11
6.0      ANALYSIS PLAN ....................................................................................................... 12
7.0      TOXICITIES/SIDE EFFECTS ..................................................................................... 13
8.0      PRIMARY OUTCOMES ............................................................................................. 13
9.0      CRITERIA FOR REMOVAL FROM STUDY ............................................................ 13
10.0     DATA MANAGEMENT............................................................................................. 13
     10.1    QUALITY ASSURANCE .................................................................................... 14
     10.2    DATA AND SAFETY MONITORING ............................................................... 14
     10.3    SYSTEM SECURITY .......................................................................................... 15
11.0     PROTECTION OF HUMAN SUBJECTS ................................................................... 15
     11.1 SERIOUS ADVERSE EVENT AND ADVERSE EVENT REPORTING ................. 15
12.0     INFORMED CONSENT PROCEDURES ................................................................... 16
     12.1    RESEARCH AUTHORIZATION ........................................................................ 16
13.0     REFERENCES ............................................................................................................. 17
14.0     APPENDICES .............................................................................................................. 17
       APPENDIX A: USABILITY TESTING INTERFACE SCREENSHOTS……….App A
       APPENDIX B: USABILITY TESTING INFORMED CONSENT………………App B
       APPENDIX C: INFORMATION SYSTEM SECURITY PLAN…………………App C




Date prepared: 1 August 2009
                                                       Table of Contents
      Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                    IRB Protocol – PRO CTCAE Usability Testing

                                                                       DUHS IRB#: Pro00018699



1.0   PROTOCOL SUMMARY

  Currently, adverse symptom events occurring in National Cancer Institute (NCI)-sponsored
  clinical trials are reported by clinical staff using items from the Common Terminology
  Criteria for Adverse Events (CTCAE). A recognition of methodological problems with the
  current collection of CTCAE symptom data, the availability of technologies that facilitate
  patient self-reporting, and the release of an FDA guidance recommending standards for
  patient-reported outcomes (PROs) in the regulatory setting have, in concert, prompted
  research on PRO approaches to the CTCAE. As it currently stands, the CTCAE contains
  1,059 discrete items, approximately 77 of which contain a subjective component sufficient to
  be amenable to patient self-reporting.

  This protocol represents one component of a recently funded NCI contract (N02-PC-85002-
  29; Principal Investigator, Basch; Co-Investigator, Abernethy). This protocol describes the
  usability testing component of the larger PRO-CTCAE initiative, which includes
  creating/validating new adverse symptom items for use in NCI-sponsored trials; conducting
  cognitive interviews to assure patient understanding of items; and developing a software
  system to administer these items. Specifically, this protocol assesses the usability of the
  software system.

  The specific objective the of usability testing portion of the contract is to evaluate the usability
  of the technology interface for collecting the PRO data of the newly developed PRO-CTCAE
  system; this protocol follows completion of cognitive interviews, which will examine
  comprehension of the newly-developed items as well as the extent to which the items
  correspond to the concepts of interest for each PRO.

  For purposes of this study, usability of the technology will be defined as its capacity to allow
  users to carry out their tasks – in this case, response to PRO-CTCAE items using the software
  interface – safely, effectively, efficiently, and enjoyably.1 Usability testing will comprise
  three main components: (1) observation of users interacting with the system, guided by “think
  aloud” protocols, and the taking of extensive field notes; (2) the gathering of user feedback
  through brief semi-structured interviews with users immediately following their use of the
  system; and, (3) web analytics including tracking for different web pages, with clickstream
  analysis.

  Duke will serve as lead site for usability testing, functioning as central coordinating site and
  data repository for the Task 6 study. SemanticBits LLC, under a contract from the NCI, is
  creating the software; this software will be open source and will comply with NCI security
  and privacy specifications.




Date prepared: 1 August 2009
                                             - Page 1 -
        Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                      IRB Protocol – PRO CTCAE Usability Testing

                                                                        DUHS IRB#: Pro00018699



  The end result of this usability testing protocol will be assurance that target users of the newly
  developed PRO-CTCAE system – namely, (a) patients with diverse characteristics, and (b)
  clinical research staff – can conduct the expected tasks using this system.

2.0      SCIENTIFIC AIM

  The aim of this study is to evaluate the usability of the technology interface developed to
  collect the PRO data elements of the PRO-CTCAE system, once it has been built and
  modified to accommodate the learning garnered during cognitive interviews (Task 4 of the
  overarching project).


3.0      BACKGROUND

      Usability of a software product or technology system is broadly defined as the capacity of
      that product/system to allow users to carry out their tasks safely, effectively, efficiently, and
      enjoyably.1 A “black-box” technique, usability testing aims to observe people, representative
      of future users, interacting with a product/system while they are in a realistic environment
      that simulates the product/system’s “real world” use. The process seeks to discover
      problems, collect quantitative data on users’ performance (e.g., time to task completion, error
      rate), determine users’ satisfaction, and identify areas for improvement. Usability testing
      generally involves collection of data with respect to four areas of user response: efficiency,
      accuracy, recall, and emotional response.2

      In a usability test, representative users perform typical tasks with the product/system;
      observers (i.e., the testers) watch, listen, and take notes but refrain from instructing or
      guiding the user while he/she performs the tasks. Usability testing often follows a “think
      aloud” protocol, a method first developed by Clayton Lewis3. In a think aloud protocol,
      users verbally describe their processes as they perform the requisite tasks with the
      product/system. For example, users talk through what they are seeing, doing, thinking, and
      feeling as they undergo the task. The think aloud protocol allows an observer to gather data
      on the entire process of product/system use, rather than on end results alone. The observer
      objectively records everything that the user says without adding any interpretation.
      Additional data is often obtained through audio- or video-taping the session, and
      supplementing observer notes with information gained through review of the recording.

      Usability testing has been employed to test various new interventions among patients of
      diverse disease and demographic characteristics. A large number of usability testing studies
      has been published in the medical literature, reporting on the usability of a range of
      information systems, user interfaces, web-based applications, and software. Of relevance to
      this usability testing protocol are studies that involved minority and low-income individuals,
      as it is important that the PRO-CTCAE system under development accommodate these
Date prepared: 1 August 2009
                                               - Page 2 -
        Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                      IRB Protocol – PRO CTCAE Usability Testing

                                                                        DUHS IRB#: Pro00018699



      potential users. McDaniel et al. performed usability testing of an interactive computer-
      mediated smoking cessation program for inner-city women (n=100).4 Sheinfeld Gorin et al.
      conducted usability testing of a physician-based prostate cancer education program in an
      African American community.5 Also of note is a study by Faulkner et al. which challenged
      the general assumption that a sample of five participants is adequate to determine usability
      concerns and performance. The investigators included 60 participants in a set of usability
      tests, randomly defining subsets of five participants from the total pool. While some
      randomly selected sets of five participants identified 99% of problems, other sets found 55%.
      Accuracy in identifying problems increased with the number of users included in the set.6

      This protocol focuses on the usability of the newly developed PRO-CTCAE system. The
      study is designed to ensure that the targeted users can conduct the expected task, namely, to
      use the technology interface to report on their symptoms via response to PRO-CTCAE items.
      Targeted users include: (a) patients from diverse backgrounds with varying educational
      levels, computer experience, and physical illness; and, (b) clinical research personnel, such as
      nurses, clinical research coordinators, research assistants, and physicians. Based on the four
      standard domains examined in usability testing (efficiency, accuracy, recall, and emotional
      response), measures have been defined so as assess five indicators of usability that pertain to
      the PRO-CTCAE technology interface.
          1. ease of learning – How fast can a user who has never seen the user interface before
              learn it sufficiently well to accomplish basic tasks?
          2. efficiency of use - Once an experienced user has learned to use the system, how fast
              can he or she accomplish tasks required by the system?
          3. memorability - If a user has used the system before, can he or she remember enough
              to use it effectively the next time, or must the user re-learn the system?
          4. error frequency and severity - How often do users make errors while using the
              system, how serious are these errors, and how do users recover from these errors?
          5. subjective satisfaction - How much do users like using the system?
      All five indicators contribute important information to each of the four standard domains of
      usability testing.


4.0      STUDY DESIGN AND IMPLEMENTATION

4.1       Design

This usability testing protocol replicates real-world scenarios in which future users will complete
the PRO-CTCAE. Testing will take place in two rounds of one-on-one (user/observer) usability
testing sessions, such that experience gained and lessons learned in the first round can inform
refinement of the system for further testing in the second round.


Date prepared: 1 August 2009
                                               - Page 3 -
      Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                    IRB Protocol – PRO CTCAE Usability Testing

                                                                      DUHS IRB#: Pro00018699



Usability testing will collect data from clinical research staff and patients in three ways: (1)
through observation of users interacting with the system, with data recorded on a data collection
form and in extensive field notes; (2) during semi-structured interviews with users immediately
following the observation (#1, preceding), with data collected on a standardized interview form
and supplemented with review of audiotapes of the session; and, (3) through web analytics
including tracking for different web pages, with clickstream analysis. These sources will capture
information from all five critical usability domains: ease of learning, efficiency of use,
memorability, error frequency and severity, and subjective satisfaction. Web analytics will
provide quantitative information on efficiency, memorability, and errors.

Semantic Bits will customize software for this study, including a form builder for creation of the
data collection forms, a user interface for both patients and clinical research staff, reporting
functions, and a study calendar. Examples of the user interface are provided in Appendix A. The
software will be hosted on a secure server at the National Cancer Institute (NCI) during the
development of the software, although no patient data will be stored during the usability testing;
only web analytics information will be collected for usability analysis.

4.1.1 Project leadership
Usability testing will be led by Dr. Abernethy (Task Leader) and Drs. Hay and Mendoza (Deputy
Task Leaders) in conjunction with Dr. Basch (Principal Investigator). The Task 6 team has
communicated frequently with Mr. Chilukuri and SemanticBits during the system design phase,
to understand and inform the platform and system design characteristics, recommend
modifications as needed, and plan for usability testing of the beta model of the PRO-CTCAE
assessment. Located at the central coordinating site for this study (Duke), Dr. Abernethy will
oversee accrual and data collection at the sites, and integration of these data into the centralized
data repository.

4.1.2 Setting
The usability testing plan is carefully designed so as to create scenarios that realistically
represent future uses of the system. The settings selected for testing will enable study personnel
to observe users’ in situ interactions with the technology. Thus, at each site, usability tests will
be conducted in the environment where the system is expected to be most commonly used.
These settings primarily include, but are not limited to, the sites’ outpatient oncology clinics.
The inclusion of inpatient tests will ensure that the system accommodates data collection from
hospitalized patients.

4.1.3 Participants
Participants will be recruited to reflect the composition of the population of future users. Two
groups of users will be included: clinical research staff and patients. In total, usability testing
will involve 60 clinical research staff evaluations and 120 patient evaluations, divided
approximately equally between the two rounds of usability testing (i.e., 30 clinical research staff

Date prepared: 1 August 2009
                                             - Page 4 -
      Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                    IRB Protocol – PRO CTCAE Usability Testing

                                                                         DUHS IRB#: Pro00018699



and 60 patients participating in each round). Among clinical research staff, each round will
include 10 nurses, 10 clinical research assistants, and 10 physician investigators. A distinct,
specific definition for each clinical research staff title will be developed and used to categorize
participants; this specificity will avert any potential issues that might arise if different institutions
use the same role terminology to describe different positions.

Among patients, each round will minimally include 60 patients. We will enroll at least 10
individuals with each of the following characteristics: over the age of 75 years, less than a high
school education, non-white race/ethnicity, and inpatient or severely ill (ECOG >2). At least 30
participants will have metastatic disease (vs. non-metastasized cancer). (Note: In certain cases,
the same individual may represent more than one of these characteristics.) After 40 patients have
been involved in each round of usability testing, we will evaluate the composition of the
participant group and enrich it with more patients of a specific characteristic (e.g., low education
level, if this characteristic is under-represented) as warranted. The purpose of this enrichment
will be to ensure that all of the designated characteristics are adequately represented in the
simulated user population.

Participants will be recruited from three institutions: Memorial Sloan Kettering Cancer Center
(MSKCC), M.D. Anderson Cancer Center, and Duke Comprehensive Cancer Institute, and their
respective satellite/network clinics. This inclusion of multiple institutions will ensure a broad
representation of academic and community-based users with diverse demographics, as well as
multiple oncology-based disciplines including medical oncology and radiation oncology. The
anticipated number of patients and clinicians to be enrolled at each site appear in Table 1, below.


   Table 1: Number of participants in usability testing by institution, per round

             Institution                                     Participants
                                                     Patients    Clinicians/Researchers
             Duke                                      25                  10
             MSKCC                                     10                  10
             MD Anderson                               25                  10


4.2    Methods
4.2.1 Recruitment and enrollment
Participants who meet eligibility criteria (Section 5.0, below) will be recruited at each site by
site-based study personnel, in accordance with best clinical research practice. Interviewers will
work closely with clinic staff and physicians to identify potentially eligible and interested
patients. These patients may receive a telephone call from clinic staff in advance of their clinic
appointment to describe the study to them, or they may be approached in the waiting room as
Date prepared: 1 August 2009
                                               - Page 5 -
      Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                    IRB Protocol – PRO CTCAE Usability Testing

                                                                       DUHS IRB#: Pro00018699



they wait for a regularly scheduled clinic appointment. In each case, the patient’s primary
physician will be aware of the study and will have given study personnel his/her agreement to
approach the patient. Once in the clinic, the study staff-person will briefly discuss the study, its
purpose and procedures, to ascertain whether or not the patient is interested in participating.

Patient registration will be coordinated by the Clinical Trials Manager with the Duke Cancer
Care Research Program at Duke University Medicine Center. Dr. Amy Abernethy will provide
oversight, and will be in frequent contact with site-based study personnel and data managers.

No participant will be considered enrolled until he/she has provided informed consent. Consent
will be obtained using the form provided in Appendix B.

4.2.2 Training and site preparation
Prior to recruiting participants for the usability testing, Drs. Abernethy, Hay, Mendoza, and
Basch, together with experienced Duke staff trained in usability testing, will work with Mr.
Chilukuri and SemanticBits programmers to finalize the think aloud protocol and to ensure that
the web tracking software is incorporated into the PRO-CTCAE site.

A standard usability testing manual will be used for training site-based study personnel. This
manual will fully describe the conduct of one-on-one usability testing sessions including
observation, data collection/recording, semi-structured interviewing, and data management. The
manual will ensure consistent practices for usability testing across the three sites. Dr. Abernethy
and the Duke team with expertise in usability testing will train research staff at each site in the
protocol, expectations, and data collection processes. A copy of the manual will be left at each
site as a reference guide for site-based study personnel, and the Duke team will remain available
by email and telephone to answer any subsequently arising questions.

Training will explicitly describe data collection procedures under this protocol. The importance
of field notes and extensive written data collection will be emphasized, in addition to proper
recording of quantitative data using the data collection form (Section 4.3, below). Training will
utilize role plays and practice sessions (observation, interview), to ensure a realistic and
pragmatic approach that allows site-based study personnel to develop competence and
confidence in the protocol. Research staff will be allowed to hand-write or dictate field notes,
though they will need to transfer handwritten field notes into typewritten text. Training will
inform site-based personnel about data management procedures.

If site-based study personnel do not feel comfortable conducting the usability tests after the
training, Duke research staff will guide the process through video conferencing and/or will travel
to the site to provide further onsite training and facilitate participant accrual.

4.2.3 Overview of methods

Date prepared: 1 August 2009
                                             - Page 6 -
      Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                    IRB Protocol – PRO CTCAE Usability Testing

                                                                      DUHS IRB#: Pro00018699



This usability testing protocol will entail one-on-one interviews between participants/users and a
member of the clinical research staff. Each participant’s involvement in the study will consist of
approximately one hour of time, spent in the clinic or hospital using the system and undergoing a
brief semi-structured interview. Usability tests and interviews will be conducted in a private
space identified by the study team in coordination with administrators and/or clinicians at each
site.

4.2.4 Detail of methods
Procedures for usability testing are as follows:
    (1) Written informed consent will be obtained from each participant prior to any data
        collection. (See Appendix B: Usability Testing Informed Consent.)

   (2) The participant’s will first be engaged in a one-on-one usability testing session, in which
       the user/participant navigates the technology interface (software housing the PRO-
       CTCAE draft questions, or clinical research interface) and describes his/her processes
       according to a think aloud protocol, while the observer/study staff-person records
       specified data. Each session will be conducted “in situ,” i.e., in the setting where that
       user type is ultimately expected to use the software system; for example, patients will
       undergo usability testing in clinic waiting area or hospital room while a clinical research
       coordinator will complete the testing in his/her research office.

       Usability testing sessions with patient participants/users will entail responding to
       approximately 25 draft PROs using an electronic platform (a desktop, laptop, or wireless
       tablet personal computer). The software will ask the patient/user to answer a series of
       questions, one question per screen; each question will address a particular symptom, the
       experience related to a symptom, the intensity of a symptom, or the symptom’s
       interference with daily life. The patient will answer each question as instructed by the
       system, most likely through touching the screen at an appropriate place. While the user
       completes this draft PRO assessment, the observer/study staff-person will record data
       onto a data collection form and will take field notes. Following a standardized script, the
       study staff-person will encourage the user to think aloud about the software, use of the
       system, and attributes of the system as well as to narrate his/her interactions with the
       system. The session will be audio-recorded to capture this verbal information; these
       qualitative results will be transcribed and collated with the transcribed field notes.

       Similarly, the clinical research personnel participant/user will test the software’s research
       interface. The usability testing protocol for this group of users will entail similar
       procedures as for the patient participant/user group: a standardized think aloud script,
       observation and field note taking on the part of study staff, and audio-recording.
       However, clinical research personnel will not use the system to respond to the draft
       ePROs (as will patient participants). Instead, they will use the software system to:

Date prepared: 1 August 2009
                                            - Page 7 -
     Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                   IRB Protocol – PRO CTCAE Usability Testing

                                                                    DUHS IRB#: Pro00018699



      nominate symptom items for adverse event reporting; create an adverse event reporting
      survey using the symptoms which they nominated; create a schedule for presentation of
      surveys to patients; and, create a plan for follow-up and monitoring of symptoms,
      including severe symptoms that exceed risk threshold scores.

      Immediately after the participant (either patient or clinical research personnel) interacts
      directly with the software interface, the study staff-person will conduct a brief semi-
      structured interview with him/her to supplement information gained through the think
      aloud protocol, observation, and field notes. Semi-structured interviews will consist of
      ten open-ended questions related to the five indicators of usability listed in Section 3.0,
      above. A different list of questions will be created for each of the two groups (patients,
      clinical research personnel). Example questions posed to patient participants/users are:
           Is this software easy to use for reporting symptoms that you may be experiencing?
           Does the system make instructions clear and easy to follow?
           What about the system would you suggest changing in order to make it easier to
              use?
           How well did you like using the system?
           Can you suggest ways that we might make the system more likeable?

      Example questions posed to clinical research personnel participants/users are:
          Is it obvious how to initiate a new study using this software system?
          Is it easy to schedule surveys for patients using this system?
          Is the scheduling system convenient and useful?
          Are email notifications provided by the system clear and feasible in a busy
            clinical practice?

      Following the completion of each one-on-one session, the study staff-person will review
      his/her data and field notes, before passing them along to the study coordinator.

   (3) Quantitative data will be captured through web analysis of software use. Specifically,
       with respect to the patient participant/user group, clickstream analysis will capture the
       amount of time which the user spends per screen (i.e., PRO-CTCAE item), number of
       errors and time taken to correct errors, and total time spent on the PRO-CTCAE survey.
       These data will be housed on secure servers at the NCI designated to house the PRO-
       CTCAE system. Notably, during this study, only a unique identifier number will be
       associated with each patient’s data on these servers, and no identifying patient
       information will be stored on the NCI servers. Hence, each participant/user will be
       assigned a unique identifier; de-identified datasets will be transferred to Duke for
       analysis.



Date prepared: 1 August 2009
                                           - Page 8 -
        Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                      IRB Protocol – PRO CTCAE Usability Testing

                                                                        DUHS IRB#: Pro00018699



      (4) Web analytics data collected in the one-on-one usability testing sessions will be de-
          identified. Each participant/user will be assigned a unique identifier; only the Task
          Leader and authorized study personnel will have access to the identifying schema. Data
          will be entered into a secure study database. Summary data (de-identified) will be posted
          in a password-protected area of the study wiki site. Audiotapes and Data Collection
          Forms will remain at the local site, under locked containment, for six years and then
          destroyed.

      (5) After completion of a first round of one-on-one usability tests (#2 above), engaging
          approximately 30 clinical research staff and at least 60 patients, data will be reviewed to
          identify major usability issues that can be addressed by modification to the system prior
          to the second round. The system will be refined accordingly, and prepared for the second
          round of one-on-one usability testing.

      (6) A second round of one-on-one usability testing will proceed as described in #2, above,
          and will again engage approximately 30 clinical research staff and 60 patients.


4.3     Data collection

4.3.1 Data collection forms
Five forms will be used to collect data for the usability component of the overarching PRO-
CTCAE project. The forms, listed below, will be created specifically for this study. (Note that it
is not possible to develop these forms until the software for the system has been finalized.)

         Participant Screening and Registration Log: This form will be used to track participants
         screened, contacted regarding participation, reason for refusal (when a person does not
         agree to participate), and enrollment in the trial (when a person agrees to participate).
         For individuals who do not agree to participate, we will only record a unique screening
         number (so that we can track total number screened) and reason for refusal; personal
         health information will not be recorded for non-consenting individuals. For individuals
         who consent to participate, we will record a unique study linking identification number
         (ID), demographics, education level, experience with computers and internet, and basic
         illness characteristics (cancer type, stage, performance status, current treatment).

         Patient Usability Testing Data Collection Form: This structured form will be used by the
         site-based study personnel for collecting data during the one-on-one usability testing
         sessions with patient participants. The form will contain blank space for free-form field
         notes.



Date prepared: 1 August 2009
                                               - Page 9 -
      Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                    IRB Protocol – PRO CTCAE Usability Testing

                                                                      DUHS IRB#: Pro00018699



       Clinical Research Personnel Usability Testing Data Collection Form: This structured
       form will be used by the site-based study personnel for collecting data during the one-on-
       one usability testing sessions with clinical research personnel participants. The form will
       contain blank space for free-form field notes.

       Patient Usability Testing Interview Form: This semi-structured form will be used by the
       site-based study personnel for recording participants’ responses to the ten questions
       posed in the brief interview following the one-on-one usability testing session.

       Clinical Research Personnel Usability Testing Interview Form: This semi-structured
       form will be used by the site-based study personnel for recording participants’ responses
       to the ten questions posed in the brief interview following the one-on-one usability
       testing session.


4.3.2 Recording
All one-on-one usability testing sessions will be audio-recorded. Audio-recording will allow for
the capture of emotion in participants’ verbal expression, which may add depth to the
information collected. Additionally, study personnel will review the audio-tapes against
information in the field notes and on the data collection forms, to ensure that all relevant data are
captured. Transcription is not planned, as the audiotapes are intended to supplement already
collected data; instead, study personnel will listen to the audiotapes and will add any new data
from that medium to the data collection form and/or field notes. If it becomes apparent that
transcription would contribute to the results of this study, we will then proceed with it.
Recording will be accomplished using digital recording equipment. The audio recordings will be
downloaded into files on protected computers at each site (Duke, MSKCC, MD Anderson), and
de-identified data will be sent to Duke.

4.3.3 Possible future methods of data collection
We do not anticipate needing to incorporate eye tracking protocols or video-based data collection
of screen movements into this study. However, if results of the first round of usability testing
indicate a need for the further information provided by these strategies, we will prepare a
protocol amendment to add either or both of those elements, will modify the study description
(for recruitment purposes) and informed consent accordingly, and will obtain IRB approval prior
to using these strategies in round two of usability testing.




Date prepared: 1 August 2009
                                            - Page 10 -
        Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                      IRB Protocol – PRO CTCAE Usability Testing

                                                                          DUHS IRB#: Pro00018699



5.0      ELIGIBILITY CRITERIA

      Patients will be eligible for the study if they are currently undergoing chemotherapy and/or
      radiotherapy for cancer, whether for primary therapy with curative intent or palliative intent.
      The symptoms experienced by patients receiving adjuvant chemotherapy are most likely
      attributable to the therapeutic agents, thus the adjuvant setting is optimal in evaluating PROs
      for treatment toxicity. However, patients with metastatic disease are also eligible because
      these individuals are included in initial drug development and examination of adverse events.

         5.1        Inclusion criteria (patients)

                   diagnosed with cancer
                   undergoing chemotherapy or radiotherapy with either curative or palliative intent
                   receiving care at one of the three study sites (Duke, MSKCC, MD Anderson)
                   aged ≥18 years
                   fluent in English
                   able to provide informed consent

         5.2        Exclusion criteria (patients)

                   significant cognitive impairment

      Clinician eligibility is intentionally broad, to include a representation of real-world clinical
      providers who may potentially use the PRO-CTCAE system in the future.

         5.3        Inclusion criteria (providers)

                   involved in clinical care of cancer patients and/or research enrolling cancer
                    patients
                   employed at one of the three study sites (Duke, MSKCC, MD Anderson)
                   fluent in English
                   able to provide informed consent

         5.4       Exclusion criteria (providers)

                   involvement in this study protocol or any other aspect of the overarching PRO-
                    CTCAE project




Date prepared: 1 August 2009
                                                 - Page 11 -
       Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                     IRB Protocol – PRO CTCAE Usability Testing

                                                                       DUHS IRB#: Pro00018699



6.0     ANALYSIS PLAN

      Analytic methods will correspond to the testing round (1 or 2), participant group (patient,
      clinical research personnel), and participant characteristics.

      Observational data obtained through field testing is the most critical information. Typed
      field notes generated through think aloud protocols and observation will be carefully
      reviewed and thematically coded using a codebook analysis approach.7 The codebook
      structure will include the code, a basic definition, a full definition, guidelines for using the
      code, and examples. Codes will be categorized into themes and sub-themes, which will
      minimally include the five usability domains: ease of learning, efficiency of use,
      memorability, error frequency and severity, and subjective satisfaction. Text will be
      reviewed by Duke research staff trained in codebook analysis, and codes applied. A
      relational database will be used to facilitate code development and application of code to
      text. The codebook analysis will provide (a) a qualitative summary of the types of problems
      encountered, and (b) a quantitative summary of the aggregate number of times a particular
      problem was encountered.

      Web analytics and clickstream analyses will supplement the field notes and user response
      data, providing additional information on efficiency, memorability, and errors.
      Benchmarking/tracking information will include, but not be limited to: (1) total time
      required to use the PRO-CTCAE system for new and repeat users; (2) aspects of the system
      that create unnecessary delays; (3) order of pages viewed and number of times the user
      needs to “go back;” and, (4) number of missed or skipped items.

      Results will be reported in aggregate and by user type (patient, clinical research staff), to
      ensure that different categories of users are equally capable of interacting with the system as
      intended.

      Summaries of the information captured in the first round of usability testing will be
      reviewed with Mr. Chilukuri and the programmers from SemanticBits; this review will
      inform iterative improvements in the system. The modified PRO-CTCAE system will be
      subjected to a second round of usability testing to ensure that issues emerging in the first
      round of testing were addressed, and that further development is not needed. Additionally,
      information gathered through both rounds of usability testing will guide refinement of the
      platform in preparation for the next phase of the overarching PRO-CTCAE initiative, a
      validation study and a feasibility study in the cooperative group setting.




Date prepared: 1 August 2009
                                              - Page 12 -
        Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                      IRB Protocol – PRO CTCAE Usability Testing

                                                                        DUHS IRB#: Pro00018699



7.0      TOXICITIES/SIDE EFFECTS

  There are no anticipated physical risks or side effects associated with participation in this
  study. In the unlikely event of extreme psychological distress observed or expressed by
  participants, site-based study personnel will refer the patient(s) directly the Principal
  Investigator at that site. Each participant will receive contact information for these
  investigators, and be informed that he/she may contact one of the investigators to discuss any
  concerns related to the study. Additionally, participants will be informed during the
  consenting process that they may discontinue participation in the study at any time.


8.0      PRIMARY OUTCOMES

  The primary outcomes associated with this usability testing study include documentation of
  the ease of learning, efficiency of use, memorability, error frequency and severity, and
  satisfaction of representative users of the technology-based system for responding to the
  newly developed PRO-CTCAE survey.


9.0      CRITERIA FOR REMOVAL FROM STUDY

  In the unlikely event that study personnel observe acute distress in an individual during that
  person’s participation in this study, he/she will ask the participant whether he/she wants to
  discontinue participation and withdraw from the study. All participants will be given the
  option to terminate their participation at any time.


10.0     DATA MANAGEMENT

      Usability testing data (collected through the semi-structured interviews) is separate from the
      software system being tested (via clickstream and web analytics). The site at Duke
      University Medical Center will be responsible for data management and analysis of the
      usability testing information. Data forms collected at each site will be sent to Duke for
      management. While patient data and user data will be inputted to the PRO-CTCAE system
      by study participants, the system will not permanently save responses during the usability
      testing phase (see Section 10.3 below). There is one aspect of participant level data that will
      be transferred to Duke for analysis – specifically web analytics and clickstream information.
      These data will transferred as de-identified datasets to Duke for analysis. A master dataset
      linking unique ID with participant personal information will be maintained by Semantic Bits
      and transferred to Duke separately by secure transfer.



Date prepared: 1 August 2009
                                              - Page 13 -
     Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                   IRB Protocol – PRO CTCAE Usability Testing

                                                                     DUHS IRB#: Pro00018699



   At Duke, an ANSI-compliant Microsoft Access 2003 relational database (Seattle, WA),
   maintained on a server behind the Duke firewall, will be created for the data collected in the
   course of this study. The database will be maintained on a secure password-protected hard
   drive. Only study personnel will have access to the database. Each participant will be
   assigned a unique identification number (ID). A master file linking ID and personal
   identifiers will be maintained in a separate location from the main study database; the master
   file will be maintained at the study site and not transferred to Duke. No personal health
   information will be entered into the main study database; information within the database
   will be linked using the ID.

   Data can be viewed or extracted using Microsoft Access 2003 (Seattle, WA), or it can be
   extracted using standard statistical packages (e.g., SAS, Cary, NC) and downloaded for
   manipulation and analysis. Analyses will be conducted according to the plan outlined in
   Section 6.0. Only aggregate reports from usability testing will be transferred to Semantic
   Bits and MSKCC in order to protect individual level, potentially personally identifying
   results.


      10.1    Quality Assurance

      Weekly registration reports will be generated to monitor patient accruals and
      completeness of registration data. Routine data quality reports will be generated to assess
      missing data and inconsistencies. Accrual rates and extent and accuracy of evaluations
      and follow-up will be monitored periodically throughout the study period and potential
      problems will be brought to the attention of the study team for discussion and action.

      10.2    Data and Safety Monitoring

      Data and safety monitoring audits will be conducted by the study team on a regular basis
      throughout the study period. Standard Data and Safety Monitoring (by the DSM
      Committee) will not be required, since this protocol fits the definition of clinical research,
      not a clinical trial. However, the Principal Investigator and research team intend to
      provide the oversight and monitoring that is required for this study. Confidentiality of
      each participant's data will be protected with utmost care with all questionnaire data
      identified solely by a code number. Study findings will be presented in aggregate form
      only, with no reference made to the individual participant's data. Aggregate data will
      never be sent to anyone unless in an encrypted file. The Principal Investigator and
      research staff will be responsible to identify, review, and report all necessary adverse
      events to the IRB and NCI as appropriate. Adverse events are identified through
      standard, routine protocol review and clinical assessment of each participant in the study.



Date prepared: 1 August 2009
                                           - Page 14 -
        Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                      IRB Protocol – PRO CTCAE Usability Testing

                                                                        DUHS IRB#: Pro00018699




         10.3     System Security

         The PRO-CTCAE system will be hosted as a web application, which is accessible via
         Internet and intranet using the system’s designated user interface and its web/grid service
         interface. During software development, the system will be hosted at the National
         Cancer Institute in Bethesda, MD. While patient data and user data will be inputted to
         the system by study participants, the system will not permanently save responses during
         the usability testing phase. Only web analytics will be saved.

         The PRO-CTCAE system is being developed to be 21 CFR Part 11 compliant, with final
         certification planned once the beta version of the software is fully tested and updated.
         The final PRO-CTCAE system will be implemented and managed in accordance with the
         Federal Information Security Management Act of 2002 (FISMA). Safeguards, practices,
         training and policies developed to comply with all relevant regulations, laws, and policies
         are described in detail in the Information System Security Plan (Appendix C).

11.0     PROTECTION OF HUMAN SUBJECTS

This research poses no more than minimal risk to participants. All potential participants will be
informed as to their rights as volunteers in a research study. The right to refuse or withdraw at
any point during the study, without compromising medical and other care will be explained.
Informed consent will be obtained from each participant in the study. The purpose of the study
and potential risks and benefits associated with it will be stated. All data gathered, whether in
paper or electronic format, will be kept in a secured location. Participants will be assigned
identification numbers, so that names will not be connected to any information.

Due to the sensitivity of the issues discussed during the interview, we will conclude each
interview by asking participants if anything mentioned during the interview triggered feelings of
discomfort or distress. In the unlikely event that participants display acute emotional distress
either during the interview or at the end of the interview, appropriate referrals will be made for
clinical care.


         11.1 Serious Adverse Event and Adverse Event reporting

       This is a minimal risk study, with no study-related adverse events expected. However,
       cancer patients enrolled in the study may experience an adverse event related to their illness,
       which overlaps in time with their participation in some aspect of this study. It will be made
       clear to patients during informed consent that the PRO-CTCAE is a tool under investigation

Date prepared: 1 August 2009
                                              - Page 15 -
        Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                      IRB Protocol – PRO CTCAE Usability Testing

                                                                       DUHS IRB#: Pro00018699



       and is not intended within this study to be a mechanism for communicating adverse
       symptoms to clinical staff. Therefore, patients will be advised to contact their clinician(s)
       via standard channels if they are experiencing an adverse symptom. It will be made clear
       that symptom information being entered into the system during usability testing is not being
       saved or transmitted to a clinician,

12.0     INFORMED CONSENT PROCEDURES

            Study personnel will contact interested participants via telephone or in clinic.
            Informed consent will be completed on the day of the one-on-one usability testing
             prior to the beginning of the session.
            A study staff-person will explain to the potential participant the study's purpose,
             procedures, risks and benefits.
            Potential participants will be informed that the one-on-one usability testing sessions
             will take approximately one hour.
            Potential participants will be asked to review the informed consent form.
            Potential participants will be asked if they have any questions about the consent form,
             the study, or their role in assisting us in our research.
            If they agree to participate, potential participants will be asked to sign the consent
             form
  .

         12.1    Research Authorization

         Procedures for obtaining Research Authorization: Before any protocol-specific
         procedures are carried out, investigators and/or designated staff will fully explain the
         details of the protocol, study procedures, and the aspects of patient privacy concerning
         research specific information. All participants must sign the IRB-approved informed
         consent documents before being involved in any study-related activity. Where applicable
         (based upon state- or institutional-based requirements), patients must also sign the
         Research Authorization component of the informed consent form. The Research
         Authorization requires a separate set of signatures from the patient. The original signed
         documents will become part of the patient’s medical record, and each patient will receive
         a copy of the signed documents.




Date prepared: 1 August 2009
                                              - Page 16 -
       Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                     IRB Protocol – PRO CTCAE Usability Testing

                                                                  DUHS IRB#: Pro00018699



13.0   REFERENCES


1.     http://www.usability.gov/basics/whatusa.html, accessed 6/1/08.
2.     http://en.wikipedia.org/wiki/Usability_testing.
3.     Lewis C, Rieman J. Task-Centered User Interface Design: A Practical Introduction,
       available via Shareware at http://hcibib.org/tcuid/; 1993.
4.     McDaniel AM, Hutchison S, Casper GR, Ford RT, Stratton R, Rembusch M. Usability
       testing and outcomes of an interactive computer program to promote smoking cessation
       in low income women. Proceedings / AMIA .. 2002;Annual Symposium.:509-513.
5.     Sheinfeld Gorin S, Franco R, Hajiani F, Senathirajah Y. Systematic development and
       usability testing of a physician-based prostate cancer education program in an African
       American community. AMIA .. 2007;Annual Symposium Proceedings/AMIA
       Symposium.:1112.
6.     Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in
       usability testing. Behavior Research Methods, Instruments, & Computers. Aug
       2003;35(3):379-383.
7.     MacQueen KM ME, Kay K, and Milstein B. . Codebook Development for Team-Based
       Qualitative Analysis. Cultural Anthropology Methods. 1998;10(2):31-36.




14.0   APPENDICES

Appendix A: Usability Testing Interface Screenshots
Appendix B: Usability Testing Informed Consents
Appendix C: Information System Security Plan




Date prepared: 1 August 2009
                                          - Page 17 -
   Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
                 IRB Protocol – PRO CTCAE Usability Testing

                                                     DUHS IRB#: Pro00018699



Appendix A: Usability Testing Interface Screenshots
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
              IRB Protocol – PRO CTCAE Usability Testing

                                                  DUHS IRB#: Pro00018699
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
              IRB Protocol – PRO CTCAE Usability Testing

                                                  DUHS IRB#: Pro00018699
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
              IRB Protocol – PRO CTCAE Usability Testing

                                                  DUHS IRB#: Pro00018699
Duke Cancer Care Reseaerch Program @ Duke Comprehensive Cancer Center
              IRB Protocol – PRO CTCAE Usability Testing

                                                  DUHS IRB#: Pro00018699

				
DOCUMENT INFO
Categories:
Tags:
Stats:
views:2
posted:5/22/2012
language:
pages:24