Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

row AND computer AND tomography - US Food and Drug Administration

VIEWS: 6 PAGES: 22

									       DEPARTMENT OF HEALTH AND HUMAN SERVICES                                                                                    Food and Drug Administration



                                                                                                                                           Memorandum
Date: October 20, 2009

From: FDA Center for Devices and Radiological Health

To: Panel Members and Consultants

Subject: Computer-Assisted Detection (CADe) devices and FDA’s proposed regulatory
          approach



                                                             Table of Contents

I. Purpose of Meeting ..................................................................................................................................................3

II. FDA’s Regulatory Authority .................................................................................................................................3

a. Premarket Notification [510(k)] .............................................................................................................................3
   b. Premarket Approval (PMA) ...................................................................................................................................4
   c. Valid Scientific Evidence .......................................................................................................................................4
   d. Least Burdensome Provision for Medical Devices ................................................................................................5
   e. Reclassification ......................................................................................................................................................6

III. Background on CADe ...........................................................................................................................................6
   a. 2008 Panel Meeting on CAD .................................................................................................................................6
   b. Proposed CADe guidance documents ....................................................................................................................6
      i. Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data-Premarket
      Notification [510(k)] Submissions.........................................................................................................................7
      ii. Clinical Performance Assessment for PMA and 510(k) Submissions: Considerations for Computer-Assisted
      Detection Devices Applied to Radiology Images and Radiological Device Data .................................................7

IV. Summary of the Epidemiologic Review of the Literature .................................................................................7
   a. Mammography .......................................................................................................................................................7
      i. Retrospective Reader Performance Studies ........................................................................................................7
      ii. Clinical Performance Studies of CADe Mammography ...................................................................................8
      iii. Meta-Analyses and Narrative Reviews ............................................................................................................8
      iv. Summary of Literature Evaluating CADe Mammography ...............................................................................9
   b. Colon CADe and Lung CADe .............................................................................................................................. 10

V. Selection of Relevant Statistical Literature as it Applies to CADe ................................................................... 10

VI. Considerations in the Premarket Evaluation of CADe Devices ...................................................................... 12
   a. Description of the Device .................................................................................................................................... 12
   b. Standalone Performance Assessment .................................................................................................................. 13
   c. Clinical Performance Assessment ........................................................................................................................ 13


                                                                                     1
   d. Areas of Continuing Concern .............................................................................................................................. 14

VII. References........................................................................................................................................................... 15

IX. Attachments (on enclosed CD) ........................................................................................................................... 22
   1.      Draft Panel Questions ..................................................................................................................................... 22
   2. Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data-Premarket
   Notification [510(k)] Submissions ........................................................................................................................... 22
   3. Clinical Performance Assessment for PMA and 510(k) Submissions: Considerations for Computer-Assisted
   Detection Devices Applied to Radiology Images and Radiological Device Data ................................................... 22
   4.      Epidemiologic Review of Mammography CADe ............................................................................................. 22
   5.      Epidemiologic Review of Colon and Lung CADe ............................................................................................ 22
   6.      Summary Briefing of the March 2008 Panel ................................................................................................... 22
   7.      Summary Minutes of the March 2008 Panel.................................................................................................... 22
   8.      Transcripts of the March 2008 Panel .............................................................................................................. 22
   9.      Reference Articles since the March 2008 Panel .............................................................................................. 22




                                                                                     2
I. Purpose of Meeting

During the last Radiologic Devices Advisory Panel held in March 2008, several important issues
were discussed and recommended for various specific computer-assisted detection (CADe)
applications, e.g., mammography, lung, colon, etc. These devices are regulated as either class II
or class III devices depending upon the indications for use and the technological characteristics
of the device. The CADe devices that are Class II require 510(k)-clearance to be legally
marketed; the CADe devices that are Class III require an approved premarket approval (PMA)
application to be legally marketed. The Agency has published two draft guidance documents:
one that describes the clinical performance data we would expect for most new CADe devices,
whether subject to premarket notification (510(k)) submissions or premarket approval (PMA)
applications, and one that describes the data and information we would expect in 510(k)
submissions for the subset of CADe devices requiring 510(k) clearance.

On November 18, 2009, the Agency is holding a meeting of the Radiologic Devices Panel (“the
panel”). The Agency is requesting that the panel review these two draft guidance documents
during the open comment period and provide input, during the meeting, on the adequacy [and
appropriateness?] of our proposed regulatory approach for CADe devices. .

This Radiologic Devices Panel is convened primarily to obtain feedback from the panel on the
scientific merits of the Agency’s proposed approach described in these two documents.
Additionally, the Agency would like to discuss and seek recommendations on specific items
related to the clinical study requirements, and how these requirements, if applicable, differ
depending on the type of CADe devices and the type of premarket submissions.


II. FDA’s Regulatory Authority

a. Premarket Notification [510(k)]

A 510(k) is a premarket submission made to FDA to demonstrate that the device to be marketed
is substantially equivalent (SE) to a legally marketed device, and is not a class III device subject
to premarket approval (PMA). A submitter must compare their device to one or more similar
legally marketed devices and demonstrate their substantial equivalency claims. A legally
marketed device, as described in 21 CFR 807.92(a)(3), is a device that was legally marketed
prior to May 28, 1976, for which a PMA is not required, or a device which has been reclassified
from Class III to Class II or I, or a device which has been found SE through the 510(k) process.
The legally marketed device(s) to which equivalence is drawn is commonly known as the
"predicate."

A device is substantially equivalent as specified in section 513(i) of the Federal Food, Drug, and
Cosmetic Act (Act) if, in comparison to a predicate it:

      has the same intended use as the predicate; and
      has the same technological characteristics as the predicate;
       or
      has the same intended use as the predicate; and




                                                  3
      has different technological characteristics and the information submitted to FDA,
       including appropriate clinical or scientific data where necessary, demonstrates that the
       device:
           o does not raise different questions of safety and effectiveness than the predicate;
               and
           o demonstrates that the device is at least as safe and effective as the predicate.

Substantial equivalence is established with respect to intended use, design, energy used or
delivered, materials, chemical composition, manufacturing process, performance, safety,
effectiveness, labeling, biocompatibility, standards, and other characteristics, as applicable.
Substantial equivalence does not require a showing that the new and predicate devices are
identical in every respect.

A device requiring 510(k) clearance may not be marketed in the U.S. until the submitter receives
a letter declaring the device to be substantially equivalent to a legally marketed predicate device.
If FDA determines that a device is not substantially equivalent (NSE), the device in effect is a
Class III device (section 513(f)(1) of the Act). The applicant therefore has the following options:

   o   resubmit another 510(k) with new information demonstrating substantial equivalence,
   o   request classification as Class I or II through what is called the “de novo classification
       process” (section 513(f)(2) of the Act)
   o   file a petition requesting reclassification to Class I or II (section 513(f)(3) of the Act), or
   o   submit a PMA.

b. Premarket Approval (PMA)

Premarket approval (PMA) is the scientific and regulatory review process whereby FDA
evaluates the safety and effectiveness of Class III medical devices. Class III devices are those
that support or sustain human life, are of substantial importance in preventing impairment of
human health, or which present a potential, unreasonable risk of illness or injury. Due to the
level of risk associated with Class III devices, FDA has determined that general and special
controls alone are insufficient to provide a reasonable assurance of device safety and
effectiveness. These devices require an approved premarket approval (PMA) application under
section 515 of the Federal Food, Drug, and Cosmetic Act (Act) in order to be legally marketed.

The PMA process is the most stringent type of device marketing application required by FDA.
PMA approval is based on a determination by FDA that the PMA contains sufficient valid
scientific evidence demonstrating that there is a reasonable assurance that the device described in
the PMA is safe and effective for its intended use(s).

c. Valid Scientific Evidence

With respect to a PMA, FDA relies upon valid scientific evidence to determine whether there is
reasonable assurance that the device is safe and effective. After considering the nature of the
device and the information in the application, FDA will determine whether the scientific
evidence submitted or otherwise available to the FDA is valid for the purpose of determining the
safety or effectiveness of a particular device and whether the available evidence, when taken as a
whole, is adequate to support a determination that there is reasonable assurance that the device is
safe and effective for its conditions of use. FDA makes this determination in light of the


                                                  4
regulations in 21 CFR § 860.7 that govern Agency determinations of safety and effectiveness
and define “valid scientific evidence.”

As defined in 21 CFR § 860.7(c)(2), ”valid scientific evidence” is evidence from well-controlled
investigations, partially controlled studies, studies and objective trials without matched controls,
well-documented case histories conducted by qualified experts, and reports of significant human
experience with a marketed device, from which it can fairly and responsibly be concluded by
qualified experts that there is reasonable assurance of the safety and effectiveness of a device
under its conditions of use. The evidence required may vary according to the characteristics of
the device, its conditions of use, the existence and adequacy of warnings and other restrictions,
and the extent of experience with its use. Isolated case reports, random experience, reports
lacking sufficient details to permit scientific evaluation, and unsubstantiated opinions are not
regarded as valid scientific evidence to show safety or effectiveness.

d. Least Burdensome Provision for Medical Devices

When considering what level of evidence is needed to demonstrate a reasonable assurance of
safety and effectiveness (PMA) or substantial equivalence (510[k]) the Agency applies the least
burdensome approach as provided for under the statute or Food, Drug, and Cosmetic Act). The
Act as Amended by FDAMA of 1997 added sections 513(a)(3)(D)(ii) and 513(i)(1)(D), two
provisions commonly referred to as the "Least Burdensome Provisions."

Section 513(a)(3)(D)(ii) provides:

   "Any clinical data, including one or more well-controlled investigations, specified in writing
   by the Secretary for demonstrating a reasonable assurance of device effectiveness shall be
   specified as a result of a determination by the Secretary that such data are necessary to
   establish device effectiveness. The Secretary shall consider, in consultation with the
   applicant, the least burdensome appropriate means of evaluating device effectiveness that
   would have a reasonable likelihood of resulting in approval."

Section 513(i)(1)(D) provides:

   "Whenever the Secretary requests information to demonstrate that devices with differing
   technological characteristics are substantially equivalent, the Secretary shall only request
   information that is necessary to making substantial equivalence determinations. In making
   such requests, the Secretary shall consider the least burdensome means of demonstrating
   substantial equivalence and request information accordingly."

The term “least burdensome” has been interpreted by the Agency as a means of addressing a
premarket issue through what amounts to the most appropriate investment of time, effort, and
resources on the part of industry and FDA. CDRH applies this concept to devices and device
components of combination products regulated by FDA under the device provisions (including in
vitro diagnostics (IVDs)). When the least burdensome concept is conscientiously applied, we
believe it helps to expedite the availability of new device technologies without compromising
scientific integrity in the decision-making process or FDA’s ability to protect the public health.

Further explanation on how these provisions apply to premarket submissions such as 510(k) and
PMA can be found at


                                                 5
http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm0
85994.htm


e. Reclassification

As experience and knowledge about a device increases, the original classification can be adjusted
via the process of reclassification. This is generally done for devices that are in Class III (PMA)
to reduce their class to Class II or I (510(k)). Changes in classification are based on new
information about a device. FDA may, on its own, or in response to an outside petition, change a
device's classification by regulation. A manufacturer who wishes to have a device reclassified
to a lower class must convince FDA that the less stringent class requirements will be sufficient to
provide reasonable assurance of safety and effectiveness. The agency often ensures that the
reclassified device type will remain safe and effective by publishing a “special controls
guidance” that details the information that should be provided in a 510(k) submission, or by
otherwise identifying applicable special controls.

FDA notifies petitioners of determinations made on petitions for reclassification by a
reclassification letter. If a determination is made to reclassify a device, FDA publishes a
proposed rule to reclassify in the Federal Register which includes the scientific justification for
reclassification and which affords a period for comment. Subsequently a final rule is published in
the Federal Register which changes the reclassification.



III. Background on CADe


a. 2008 Panel Meeting on CAD

On March 4 and 5, 2008, a panel meeting was held to discuss and make recommendations about
computer aided detection and diagnosis (CAD) devices for radiological images (e.g.,
mammograms, chest x-rays, and computed tomography (CT) images of the lungs or colon).
Discussion focused on the general methodologies for CAD, including how CAD devices are used
in clinical decision-making, how the devices are tested, and the information needed to properly
assess their safety and effectiveness. Specific discussions focused on mammography CAD
devices, colon CAD devices, and lung CAD devices. The discussions also included how the
different types of CAD devices are used and the literature published regarding these devices,
with focus on testing issues related to the different devices. More detailed information, including
the panel briefing materials, panel agenda, summary minutes, and the full transcripts are
available at http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfAdvisory/details.cfm?mtg=694.

b. Proposed CADe guidance documents

Based on the discussions and recommendations presented at the March 2008 panel meeting, the
Agency began deliberations on the development of draft guidance documents to articulate a
proposed regulatory strategy for CADe devices. Due to the complexity of the review questions,
the multitude of submissions incorporating CADe, and the urgency for guidance on the
premarket requirements for these devices, the Agency has decided to begin this process with a


                                                 6
general framework for all CADe devices. The two draft guidance documents are currently out
for public comment and can be found at the links below.

i. Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device
Data-Premarket Notification [510(k)] Submissions

ii. Clinical Performance Assessment for PMA and 510(k) Submissions: Considerations for
Computer-Assisted Detection Devices Applied to Radiology Images and Radiological Device
Data

IV. Summary of the Epidemiologic Review of the Literature

The March 2008 Radiological Devices Advisory Panel briefing document contains a review of
the peer-reviewed literature related to computer-assisted detection and computer-assisted
diagnosis published prior to March 2008. A number of additional computer-assisted detection
(CADe) related studies have been published since the March 2008 panel meeting. As part of the
preparation for this panel meeting, the Agency conducted an epidemiological review of the
literature for each device type. Summaries of these findings are presented here and the full
reviews are included in the panel pack.

a. Mammography

A systematic search of the MEDLINE database yielded 18 unique clinical evaluations and 12
reviews (including those with meta-analyses) published after 1996 that were conducted in the
United States (Attachment X). Among these, three unique clinical evaluations (24, 28, 30) and
two meta-analyses (57, 79) were published after the March 2008 panel meeting. Studies from
1996-2008 were included in the review if they met specific criteria as outlined in Attachment X.
Studies were categorized as Retrospective Reader Performance Studies, Clinical Studies of
CADe Mammography and Meta-Analyses and Narrative Reviews.

i. Retrospective Reader Performance Studies

There are a number of limitations for retrospective reader performance studies of CADe
mammography performance. First, in order to permit the time commitment of participating
radiologists to be reasonable, the case mix does not reflect real world practice. In the actual
practice of mammography screening only a tiny minority of examinations, less than one percent,
will be true positives. The retrospective reader performance studies have true positive case mixes
that exceed this amount. The greater number of true positives can bias the results in a number of
different ways. Radiologists will be aware of the greater baseline probability of true positive
examinations and may adjust their interpretations accordingly. In addition, there will be fewer
false positive marks per true positive case for the radiologist to dismiss as the proportion of true
negative cases will be smaller than the proportion encountered in actual practice. Second,
valuable patient data, such as a family history of breast cancer, are often not provided in
retrospective reader performance studies. Third, access to and comparison with previous
mammographic examinations, an integral component of the interpretative process in
mammography, are often not provided. Comparison with previous examinations can be
instrumental in determining that a given abnormality is benign. Fourth, the retrospective reader
performance setting does not fully reflect clinical reality. The human cost of failing to detect a




                                                 7
cancer in practice is not found in this setting. Fifth, the potential for malpractice liability, which
may affect a radiologist’s decision-making in actual practice, are absent in this setting.

ii. Clinical Performance Studies of CADe Mammography

The findings among individual studies of clinical performance were mixed. Most found that the
introduction of CADe yielded similar or improved cancer detection rates, but some noted an
increased recall rate. To date, there are no randomized clinical trials that compare single-reader
or double-reader to single reader plus CADe for the evaluation of mammography scans. A future
three-arm randomized controlled trial that compares the effectiveness of single reader, double-
reader and single-reader plus CADe for screening mammography may resolve the problems in a
consistent set of findings in clinical studies. The inclusion of double-reader is important since
studies have found improvements in sensitivity with a decrease in recall rates; however
specificity also decreased (97).

iii. Meta-Analyses and Narrative Reviews

Two meta-analyses (57, 79) and one narrative review (97) were found among the publications
that resulted from the search criteria. Each meta-analysis included a different set of published
studies because the questions that the authors aimed to answer were different for each of them.

Although the two meta-analyses were designed to address different questions, both came-up with
fundamentally similar conclusions.

Taylor and Potts (79) concluded that: (1) their pooled estimates suggest that CADe may change
the threshold for recalls rather than improve the accuracy of screening; (2) CADe impact is
diminished by the high number of false positive prompts; (3) CADe increases recall rate; (4) Not
enough evidence to conclude CADe improves cancer detection rate; (5) there is evidence that
double reading increases cancer detection rates; and (5) double reading with arbitration can
lower the recall rates. They recommended: (1) CADe developers should improve specificity; and
(2) further research to evaluate heterogeneity in reported recall rates.

This meta-analysis was well designed with clearly stated inclusion criteria for the studies that
were selected and the endpoints of interest (cancer detection rates and recall rates). The statistical
analyses performed are valid and commonly used in meta-analysis. They considered the studies
that have previously created some controversy (Fenton et al (98) and Gur et al (99)), which were
driving heterogeneity in recall rates. Their sensitivity analyses show their overall estimates that
remain the same when including or excluding those two studies.

Noble, et. al. (15) used different inclusion criteria and different study endpoints (sensitivity,
specificity, recall rate, biopsy rate and cancer diagnosis rate). These endpoints determined the
inclusion or exclusion of certain studies. However, the authors did a good job explaining the
reasons for excluding other studies from the analysis. Despite the differences in methodology
(compared to Taylor and Potts), similar findings were reported. These authors conclude: (1)
CADe increases recall rate of healthy women; (2) increases biopsy rate of healthy women; (3)
limited impact: use of CADe will identify 50 additional cases in 100, 000 screened women.
Although they believe their results are robust, they recommend frequent monitoring of the
literature for this type of technology.



                                                   8
This meta-analysis was also well designed with clearly stated objectives, inclusion/exclusion
criteria and the endpoints of interest. Although, charts common to meta-analysis are not
provided, the statistical analysis was valid and commonly used for this type of analysis.

The narrative review by Helvie et al (97) was well written and seems to cover a wide range of
publications. However, the review does not include the methodology used for the literature
search, the inclusion and exclusion criteria used to select the studies and no formal meta-analysis
was performed. Due to differences in the methodologies of the studies that were included, the
estimates are not comparable. The review does, however, provide valuable input on the factors
related with performance of radiologist (independent from CADe and double reading) and
provides a narrative of the results of studies that evaluated performance of CADe and double
reading.

iv. Summary of Literature Evaluating CADe Mammography

This review included a number of research studies, meta-analyses and review papers conducted
to evaluate the safety and effectiveness of incorporating CADe readers in the interpretation of
mammography exams. All currently approved CADe devices are labeled for use as “second
readers” in both routine screening and diagnostic mammograms. For use in diagnostic
mammography, CADe devices are only approved for symptomatic patients with standard
mammographic views. These studies do not present a clear and reproducible set of results that
clearly support the safety and effectiveness in real-world situations, leading to conflicting
recommendations regarding the incorporation of this technology into clinical practice. Although
there is evidence that double-reading increases cancer detection rates and that double-reading
with arbitration lowers the recall rate (79), the findings using CADe as the second reader are
mixed. Taylor and Potts stress that changes noted in CADe may be explained by change in the
threshold for the threshold for the recall rate (since the recall rate is increased) rather than
improvement in the accuracy of screening. This concern raises an area of question that must be
evaluated in future studies.

The benefits of mammography CADe must be weighed against the potential harms. The
assessment of mammography CADe is complicated by the absence of a clear and acceptable
trade-off between increased recall and negative biopsy rates and increased sensitivity in the
detection of breast cancer. Increased recall and negative biopsy rates are intrinsically undesirable
on the grounds of cost, morbidity, and psychological stress. This is a value judgment for
individual clinicians and turns into a largely philosophical rather than quantitative decision. The
potential for malpractice liability may also influence decision making using mammography
CADe performance during actual conditions of use; the effect of this influence should not be
underestimated. Such influences may cause the performance of CADe in actual practice to be
less accurate than the performance observed in laboratory settings.

Research indicates that the addition of a second reader with arbitration can improve sensitivity
and decreases the recall rate; however, when CADe is used in place of the second reader,
findings on its safety and effectiveness are mixed. There is no objective number of recalls and
negative biopsies that offset the detection of one breast cancer. Currently there are no
randomized clinical trials that have compared the different reader approaches, nor is there a
consistency in data collected in observational studies to allow extensive comparison.
Additionally, studies using test sets often lack sufficient conditions to simulate real-life settings,



                                                   9
limiting the ability to generalize findings. Including long-term follow-up may be a preferred
option due to cost considerations.

Awareness of the potential unintended adverse consequences from the use of mammography
CADes in actual practice is an important consideration for FDA during the review process and
for clinicians once the device has been approved for marketing. Helvie et al (97) advances a
cogent argument that the use of the CADe may create the potential for a decrease in the effort of
the radiologist in examining the mammogram, diminishing the benefits of CADe utilization.

b. Colon CADe and Lung CADe

This review included a number of research studies conducted to evaluate the safety and
effectiveness of incorporating CADe readers in the interpretation of lung and colon exams. All
the studies were standalone reader studies and do not present a clear and reproducible set of
results that clearly support the safety and effectiveness in real-world situations, leading to
conflicting recommendations regarding the incorporation of this technology into clinical
practice. Although all of the studies identified an increase in sensitivity using CADe systems,
often the false-positive rate increased and it often only influenced decision-making in novice
readers. As with CADe mammography, the benefits of incorporating CADe readers in the
interpretation of lung and colon exam must be evaluated with consideration of the potential
harms including unnecessary biopsies by both FDA during reviews and clinicians when deciding
to use the device. Further research using a variety of study modalities is important when
considering the issues.

V. Selection of Relevant Statistical Literature as it Applies to CADe

A number of statistical and disease specific considerations arise when evaluating CADe devices.
The primary goal is to evaluate the potential impact of the CADe device on the intended user and
in the intended use population. Multi-reader multi-case (MRMC) studies are a standard approach
for evaluating clinical CADe performance. For the 2008 panel meeting on CAD devices, panel
members were provided a panel package that contained many references on CAD studies, their
study design, and their statistical analysis. In the addition to those references, the following
references cite the Agency’s current statistical guidance on diagnostic studies, and reference
recent methodological research that may improve MRMC studies and ROC analysis.

On March 13, 2007, the FDA issued, “Statistical Guidance on Reporting Results from Studies
Evaluating Diagnostic Tests.” Intended for both statisticians and non-statisticians, the guidance
addresses reporting of results from studies evaluating diagnostic devices with two possible
outcomes (positive or negative) in PMAs and 510(k)s. This guidance does not address statistical
issues associated with the design of clinical studies for diagnostic devices.

The March 2008 Radiological Devices Advisory Panel briefing document contains a review of
the peer-reviewed literature related to statistical consideration evaluating CADe device
submissions. We have included a brief review of a few additional statistical references that have
appeared since March 2008 or were not included in the March 2008 panel package. This review
is not a compressive review of the statistical literature but is included to provide some
information to the panel on additional relevant statistical publications.




                                                10
Obuchowski (101) provides a tutorial of ROC analysis for radiologists, including basic statistical
methods for fitting ROC curves, for comparing them, and for determining sample size. She
briefly describes multiple-reader multiple case studies of ROC curves. In the article, she
describes a hypothetical example of mammographic BI-RADs score data from which ROC
curves are constructed. She also describe an ROC study of a mammographic CAD by Zheng et
al. (Zheng B, Leader JK, Abrams G, et al., Computer-aided detection schemes: the effect of
limiting the number of cued regions in each case, AJR, 2004; 182: 579-583), which is analyzed
using free-response ROC methodology. She lists available software for analysis of ROC studies
and provides a large number of references on advanced statistical methods. In reference to the
paper, a letter to the editor by Skaane and the reply by Obuchowski (Skaane, P. Receiver
Operating Characteristic Analysis: A Proper Measurement for Performance in Breast Cancer
Screening?” (102)) provide clarification on the relevance of ROC curves for the evaluation of
mature screening and diagnostic tests.

Toledano (95) compares three methods for analyzing correlated ROC curves: a generalized
estimating equations approach for marginal non-proportional ordinal regression, the popular
Dorfman-Bernbaum-Metz method of using jackknifed pseudovalues of area under the ROC
curve (AUC), and a method using a correct F-test from analysis of variance of AUC. The
methods were compared on real data sets from multi-reader, multi-case studies, that is, multiple
readers interpreting images obtained with each test modality on each study subject. While point
estimates of differences between test modalities are similar, the standard errors of these
differences do not agree for all three methods. A simulation study supports the standard errors
provided by the generalized estimating equations approach.

Obuchowski et al. (96) perform an empirical comparison of five methods for multireader,
multicase ROC analysis using data from three previously published studies. Based on their
results, the authors conclude “in choosing and applying MRMC methods, it is important to
recognize: (1) the distinction between random-reader and fixed-reader models, the uncertainties
accounted for by each, and thus the level of generalizeability expected from each; (2)
assumptions made by the various MRMC methods; and (3) limitations of a five- or six-reader
study when the reader variability is great”.

Wang et al. (91) explore the design and analysis of multi-reader, multi-modality studies of
medical tests using hierarchical models for ROC curve summary measures (e.g., AUC). The
models are intended to allow the variance of the estimates to depend on the actual value of the
index (e.g., reader, modality) and account for the correlation in the data both explicitly via
parameters and implicitly via the hierarchical structure. The methodology is illustrated on a study
conducted to evaluate a computer-aided diagnosis tool for screen film mammography. Design of
multi-reader, multi-modality studies is considered using the simulation-based, Bayesian design
approach of Wang and Gelfand (Stat. Sci. 2002; 17(2):193–208) and illustrated on a multi-reader
study comparing display modes for digital mammography.

Gallas et al. (20) used computer simulation models to assess the statistical power of a variety of
study designs. Human reader variability was considered in the context of medical imaging
computer-assisted diagnosis (CADe) systems. Methods to estimate multi-reader multi-case
(MRMC) variance analysis were extended for cases of arbitrary study design. The methods are
applicable to the analysis of the detection performance using receiver operating characteristic
(ROC) experiments, summarized by the reader-averaged area under the ROC curve (AUC) of
readers with and without the CADe. Gallas et al. (100) has previously reported on unbiased


                                                11
method for estimating MRMC variance in binary experiments where the summary performance
measure is a percent correct (e.g., sensitivity).

VI. Considerations in the Premarket Evaluation of CADe Devices

The two draft guidance documents apply to computer-assisted detection (CADe) devices. The
Agency has defined a CADe device in the Scope section of each guidance as a computerized
system that incorporates pattern recognition and data analysis capabilities (i.e., combines values,
measurements, or features extracted from the patient radiological data) and is intended to
identify, mark, highlight, or in any other manner direct attention to portions of an image, or
aspects of radiology device data, that may reveal abnormalities during interpretation of patient
radiology images or patient radiology device data by the intended user (i.e., a physician or other
health care professional).

The draft guidance documents discuss different information and data the FDA has identified as
being important in understanding the technical specification of a CADe device and in assessing
device performance. Three major types of information including a description of the device,
standalone performance assessment and clinical performance assessment are discussed in the
draft guidance documents.

a. Description of the Device

CADe devices are computer software that perform processing and analysis of radiological
images or radiological device data. Different CADe devices often contain different software and
utilize different processing, algorithms and features to identify abnormalities. The process of
training algorithms or selecting features can vary across CADe devices and this influences the
performance of these devices. Most processing methods, algorithms, and training and selection
techniques are known and well described in published literature. There are numerous ways in
which these methods can be combined and optimized to make a CADe device. At the March
2008 Radiologic Panel meeting, panel members commented that algorithm details should be
available to the Agency to evaluate new or subsequent algorithm updates to the extent the
Agency needs this information to evaluate algorithm stability and future changes. The Agency
agreed and, as part of the algorithm details, recommends that the following information be
submitted:

      the technical characteristics of the device;
      the training process used in algorithm development;
      the databases used to train and test the device;
      the references standard used to determine whether or not a disease/condition/abnormality
       is present and potentially its location and extent; and
      the scoring methodology used to determining the correspondence between the CADe
       output and the reference standard (e.g., disease location);

Section 4 of the 510(k) guidance describes the different types of information the Agency has
identified as relevant in 510(k) CADe submissions.




                                                12
b. Standalone Performance Assessment

Most CADe devices will represent a new implementation of software so that each new CADe
device (as well as software and other design, technology, or performance changes to an already
cleared CADe device) will have different technological characteristics from previously marketed
CADe devices even when sharing the same intended use.

Standalone performance indicates the performance of the device in the absence of an intended
user (i.e., how well does the CADe device mark regions of known abnormalities (true positives)
and how often does it mark regions that do not contain abnormalities (false positives)); it
provides a measure of the intrinsic functionality of the CADe device. Section 5 of the 510(k)
draft guidance describes recommendations on conducting and reporting standalone performance.
This section includes a discussion of study population, detection accuracy, location accuracy,
reproducibility testing, algorithm stability testing, and algorithm training performance.

Standalone performance can be important to users of CADe devices in that it may provide them
with some level of confidence in the device. Standalone performance can also provide
information about the effectiveness of the device to mark important subtypes of disease, patient
and imaging characteristics, or help determine which patients might benefit most from the
CADe. Therefore, standalone performance stratified by important covariates can be useful in
labeling a CADe device providing the end users with additional information to better interpret
the meanings of the CADe marks.

Furthermore, at the March 2008 Panel meeting, panel members “could not think of an instance in
which standalone performance testing would not be important.” The algorithm change was also
discussed. Panel members agreed that “standalone testing would probably be sufficient for
minor modifications.” However, the panel did not develop a consensus on a definition for
“minor modifications.” Also, panel members stated that “changes in the algorithm should
require testing”; however, they did not decide exactly where the line between standalone only
and more extensive testing should be drawn. The Agency believes this issue warrants further
discussion at the upcoming November panel meeting.

c. Clinical Performance Assessment

As noted above, the standalone performance does not measure the impact of the CADe device on
the performance of the intended user. A clinical performance assessment (i.e., a reader study)
estimates the impact of the CADe device on user performance. Because the detection
process/decision cannot be done by the CADe device alone (i.e., the reader is an integral part of
the diagnostic process), the Agency is recommending that performance data of the CADe device
when used by the intended user be included to demonstrate the clinical performance of a CADe
device.

Section 7 of the 510(k) draft guidance also discusses when a clinical performance assessment is
recommended in a 510(k) submission and the merits of various control arms (e.g., unaided read,
read with the predicate device) that might be considered when comparing a CADe device to a
predicate device.




                                               13
The Clinical Performance draft guidance provides general guidance on conducting a clinical
performance assessment, making it applicable to both PMA and 510(k) submissions. The
Clinical Performance draft guidance discusses the following:

      the clinical study design;
      the study population used in the assessment;
      the references standard used to determine whether or not a disease/condition/abnormality
       is present and potentially its location and extent;
      reporting of performance results; and
      postmarket planning for Premarket Approval (PMA) submissions.

While the draft guidance documents provide general recommendations on CADe device
submissions, they do not directly address application-specific CADe submissions (e.g., colonic
polyp CADe and mammography CADe) nor do they completely clarify the specific performance
information and study design recommendations for individual types of submissions. The
Agency anticipates including an additional section containing frequently asked questions (FAQs)
in each guidance, when finalized, and seeks panel discussion and recommendations to help
identify and clarify relevant and important questions and answers. These FAQ sections will be
used to further clarify the Agency’s recommendations and to assist industry in the interpretation
of the guidance documents as they apply to specific devices.

d. Areas of Continuing Concern

This panel is being convened to help the Agency develop the specific questions and answers to
be included in a “FAQs” section in each guidance, when finalized. Likewise, the panel will be
asked to provide input on data requirements and appropriate study designs for Class II and Class
III CADe device submissions, potentially leading to further clarification in the two guidance
documents.

One area for which the Agency is requesting feedback concerns the selection of a control arm in
the clinical performance assessment for CADe 510(k) device submissions. As mentioned in
Section II above, in a CADe 510(k) device submission the sponsor needs to compare their device
to a legally marketed predicate device. One pathway is to directly compare the clinical
performance of the new device to the clinical performance of the predicate using the same data,
readers, study design scoring, etc. A second pathway might be for the sponsor to indirectly
compare the new CADe device to the predicate device based on a clinical assessment using
unaided reading as the control. In this case, both devices would need to have used unaided
reading as a control but not necessarily the same cases and readers. The panel will be asked to
comment on the appropriateness of this pathway in CADe 510(k) device submissions as
evidence for comparing the new device with a predicate device. For CADe PMA device
applications, the clinical performance assessment submitted to demonstrate a reasonable
assurance of device safety and effectiveness will generally be an aided read (using the device not
yet approved) compared with unaided reading.

A second area for which the Agency is seeking feedback concerns the types of changes to a
CADe device that necessitate a new clinical performance assessment versus the types of changes
to a CADe device that could be adequately addressed through the inclusion of device information
and standalone performance assessment only. These latter changes are the so-called “minor



                                                14
modifications” or changes to a CADe device which were discussed during the March 2008 panel
meeting but were not clearly defined during the course of the panel discussions.

A third area for which the Agency is requesting panel input relates to statistical considerations in
CADe clinical assessment study designs and their endpoints. In particular, the Agency is
requesting additional clarification on the reuse of test data in CADe device submissions as well
as clarification on which data subgroups and endpoints should be powered for statistical
inference. For the case of data reuse, the March 2008 panel commented that they had severe
concerns about the reuse of test data, and that optimally a new test set should be obtained.
However, they also commented that they did realize that there will be certain circumstances
where acquiring completely new data will either be unnecessary or be so burdensome that a
lesser solution would be acceptable.

The questions in Attachment 1 have been developed and grouped to outline the Agency’s
remaining issues to be addressed in order to refine the guidance documents and to develop
specific FAQs for the guidance documents, when finalized. On these questions, we seek specific
feedback from the panel.

VII. References
       1.    Acharya UR, Ng UE, Chang YH, et al. Computer-based identification of breast
             cancer using digitized mammograms. Journal of Medical Systems 2008; 32:499-
             507.
       2.    Baltzer PA, Renz DM, Kullnig PE, et al. Application of computer-aided diagnosis
             (CAD) in MR-mammography (MRM): do we really need whole lesion time curve
             distribution analysis? Academic Radiology 2009; 16:435-442.
       3.    Bley TA, Baumann T, Saueressig U, et al. Comparison of radiologist and CAD
             performance in the detection of CT-confirmed subtle pulmonary nodules on
             digital chest radiographs. Investigative Radiology 2008; 43:343-348.
       4.    Boyer B, Balleyguier C, Granat O, et al. CAD in questions/answers Review of the
             literature. European Journal of Radiology 2009; 69:24-33.
       5.    Brancato B, Houssami N, Francesca D, et al. Does computer-aided detection
             (CAD) contribute to the performance of digital mammography in a self-referred
             population? Breast Cancer Research & Treatment 2008; 111:373-376.
       6.    Burling D, Moore A, Marshall M, et al. Virtual colonoscopy: effect of computer-
             assisted detection (CAD) on radiographer performance.[erratum appears in Clin
             Radiol. 2008 Jul;63(7):841 Note: Pickhardt, P [corrected to Pickhardt, P J];
             Taylor, S [corrected to Taylor, S A]]. Clinical Radiology 2008; 63:549-556.
       7.    Chakraborty DP, Yoon HJ, Yoon H-J. Operating characteristics predicted by
             models for diagnostic tasks involving lesion localization.[comment]. Medical
             Physics 2008; 35:435-445.
       8.    Chan HP, Wei J, Zhang Y, et al. Computer-aided detection of masses in digital
             tomosynthesis mammography: comparison of three approaches. Medical Physics
             2008; 35:4087-4095.
       9.    Changizi V, Giti M, Kheradmand AA, Kheradmand AA. Application of computed
             aided detection in breast masses diagnosis. Indian Journal of Cancer 2008;
             45:164-166.
       10.   Chen JJ, White CS, Chen JJ, White CS. Use of CAD to evaluate lung cancer on
             chest radiography. Journal of Thoracic Imaging 2008; 23:93-96.



                                                 15
11.   Choi EJ, Jin GY, Han YM, et al. Solitary pulmonary nodule on helical dynamic
      CT scans: analysis of the enhancement patterns using a computer-aided diagnosis
      (CAD) system. Korean Journal of Radiology 2008; 9:401-408.
12.   Chowdhury TA, Whelan PF, Ghita O, Chowdhury TA, Whelan PF, Ghita O. A
      fully automatic CAD-CTC system based on curvature analysis for standard and
      low-dose CT data. IEEE Transactions on Biomedical Engineering 2008; 55:888-
      901.
13.   Das M, Muhlenbruch G, Heinen S, et al. Performance evaluation of a computer-
      aided detection algorithm for solid pulmonary nodules in low-dose and standard-
      dose MDCT chest examinations and its influence on radiologists. British Journal
      of Radiology 2008; 81:841-847.
14.   Das M, Muhlenbruch G, Helm A, et al. Computer-aided detection of pulmonary
      embolism: influence on radiologists' detection performance with respect to vessel
      segments. European Radiology 2008; 18:1350-1355.
15.   Dundar MM, Fung G, Krishnapuram B, et al. Multiple-instance learning
      algorithms for computer-aided detection. IEEE Transactions on Biomedical
      Engineering 2008; 55:1015-1021.
16.   Duszak R, Jr., Duszak R, Jr. Coding for CAD. Journal of the American College of
      Radiology 2008; 5:619-620.
17.   Feldman MD, Feldman MD. Beyond morphology: whole slide imaging,
      computer-aided detection, and other techniques. Archives of Pathology &
      Laboratory Medicine 2008; 132:758-763.
18.   Filev P, Hadjiiski L, Chan HP, et al. Automated regional registration and
      characterization of corresponding microcalcification clusters on temporal pairs of
      mammograms for interval change analysis. Medical Physics 2008; 35:5340-5350.
19.   Fujita H, Uchiyama Y, Nakagawa T, et al. Computer-aided diagnosis: the
      emerging of three CAD systems induced by Japanese health care needs. Computer
      Methods & Programs in Biomedicine 2008; 92:238-248.
20.   Gallas BD, Brown DG, Gallas BD, Brown DG. Reader studies for validation of
      CAD systems.[erratum appears in Neural Netw. 2008 May;21(4):698]. Neural
      Networks 2008; 21:387-397.
21.   Garcia-Orellana CJ, Gallardo-Caballero R, Gonzalez-Velasco HM, et al. Study of
      a mammographic CAD performance dependence on the considered mammogram
      set. Conference Proceedings: Annual International Conference of the IEEE
      Engineering in Medicine & Biology Society 2008; 2008:4776-4779.
22.   Giger ML, Chan HP, Boone J, Giger ML, Chan H-P, Boone J. Anniversary paper:
      History and status of CAD and quantitative image analysis: the role of Medical
      Physics and AAPM. Medical Physics 2008; 35:5799-5820.
23.   Gilbert FJ, Astley SM, Boggis CR, et al. Variable size computer-aided detection
      prompts and mammography film reader decisions. Breast Cancer Research 2008;
      10:R72.
24.   Gilbert FJ, Astley SM, Gillan MG, et al. Single reading with computer-aided
      detection for screening mammography.[see comment]. New England Journal of
      Medicine 2008; 359:1675-1684.
25.   Girvin F, Ko JP, Girvin F, Ko JP. Pulmonary nodules: detection, assessment, and
      CAD. AJR American Journal of Roentgenology 2008; 191:1057-1069.
26.   Godoy MC, Cooperberg PL, Maizlin ZV, et al. Detection sensitivity of a
      commercial lung nodule CAD system in a series of pathologically proven lung
      cancers. Journal of Thoracic Imaging 2008; 23:1-6.


                                      16
27.   Goo JM, Kim HY, Lee JW, et al. Is the computer-aided detection scheme for lung
      nodule also useful in detecting lung cancer? Journal of Computer Assisted
      Tomography 2008; 32:570-575.
28.   Gromet M, Gromet M. Comparison of computer-aided detection to double
      reading of screening mammograms: review of 231,221 mammograms.[see
      comment]. AJR American Journal of Roentgenology 2008; 190:854-859.
29.   Guliato D, de Carvalho JD, Rangayyan RM, et al. Feature extraction from a
      signature based on the turning angle function for the classification of breast
      tumors. Journal of Digital Imaging 2008; 21:129-144.
30.   Gur D, Bandos AI, King JL, et al. Binary and multi-category ratings in a
      laboratory observer performance study: a comparison. Medical Physics 2008;
      35:4404-4409.
31.   Gur D, Bandos AI, Klym AH, et al. Agreement of the order of overall
      performance levels under different reading paradigms. Academic Radiology
      2008; 15:1567-1573.
32.   Hardie RC, Rogers SK, Wilson T, et al. Performance analysis of a new computer
      aided detection system for identifying lung nodules on chest radiographs. Medical
      Image Analysis 2008; 12:240-258.
33.   Helm EJ, Silva CT, Roberts HC, et al. Computer-aided detection for the
      identification of pulmonary nodules in pediatric oncology patients: initial
      experience. Pediatric Radiology 2009; 39:685-693.
34.   Hirose T, Nitta N, Shiraishi J, et al. Evaluation of computer-aided diagnosis
      (CAD) software for the detection of lung nodules on multidetector row computed
      tomography (MDCT): JAFROC study for the improvement in radiologists'
      diagnostic accuracy. Academic Radiology 2008; 15:1505-1512.
35.   Hock D, Ouhadi R, Materne R, et al. Virtual dissection CT colonography:
      evaluation of learning curves and reading times with and without computer-aided
      detection. Radiology 2008; 248:860-868.
36.   Horsch K, Giger ML, Metz CE, Horsch K, Giger ML, Metz CE. Potential effect
      of different radiologist reporting methods on studies showing benefit of CAD.[see
      comment]. Academic Radiology 2008; 15:139-152.
37.   Houssami N, Given-Wilson R, Ciatto S. Early detection of breast cancer:
      overview of the evidence on computer-aided detection in mammography
      screening. Journal of Medical Imaging & Radiation Oncology 2009; 53:171-176.
38.   James JJ, Cornford EJ. Does computer-aided detection have a role in the
      arbitration of discordant double-reading opinions in a breast-screening
      programme? Clinical Radiology 2009; 64:46-51.
39.   Jiang L, Song E, Xu X, et al. Automated detection of breast mass spiculation
      levels and evaluation of scheme performance. Academic Radiology 2008;
      15:1534-1544.
40.   Jorgensen KJ, Gotzsche PC, Jorgensen KJ, Gotzsche PC. Overdiagnosis in
      publicly organised mammography screening programmes: systematic review of
      incidence trends.[see comment]. British Medical Journal 2009; 339:b2587.
41.   Kallenberg M, Karssemeijer N, Kallenberg M, Karssemeijer N. Computer-aided
      detection of masses in full-field digital mammography using screen-film
      mammograms for training. Physics in Medicine & Biology 2008; 53:6879-6891.
42.   Kim SH, Lee JM, Shin CI, et al. Effects of spatial resolution and tube current on
      computer-aided detection of polyps on CT colonographic images: phantom study.
      Radiology 2008; 248:492-503.


                                      17
43.   Kim SJ, Moon WK, Cho N, et al. Computer-aided detection in full-field digital
      mammography: sensitivity and reproducibility in serial examinations. Radiology
      2008; 246:71-80.
44.   Krupinski EA, Krupinski EA. What can the radiologist teach CAD: lessons from
      CT colonoscopy.[comment]. Academic Radiology 2009; 16:1-3.
45.   Lauria A, Lauria A. GPCALMA: implementation in Italian hospitals of a
      computer aided detection system for breast lesions by mammography
      examination. Physica Medica 2009; 25:58-72.
46.   Li F, Engelmann R, Metz CE, et al. Lung cancers missed on chest radiographs:
      results obtained with a commercial computer-aided detection program. Radiology
      2008; 246:273-280.
47.   Li H, Giger ML, Yuan Y, et al. Evaluation of computer-aided diagnosis on a large
      clinical full-field digital mammographic dataset. Academic Radiology 2008;
      15:1437-1445.
48.   Linguraru MG, Zhao S, Van Uitert RL, et al. CAD of colon cancer on CT
      colonography cases without cathartic bowel preparation. Conference Proceedings:
      Annual International Conference of the IEEE Engineering in Medicine & Biology
      Society 2008; 2008:2996-2999.
49.   Malich A, Schmidt S, Fischer DR, et al. The performance of computer-aided
      detection when analyzing prior mammograms of newly detected breast cancers
      with special focus on the time interval from initial imaging to detection. European
      Journal of Radiology 2009; 69:574-578.
50.   Matsumoto S, Ohno Y, Yamagata H, et al. Computer-aided detection of lung
      nodules on multidetector row computed tomography using three-dimensional
      analysis of nodule candidates and their surroundings. Radiation Medicine 2008;
      26:562-569.
51.   Matsumoto S, Ohno Y, Yamagata H, et al. Computer-aided detection of lung
      nodules on multidetector row computed tomography using three-dimensional
      analysis of nodule candidates and their surroundings.[erratum appears in Radiat
      Med. 2009 Apr;27(3):161]. Radiation Medicine 2008; 26:562-569.
52.   Mazurowski MA, Habas PA, Zurada JM, et al. Decision optimization of case-
      based computer-aided decision systems using genetic algorithms with application
      to mammography. Physics in Medicine & Biology 2008; 53:895-908.
53.   McGarry T, McHugh B, Buis A, McKay G. Evaluation of the effect of shape on a
      contemporary CAD system. Prosthetics & Orthotics International 2008; 32:145-
      154.
54.   Morimoto T, Iinuma G, Shiraishi J, et al. Computer-aided detection in computed
      tomography colonography: current status and problems with detection of early
      colorectal cancer. Radiation Medicine 2008; 26:261-269.
55.   Mu T, Nandi AK, Rangayyan RM, Mu T, Nandi AK, Rangayyan RM.
      Classification of breast masses using selected shape, edge-sharpness, and texture
      features with linear and kernel-based classifiers. Journal of Digital Imaging 2008;
      21:153-169.
56.   Nishikawa RM, Pesce LL, Nishikawa RM, Pesce LL. Computer-aided detection
      evaluation methods are not created equal. Radiology 2009; 251:634-636.
57.   Noble M, Bruening W, Uhl S, et al. Computer-aided detection mammography for
      breast cancer screening: systematic review and meta-analysis. Archives of
      Gynecology & Obstetrics 2009; 279:881-890.



                                       18
58.   Ochs RA, Abtin F, Ghurabi R, et al. Computer-aided detection of endobronchial
      valves using volumetric CT. Academic Radiology 2009; 16:172-180.
59.   Papadopoulos A, Fotiadis DI, Costaridou L. Improvement of microcalcification
      cluster detection in mammography utilizing image enhancement techniques.
      Computers in Biology & Medicine 2008; 38:1045-1055.
60.   Park EA, Goo JM, Lee JW, et al. Efficacy of computer-aided detection system
      and thin-slab maximum intensity projection technique in the detection of
      pulmonary nodules in patients with resected metastases. Investigative Radiology
      2009; 44:105-113.
61.   Park SC, Pu J, Zheng B, Park SC, Pu J, Zheng B. Improving performance of
      computer-aided detection scheme by combining results from two machine
      learning classifiers. Academic Radiology 2009; 16:266-274.
62.   Park SH, Kim SY, Lee SS, et al. Sensitivity of CT colonography for nonpolypoid
      colorectal lesions interpreted by human readers and with computer-aided
      detection. AJR American Journal of Roentgenology 2009; 193:70-78.
63.   Petrick N, Haider M, Summers RM, et al. CT colonography with computer-aided
      detection as a second reader: observer performance study.[erratum appears in
      Radiology. 2008 Aug;248(2):704]. Radiology 2008; 246:148-156.
64.   Regge D, Hassan C, Pickhardt PJ, et al. Impact of computer-aided detection on
      the cost-effectiveness of CT colonography. Radiology 2009; 250:488-497.
65.   Reiser I, Nishikawa RM, Edwards AV, et al. Automated detection of
      microcalcification clusters for digital breast tomosynthesis using projection data
      only: a preliminary study. Medical Physics 2008; 35:1486-1493.
66.   Robinson C, Halligan S, Taylor SA, et al. CT colonography: a systematic review
      of standard of reporting for studies of computer-aided detection. Radiology 2008;
      246:426-433.
67.   Rojas Dominguez A, Nandi AK, Rojas Dominguez A, Nandi AK. Detection of
      masses in mammograms via statistically based enhancement, multilevel-
      thresholding segmentation, and region selection. Computerized Medical Imaging
      & Graphics 2008; 32:304-315.
68.   Sampat MP, Bovik AC, Whitman GJ, et al. A model-based framework for the
      detection of spiculated masses on mammography. Medical Physics 2008;
      35:2110-2123.
69.   Sampat MP, Whitman GJ, Bovik AC, et al. Comparison of algorithms to enhance
      spicules of spiculated masses on mammography. Journal of Digital Imaging 2008;
      21:9-17.
70.   Serrano D, Gandini S, Mariani L, et al. Computer-assisted image analysis of
      breast fine needle aspiration in a randomized chemoprevention trial of fenretinide
      vs. placebo in HRT users. Breast 2008; 17:91-97.
71.   Shi J, Sahiner B, Chan HP, et al. Characterization of mammographic masses
      based on level set segmentation with new image features and patient information.
      Medical Physics 2008; 35:280-290.
72.   Singh AK, Hiroyuki Y, Sahani DV, Singh AK, Hiroyuki Y, Sahani DV.
      Advanced postprocessing and the emerging role of computer-aided detection.
      Radiologic Clinics of North America 2009; 47:59-77.
73.   Singh S, Tourassi GD, Baker JA, et al. Automated breast mass detection in 3D
      reconstructed tomosynthesis volumes: a featureless approach. Medical Physics
      2008; 35:3626-3636.



                                      19
74.   Sriraam N, Roopa J, Saranya M, Dhanalakshmi M. Performance evaluation of
      computer aided diagnostic tool (CAD) for detection of ultrasonic based liver
      disease. Journal of Medical Systems 2009; 33:267-274.
75.   Summers RM, Frentz SM, Liu J, et al. Conspicuity of colorectal polyps at CT
      colonography: visual assessment, CAD performance, and the important role of
      polyp height.[see comment]. Academic Radiology 2009; 16:4-14.
76.   Summers RM, Handwerker LR, Pickhardt PJ, et al. Performance of a previously
      validated CT colonography computer-aided detection system in a new patient
      population. AJR American Journal of Roentgenology 2008; 191:168-174.
77.   Suzuki K, Yoshida H, Nappi J, et al. Mixture of expert 3D massive-training
      ANNs for reduction of multiple types of false positives in CAD for detection of
      polyps in CT colonography. Medical Physics 2008; 35:694-703.
78.   Tang J, Rangayyan RM, Xu J, et al. Computer-aided detection and diagnosis of
      breast cancer with mammography: recent advances. IEEE Transactions on
      Information Technology in Biomedicine 2009; 13:236-251.
79.   Taylor P, Potts HW, Taylor P, Potts HWW. Computer aids and human second
      reading as interventions in screening mammography: two systematic reviews to
      compare effects on cancer detection and recall rate. European Journal of Cancer
      2008; 44:798-807.
80.   Taylor SA, Brittenden J, Lenton J, et al. Influence of computer-aided detection
      false-positives on reader performance and diagnostic confidence for CT
      colonography. AJR American Journal of Roentgenology 2009; 192:1682-1689.
81.   Taylor SA, Burling D, Roddie M, et al. Computer-aided detection for CT
      colonography: incremental benefit of observer training. British Journal of
      Radiology 2008; 81:180-186.
82.   Taylor SA, Charman SC, Lefere P, et al. CT colonography: investigation of the
      optimum reader paradigm by using computer-aided detection software. Radiology
      2008; 246:463-471.
83.   Taylor SA, Greenhalgh R, Ilangovan R, et al. CT colonography and computer-
      aided detection: effect of false-positive results on reader specificity and reading
      efficiency in a low-prevalence screening population. Radiology 2008; 247:133-
      140.
84.   Taylor SA, Iinuma G, Saito Y, et al. CT colonography: computer-aided detection
      of morphologically flat T1 colonic carcinoma. European Radiology 2008;
      18:1666-1673.
85.   Taylor SA, Suzuki N, Beddoe G, et al. Flat neoplasia of the colon: CT
      colonography with CAD. Abdominal Imaging 2009; 34:173-181.
86.   The JS, Schilling KJ, Hoffmeister JW, et al. Detection of breast cancer with full-
      field digital mammography and computer-aided detection. AJR American Journal
      of Roentgenology 2009; 192:337-340.
87.   Tourassi GD, Ike R, 3rd, Singh S, et al. Evaluating the effect of image
      preprocessing on an information-theoretic CAD system in mammography.
      Academic Radiology 2008; 15:626-634.
88.   Velikova M, Samulski M, Lucas PJ, et al. Improved mammographic CAD
      performance using multi-view information: a Bayesian network framework.
      Physics in Medicine & Biology 2009; 54:1131-1147.
89.   Verma B, Verma B. Novel network architecture and learning algorithm for the
      classification of mass abnormalities in digitized mammograms. Artificial
      Intelligence in Medicine 2008; 42:67-79.


                                       20
90.   Walsham AC, Roberts HC, Kashani HM, et al. The use of computer-aided
      detection for the assessment of pulmonary arterial filling defects at computed
      tomographic angiography. Journal of Computer Assisted Tomography 2008;
      32:913-918.
91.   Wang F, Gatsonis CA, Wang F, Gatsonis CA. Hierarchical models for ROC curve
      summary measures: design and analysis of multi-reader, multi-modality studies of
      medical tests. Statistics in Medicine 2008; 27:243-256.
92.   White CS, Flukinger T, Jeudy J, et al. Use of a computer-aided detection system
      to detect missed lung cancer at chest radiography. Radiology 2009; 252:273-281.
93.   White CS, Pugatch R, Koonce T, et al. Lung nodule CAD software as a second
      reader: a multicenter study. Academic Radiology 2008; 15:326-333.
94.   Yang HC, Chang CH, Huang SW, et al. Correlations among acoustic, texture and
      morphological features for breast ultrasound CAD. Ultrasonic Imaging 2008;
      30:228-236.
95.   Toledano AY, Toledano AY. Three methods for analysing correlated ROC
      curves: a comparison in real data sets from multi-reader, multi-case studies with a
      factorial design. Statistics in Medicine 2003; 22:2919-2933.
96.   Obuchowski NA, Beiden SV, Berbaum KS, et al. Multireader, multicase receiver
      operating characteristic analysis: an empirical comparison of five methods.
      Academic Radiology 2004; 11:980-995.
97.   Helvie M. Improving mammographic interpretation: double reading and
      computer-aided diagnosis. Radiol Clin North Am. Sep 2007;45(5):801-811, vi.
98    Fenton JJ, Taplin SH, Carney PA, et al. Influence of computer-aided detection on
      performance of screening mammography. N Engl J Med. Apr 5
      2007;356(14):1399-1409.
99    Gur D, Sumkin JH, Rockette HE, et al. Changes in breast cancer detection and
      mammography recall rates after the introduction of a computer-aided detection
      system. J Natl Cancer Inst. Feb 4 2004;96(3):185-190.
100   Gallas BD, Pennello GA, Myers KJ. Multireader multicase variance analysis for
      binary data. J. Opt. Soc. Am. A 2007; 24:B70-B80.
101   Obuchowski NA. ROC analysis. AJR American Journal of Roentgenology 2005;
      184:364-372.
102   Skaane P, Niklason L. Receiver operating characteristic analysis: a proper
      measurement for performance in breast cancer screening?[comment]. AJR
      American Journal of Roentgenology 2006; 186:579-580; author reply 580.




                                      21
IX. Attachments (on enclosed CD)

1.    Draft Panel Questions

2.    Computer-Assisted Detection Devices Applied to Radiology Images and Radiology
      Device Data-Premarket Notification [510(k)] Submissions

3.    Clinical Performance Assessment for PMA and 510(k) Submissions: Considerations
      for Computer-Assisted Detection Devices Applied to Radiology Images and
      Radiological Device Data

4.    Epidemiologic Review of Mammography CADe

5.    Epidemiologic Review of Colon and Lung CADe

6.    Summary Briefing of the March 2008 Panel

7.    Summary Minutes of the March 2008 Panel

8.    Transcripts of the March 2008 Panel

9.    Reference Articles since the March 2008 Panel




                                            22

								
To top