Advancing Regulatory Science for Highly Multiplexed Microbiology

Document Sample
Advancing Regulatory Science for Highly Multiplexed Microbiology Powered By Docstoc
					                                                     DRAFT, 9/14/11 

                                                    Concept Paper

          Advancing Regulatory Science for Highly Multiplexed Microbiology/Medical
                                  Countermeasure Devices
               FDA Discussion of Possible Performance Validation Considerations

      FDA is seeking input from academia, government, industry, clinical laboratories, and other
      stakeholders regarding the approach towards the science-based validation of nucleic acid based
      highly multiplexed microbiology/medical countermeasures (MCM) devices. FDA recognizes
      performance validation as the most challenging issue for developers of these tests, especially for
      those that are MCM-related.

      To this end, FDA is sponsoring a public meeting on October 13, 2011 --- Advancing Regulatory
      Science for Highly Multiplexed Microbiology/Medical Countermeasure Devices. The purpose of
      this document is to introduce options that are being considered and to help direct discussions at
      the upcoming public meeting. The discussion and public input solicited in this meeting may
      inform FDA’s approach and, ultimately, help manufacturers overcome some of the current
      challenges associated with generating performance data for highly multiplexed
      microbiology/MCM-related devices.

             *NOTE: This document contains a description of potential studies that might be
             conducted, but in no way constitutes guidance from the Agency on device
             development. We are providing this document before the public meeting to obtain
             thoughtful feedback from those who have an interest and information to
             contribute to this process and to generate fruitful discussions during the meeting.



 3    1.      BACKGROUND
 5    For the purposes of this discussion, highly multiplexed microbiological/medical countermeasure
 6    devices are defined as those that are nucleic acid-based and intended to simultaneously detect
 7    and identify a large number of organisms (≥20)1 using a single, direct clinical specimen. These
 8    devices involve testing multiple targets through a common process of sample preparation,
 9    amplification and/or detection, and result interpretation. Organism identification is often based
10    on sequence information compared to databases created either by the device manufacturer or
11    otherwise available publicly.

       The selection of multiplexes with ≥20 organisms is based on experience with devices approaching this level of
      complexity and the challenges with validation.

                                               DRAFT, 9/14/11 

13   These diagnostic devices present several advantages, such as simultaneously detecting co-
14   infections and efficiently identifying disease etiology in situations where many different
15   pathogens share a common clinical presentation. However, validating performance of these
16   devices with the confidence needed to inform clinical and public health decisions poses
17   significant scientific challenges. These challenges include:
19   1. Reliability of a positive or negative result. In general, the performance validation of these
20      devices leads to a large amount of analytical and clinical testing because each target analyte
21      or pathogen should be individually assessed as well as in combination with all other analytes.
22      These performance studies are useful to inform laboratorians and physicians about the
23      percentage of false positive and false negative test results that are expected for each analyte
24      and, ultimately, the entire multiplexed device.
25   2. Multiplex assay design (microorganism composition). The assay menu for a particular
26      device is designed in a way that takes into consideration the differential diagnosis for the
27      patient’s symptoms, and the intended specimen type. Inclusion of organisms that are not
28      relevant to the symptoms and the selected specimen type can unnecessarily confound and
29      complicate the validation of a highly multiplexed device and will increase the potential for
30      false positive results.
31   3. Result interpretation. Simultaneous testing for multiple organisms including low and high
32      prevalence pathogens makes result interpretation difficult in part because pathogen
33      prevalence affects corresponding predictive values2 of the test result. Because the prevalence
34      of the tested organisms is different, the interpretation of positive results, which depends on
35      positive predictive values for each organism, can present a challenge. Also, assessing the
36      clinical significance of multiple positive test results from one specimen is difficult as there is
37      no precedent for the clinical application of detecting a large number of organisms
38      simultaneously.
39   4. Specimen type. Each claimed specimen type that is appropriate and clinically relevant for
40      the device indications should be validated, which increases the amount of testing.
41   5. Determination of “Clinical Truth”. “Clinical truth” for each organism tested is defined by
42      the corresponding Comparator Method (CM). For some specimen types, the sample volume
43      available may preclude running all CMs on the same specimen, and thus the determination of
44      clinical truth and determination of performance (especially specificity, negative agreement)
45      for each targeted organism usually leads to more complex study designs.
46   6. Specimen availability. For MCM-targets and emerging infectious diseases, the numbers of
47      prospectively available specimens is unpredictable or extremely limited due to prohibitively
48      low prevalence to conduct effective clinical studies. Likewise, the infrastructure and
49      capability to culture and work with live MCM-pathogens is not widely available.
50   7. Reliability of sequence databases. Test results are often based on sequence information
51      queried against public databases that contain widely accepted information obtained from
52      many sources and may not be appropriately validated or correctly annotated to assure that the

      The positive predictive value of a test is the probability that a person with a positive test result
     actually has the infection

                                              DRAFT, 9/14/11 

53        data are accurate. Similarly, many devices may use proprietary databases to evaluate test
54        results, which will also need validation to demonstrate reliability.
55   8. Device modification. When a new target is added or a modification is made to a previously
56      cleared or approved multiplex device, not only the performance of the newly added organism
57      be demonstrated, but also the effect of the new analyte on the performance of the previously
58      validated, cleared or approved analytes must be demonstrated. As the number of targets of
59      the device increases, the amount of validation necessary to modify the existing device also
60      increases. This is an important consideration since some viruses have the potential to mutate
61      with a higher frequency, thus necessitating the ability to modify devices rapidly to detect
62      variable strains.
64   The performance validation studies for highly multiplexed Microbiology/MCM device
65   validation present unique challenges and require innovative approaches and regulatory science
66   solutions to validate their performance. The ultimate goal of any proposed concept for validation
67   of highly multiplexed Microbiology/MCM devices is to balance the need for validation studies to
68   ensure the highest level of performance and protect patient and public health, and the need to
69   avoid expectations that are overly burdensome on developers.
71   To properly evaluate a device, it is important to use appropriately designed assays, including the
72   selection of target signatures, selection of relevant specimen type(s), and incorporation of
73   controls. For a multiplexed device, the reliability of a positive or negative result from the
74   indicated specimen type increases if the analyte composition is clinically relevant and the
75   organism can be found in this specific specimen type. In addition, it is important to select the
76   correct specimen type for the organisms included in the multiplexed device assay menu. For
77   example, assay menus to aid in the diagnosis of an upper or lower respiratory tract infection are
78   significantly different from an assay menu to aid in the diagnosis of gastrointestinal infections.
79   Proper design of the assay menu and selection of the proper sample specimen greatly reduces the
80   potential occurrence of false positive results due to non-specific events. As a result, the overall
81   size of the multiplexed diagnostic device becomes more manageable in size, allows for
82   utilization of the less burdensome analytical and clinical performance validation, and promotes
83   clinically relevant highly multiplexed technologies.
85   The clinical and analytical evaluations discussed below are intended to include the appropriate
86   controls, including any positive and negative controls that are provided with the final device, as
87   well as any appropriate external controls necessary to ensure proper function of the device and
88   all its associated pre-analytical systems. The studies described below delineate fundamental
89   changes in the studies that FDA might consider for validation of highly multiplexed devices.
94   FDA is considering evaluating the clinical performance of highly multiplexed
95   Microbiology/MCM devices by evaluating clinical sensitivity, with archived clinical specimens
96   and specificity, through a limited prospective study. Under this framework, clinical sensitivity
97   for each analyte (or positive percent agreement) could be evaluated using well defined

                                              DRAFT, 9/14/11 

 98   retrospective archived specimens confirmed positive by a corresponding comparator method
 99   (CM). Clinical specificity (or negative percent agreement) could be determined using
100   prospective samples. Many of the fundamental hallmarks of a traditional clinical study would
101   still be applicable, where feasible, and are briefly discussed in this document.
103   The clinical evaluation of these devices could be conducted using specimens from the intended
104   use patient populations. Specifically, specimens from patients with the signs and symptoms of
105   infection for which the device is indicated could be used to evaluate performance for the claimed
106   specimen type and organisms included on the multiplex assay menu. The clinical samples could
107   be collected from clinical sites in different geographical locations representative of U.S.
108   demographics. If clinical specimens are not available in the United States (U.S.), then the use of
109   samples from outside of the U.S. might be warranted. In specialized cases of localized
110   outbreaks, a single geographic collection site could be considered. If the use of samples from
111   outside of the U.S. is necessary, the relevance of the studies would need to be documented and
112   justification for inclusion of those samples provided.
114   All relevant clinical and laboratory information available for the clinical study patients would
115   need to be collected. This would include age (children, adults, and geriatric population), days
116   since onset of symptoms, gender, patient population (i.e., outpatient, ER, hospitalized, immuno-
117   compromised), signs and symptoms, indications for testing, and any medications taken or
118   administered. The clinical information appropriate for consideration would vary with the study
119   group of interest.
121   Comparator methods would be FDA-cleared or approved devices, if available (appropriate
122   cleared/approved multiplexed devices would be encouraged). When FDA cleared or approved
123   devices are not available, a composite reference method of two well-validated amplification-
124   based assays followed by bi-directional sequencing could be used. Information regarding the
125   minimal validation steps for PCR based assays and sequencing criteria could be obtained through
126   Agency interaction.
131          Discussion Point: Determination through use of positive archived specimens
133   Clinical sensitivity is the probability that a person infected with a certain pathogen will be
134   correctly identified by the device. Generally, clinical sensitivity is established through a
135   prospective clinical study. The number of patients enrolled in a prospective study is based on the
136   predicted prevalence and the requirement to achieve a predetermined point estimate of
137   performance and its associated measure of confidence (length of confidence interval). This
138   approach provides robust clinical evaluation for many types of diagnostic devices; however,
139   there are many considerations that preclude this format to be used for evaluating clinical
140   sensitivity for highly multiplexed devices.
142   Many pathogens, especially those associated with deliberate biological threats, have extremely
143   low prevalence; therefore prospective studies driven by disease prevalence become excessively

                                                    DRAFT, 9/14/11 

144   burdensome or impossible to conduct. In order to address this challenge FDA is considering the
145   use of retrospective archived clinical specimens as the sole source of positive specimens to
146   evaluate clinical sensitivity. The analysis would include a minimum of fifty positive specimens
147   per claimed pathogen (as determined by the CM) and negative specimens to eliminate bias.
149   The final number of positive samples for each pathogen included in the device might be driven
150   by the point estimate of sensitivity and the lower bound of 95% two-sided confidence interval.
151   These values would vary depending on the intended use of the device and would require
152   discussions with the FDA to determine the appropriate clinical sensitivity levels for each
153   pathogen of the multiplexed device. For example, a multiplexed device with an assay menu
154   composed of upper respiratory organisms could include a sufficient number of
155   archived/retrospective samples for each claimed analyte to generate a result with at least 90%
156   sensitivity (46/50) with a lower bound of the two-sided 95% confidence interval (CI) greater than
157   80% (e.g., for 46/50 the lower bound of the 95% CI is 81.2%)3. All positive archived samples
158   could be verified by the corresponding CM to ensure samples were properly archived, no
159   specimen degradation had occurred during storage, and that the samples were properly
160   identified4. In addition, any positive determination by the multiplex device could also be verified
161   by the CM as this provides additional information about multiplexed device performance.
162   Alternative approaches to establishing the positive disposition of samples could be considered,
163   however, and FDA would encourage discussion with the review division for interactive feedback
164   before executing the study. In this case, positive samples could be of the same specimen type,
165   collected from the appropriate intended use population. The samples selected for inclusion in
166   such a study could represent the clinically relevant range of concentrations for the particular
167   pathogen and not be biased to preferentially include samples with a higher level of target
168   pathogen. Archived nucleic acid specimens could be considered for inclusion in the analysis
169   provided that the original specimen was collected from the intended use population, the indicated
170   specimen type was collected and processed using the indicated pre-analytical steps, and
171   confirmation by the corresponding CM was done.
173   For MCM-related pathogens, mock positive clinical samples could be used to evaluate clinical
174   sensitivity since actual clinical human specimens, archived or otherwise, may not be available.
175   The mock clinical samples could be prepared by spiking cultured pathogen into individual
176   negative clinical specimens. For this analysis, fifty percent of the spiked specimens could be
177   made at the limit of detection (LoD) concentration, while the remaining fifty percent could span
178   the expected clinical range of pathogen concentrations. Justification of the expected clinical
179   range of pathogen concentrations for the indicated sample type would be provided by the
180   developers.
182   A multiple testing site approach could be used to evaluate clinical sensitivity of most highly
183   multiplexed microbiology devices. The archived samples (positive and negative) could be
184   randomly distributed evenly among three testing sites for analysis. Given the restrictions
185   associated with handling many of the MCM-related pathogens, arrangements to validate the

        If the point estimate of 90% and lower bound of the 95% CI of 80% is not met with the minimum number of 50

      specimens then additional archived samples would be tested (e.g., 50/55 = 90.9% (95% CI:80.4% to 96.1%) 54/60 = 

      90.0% (95%CI: 79.9% to 95.3%)).

         Analysis with the CM can be conducted using a single reference site 

                                                     DRAFT, 9/14/11 

186   clinical performance at qualified institutions with the capability to conduct the proper studies
187   would need to be made. Due to the logistical issues with this facet of the clinical sensitivity
188   validation, testing for MCM-related pathogens would need to be conducted at a single site.
193           Discussion Point: Determination using a reduced number of prospective specimens
195   Clinical specificity is the probability that a person uninfected with a certain pathogen will be
196   correctly identified by the device. Generally, clinical specificity is evaluated through a
197   prospective study and comparison to “clinical truth” as determined by the corresponding CM.
198   There are many considerations that preclude the use of traditional clinical evaluation formats for
199   the evaluation of clinical specificity for highly multiplexed microbiology/MCM devices. For
200   example, current practice requires sufficient specimen volume for all CMs to be run on each
201   clinical specimen. This approach becomes problematic as the number of CMs necessary to
202   establish clinical truth increases with multiplexing. Likewise, some clinical specimens may not
203   have adequate volume to run all CMs (e.g., upper respiratory swabs). To overcome the
204   foreseeable challenges in evaluating clinical specificity due to increases in the number of CMs
205   and the limited available volume of certain sample types, the following study design might be
206   considered.
208   Evaluation of clinical specificity could take place using prospective collected specimens and
209   analysis at a minimum of three clinical sites in the U.S. All patients enrolled in the study would
210   need to have signs and symptoms and meet any additional inclusion criteria for the study.
212   Depending on the number of organisms of the multiplexed device and specimen type, the
213   following approaches might be considered suitable under such a framework:
215   1. Assuming all CMs could be run on each clinical specimen for the indicated sample type, a
216      minimum of 400 prospective samples would be needed for any prospective clinical study to
217      ensure a robust analysis of the general performance of the multiplex device in the hands of
218      the end user and in the intended environment. For example, a multiplexed device with an
219      upper respiratory menu would establish clinical specificity with the lower bound of the 95%
220      CI to exceed 90%5. For select agents, clinical specificity would be demonstrated to achieve a
221      point estimate of 99.5% with a lower bound of the 95% CI greater than 99%.
222   2. If sample volume becomes prohibitive to run all comparator tests, then a randomized
223       approach could be taken such that a minimum of 100 of each comparator test for each analyte
224       would be analyzed (see Figure 1 for additional details). In addition, provisions could be
225       made so that an adequate number of specimens could be analyzed for select agents in order to
226       meet the specificity performance criteria. Using the example of a 20-plex device where each
227       CM needs equal test volumes and allows five CM tests, the first sample could be tested with
228       comparative methods (CM1, CM2, CM3, CM4, CM5), the second sample tested with CMs as

       For example, using the minimum number of 400 prospectively collected specimens the clinical specificity point
      estimate would be 93.0% (372/400) with a 95% CI of 90.1% to 95.1%.

                                               DRAFT, 9/14/11 

229      (CM6, CM7, CM8, CM9, CM10) and so on. After testing four samples, each CM would have
230      been applied one time. After testing the first four samples, a new array of integer numbers
231      from 1 to 20 in a random order could be generated and the next four samples could be tested
232      with comparative methods according to this new array. In this example, to generate 100 CM
233      tests, a minimum of 400 prospectively collected samples would be needed.
235   It is expected that during the analysis of prospectively collected specimens some may be positive
236   for pathogens by the multiplexed device. Therefore, in addition, if a sample has positive results
237   by the multiplexed device for a pathogen, this sample would need to be tested also by the
238   corresponding CM. Information about CM results that were driven by a positive result obtained
239   by the multiplex device would not be used directly in calculation of sensitivity and specificity as
240   it introduces bias into estimation of the multiplexed device performance. However, this
241   information would be useful to understand the multiplexed device overall performance.
243   Comparative performance of the multiplex device could be established using FDA-cleared or
244   approved devices, if available. Use of cleared multiplexed devices, when appropriate, would be
245   encouraged. When FDA cleared or approved devices are not available, a composite reference
246   method of two well validated PCR based assays followed by bi-directional sequencing could be
247   used. Information regarding the minimal validation steps for PCR based assays and sequencing
248   criteria could be obtained through Agency interaction.

                                                   DRAFT, 9/14/11

                                     Comparator Method (CM) Sufficient Test Volume (20-plex device)
                        CM1 CM2 CM3 CM4 CM5 CM6 CM7 CM8 CM9 CM10 CM11 CM12 CM13 CM14 CM15 CM16 CM17 CM18 CM19 CM20
      Specimen 1
      Specimen 2
      Specimen 3
      Specimen 4
      Specimen 5
       … Specimen 400
                              Comparator Method (CM) Proposed Concept - Insufficient Test Volume (20-plex device)
                        CM1 CM2 CM3 CM4 CM5 CM6 CM7 CM8 CM9 CM10 CM11 CM12 CM13 CM14 CM15 CM16 CM17 CM18 CM19 CM20
    Specimen 1
    Specimen 2
    Specimen 3
    Specimen 4
    Specimen 5
251 … Specimen 400
255         Figure 1. “Clinical Truth” Determination.
257         The tables above depict a simplification of two potential scenarios for the determination of
258         “clinical truth.” The upper table, labeled Current Practice/Sufficient Test Volume, depicts 20
259         comparator methods (CM1-CM20). The left column depicts the specimens being tested. In this
260         case, 6 samples are being tested, and the specimen type has sufficient volume to do complete
261         testing with all comparator assays as indicated by the color shading. In the lower table, labeled
262         Proposed Concept/Prohibitive Test Volume, the specimen volume is limiting, thus not allowing
263         for all comparator tests to be run. In this example, only 5 comparator tests can be run and are
264         indicated by the color-coded regions on the lower half of the table forming a checkered pattern.
            As shown above, when specimen volume precludes testing with all comparators, Sample 1 was
            tested with comparators CM1-CM5, Sample 2 with CM6-CM10, etc. In this scenario, using the
            study size criteria as proposed in the clinical specificity (negative agreement) section of this
            document, the minimum total number of prospective samples needed to be collected and analyzed
            would be 400, thus providing a minimum of 100 prospective samples tested by each comparator
            method for each analyte on the multiplex assay menu.

                                               DRAFT, 9/14/11 

267          Discussion Point: Analysis using representative organisms and pooling
269   While reproducibility studies are usually conducted in three different sites and include all
270   organisms detected by the device, we are considering that for MCM-targets that are select agents,
271   reproducibility could be evaluated using a single site due to organism handling restrictions.
272   Arrangements to validate reproducibility at qualified institutions with the capability to handle
273   select agents and conduct the proper studies would have to be made. Testing at the single site
274   could be conducted with multiple operators and instruments to assess reproducibility between
275   operators, runs and days at a site that represents the intended use environment. For organisms
276   that are not MCM-related, the reproducibility might be evaluated using a multisite approach, as
277   is usually done for singleplex assays.
279   The operator conducting the reproducibility study could carry out all pre-analytical steps in
280   addition to the testing. Intact cultured organisms could be spiked into the negative clinical
281   sample matrix.
283   In order to reduce the overall size of the study, while still collecting sufficient information to
284   determine the reproducibility of the multiplexed device, representative panels of organisms
285   might be used. Panels could be composed of representative organisms selected from the
286   multiplex menu and agreed upon through Agency feedback prior to initiating the evaluation.
287   The representative panel could be tested at the following proposed concentrations and multiple
288   pathogens could be pooled:
290          –   Negative clinical matrix - If negative matrix is not available, the sponsor could
291              consult with FDA regarding the use of an artificial matrix
292          –   LoD or cut-off (C95, positive approximately 95% of the time) - Cultured organisms
293              could be spiked into negative matrix at a determined LoD level.
294          –   2-3x LoD (C100, positive 100% of the time) - Cultured organisms could be spiked
295              into negative matrix at 2-3x LoD level (not 10x LoD)
297   The study factors that are held constant could be clearly identified in the study design and all
298   computation and statistical analysis would need to be indicated. The following variables also are
299   examples of those that would need to be assessed and included in such a study design:
301          –   Extraction variability: samples used in reproducibility testing could be processed
302              from contrived spiked clinical specimens at the test site, including all pre-analytical
303              steps
304          –   Site and operator variability: the sponsor might include three or more sites (at least
305              two external and one in-house site) with multiple operators at each site. Operators
306              could reflect the intended end users of the device in terms of education and
307              experience
308          –   Between-instrument and between lot variability

                                              DRAFT, 9/14/11 

309          –   A minimum of five non-consecutive days to cover day-to-day variability of the
310              representative panel on the multiplex device
311          –   A minimum of two runs per day (unless the assay design precludes multiple runs per
312              day) and two replicates of each panel member or pool per run to assess between-run
313              and within-run precisions
314          –   For each analyte level (regardless of pooling) a total of 96 data points would be
315              collected
317   This type of evaluation would need to be done for all MCM-related targets using the same test
318   levels outlined above.
321   6.     CUTOFF
323   A sponsor would need to explain how the cut-off for each target/analyte was initially determined
324   as well as how it was validated. The cut-off could be determined using appropriate statistical
325   methods. For example, a distribution of the results, the 95th and 99th percentiles, and percentage
326   of the positive results for the clinical samples in a pilot study could be provided to support the
327   determination of the cutoff. The selection of the appropriate cut-off could be justified by the
328   relevant levels of sensitivity and specificity based on Receiver Operating Curve (ROC) analysis
329   of pilot studies with clinical samples (for details about ROC analysis, see CLSI document GP10-
330   A Assessment of the Clinical Accuracy of Laboratory Tests Using Receiver Operating
331   Characteristics (ROC) Plots; Approved Guideline; 1995). The performance of the candidate
332   multiplexed device using a pre-determined cut-off could be validated in an independent study
333   consistent with the claimed intended use of the device.
338          Discussion Point: Use of molecular calibration of cultures, pooled measurements, and
339          Probit analysis
341   The Limit of Detection (LoD) provides a measure of the analytical sensitivity of an assay for a
342   particular target and is defined as the lowest concentration of target distinguishable from
343   negative samples and consistently detected in 95% of the samples. Proper determination of the
344   LoD is critical since many of the analytical validation studies, as well as the levels included in
345   the reproducibility analysis, are based on this target concentration. Generally, the LoD is
346   determined on a per analyte basis using cultured and titered stocks expressed as TCID50 or
347   pfu(cfu)/mL. Given many of the inherent problems with non-standardized culture methods along
348   with the variation in preparations, especially with regard to NAAT-based technologies, we are
349   considering the use of a more standardized approach. To this end, we might calibrate cultures
350   using molecular techniques with results expressed in genome equivalents. This approach would
351   provide more standardized information, and allow the FDA to compare analytical sensitivity
352   across submissions, reduce the overall cost investment for developers, especially in cases where
353   a large numbers of fastidious microorganisms need to be titered.

                                               DRAFT, 9/14/11 

355   Determination of the LoD for each target included in the assay menu and each specimen type
356   would be necessary. This could be accomplished by limiting dilutions of calibrated target
357   material into negative (non-infected) clinical matrix. The target material could be made from
358   isolated culture material and could be calibrated using acceptable molecular approaches and
359   expressed as genome equivalents/mL. A preliminary evaluation of LoD could be done
360   individually and then substantiated with multiple replicates using pools of multiple targets. One
361   approach would be to prepare serial dilutions of these evaluation pools using appropriate pooled
362   negative sample matrix as a diluent. These pools could include sufficient volume for 3-5
363   replicates of each dilution to establish the preliminary LoD range. After the preliminary range is
364   established, the preliminary LoD could be confirmed by demonstrating a detection rate of 95%
365   using a minimum of 20 samples. Alternatively, sponsors could pool targets for the preliminary
366   estimation and final confirmation of LoD. This approach could be taken with the understanding
367   that developers provide justification for the number of targets included in the evaluation pool. It
368   is important to note that pooling of multiple targets may negatively affect the LoD; therefore, the
369   justification provided by developers would need to indicate what steps were taken to ensure that
370   the LoD obtained in the study is accurate. The use of Probit analysis could also be used to
371   establish LoD, provided that the study is appropriately designed.
373   An additional step for consideration during the determination of LoD could be the inclusion of a
374   preliminary fresh/frozen equivalence analysis. This would provide an early indication of assay
375   performance with samples that have been frozen. If significant differences were observed, frozen
376   specimens could not be used. If no differences in LoD were observed then fresh/frozen
377   equivalence could be further confirmed in other analytical and clinical studies.
379   We have concluded that there could be some risks associated with the changes proposed in the
380   determination of LoD. For example, culture stocks calibrated using molecular methods will
381   eliminate information associated with infectivity (i.e., the number of infectious
382   particles/organisms per volume). This could be an issue for molecular-based assays as they have
383   the potential to detect both intact, infectious organisms and residual nucleic acid. However, the
384   risk is perceived to be minimal as the presence of nucleic acid would be indicative of low
385   organism levels or possibly an infection in the process of being resolved.
390          Discussion Point: Use of extracted and calibrated comprehensive nucleic acid panels
392   The purpose of this investigation is to assess potential cross-reactivity in the absence of target
393   organisms. In this study the developer examines the performance of multiplexed devices in the
394   presence of other organisms found in the indicated specimen matrix. Comprehensive panels of
395   organisms are designed to include all closely related organisms and other
396   pathogens/microorganisms that are likely found in similar specimens or known to commonly
397   cause similar symptoms as those included in the assay menu.

                                               DRAFT, 9/14/11 

399   Under this framework, a developer would test medically relevant levels of viruses and bacteria
400   (106 cfu/ml or higher for bacteria and 105 pfu/ml or higher for viruses). Generally, the bacterial
401   and viral stocks would be cultured and titered using traditional methods; however, for the
402   purposes of highly multiplexed devices and the expanded size of the panels to be used, we are
403   considering the use of cultured stocks that have been calibrated using molecular approaches.
404   From these calibrated stocks, preparations of highly purified and qualified nucleic acids could be
405   made and used to evaluate cross-reactivity at the previously mentioned test levels.
407   Developers could pool targets for the evaluation of cross-reactivity, provided that they justify the
408   number of targets included in the evaluation pool. It is important to note that pooling of multiple
409   targets could mask cross-reactive events due to overloading of nucleic acid; therefore, developers
410   would need to indicate what steps were taken to obviate this potential issue. If a pooling
411   approach was employed, and justified, all cross reactive pools could be further analyzed as
412   individual reactions to determine the cross-reactive organism(s).
417          Discussion Point: Extracted and calibrated nucleic acid panels and culture stocks, in
418          silico analysis
420   A validation of assay analytical reactivity evaluates clinically relevant organisms that represent
421   temporal, geographical, and phylogenic diversity for each claimed target at or near the LoD. For
422   this evaluation, FDA is considering the use of panels designed to include different strains,
423   laboratory isolates, serotypes, and other closely related subspecies relevant to the specimen type.
424   Panel design for inclusivity would need to incorporate a diverse, and clinically relevant, sample
425   set.
427   Generally, the bacterial and viral stocks would be cultured and titered using traditional methods.
428   However, for the purposes of highly multiplexed devices and the expanded size of the analytical
429   reactivity panels, FDA is considering the use of cultured stocks that would have been calibrated
430   using molecular approaches. From these calibrated stocks, preparations of highly purified and
431   qualified nucleic acids could be made and evaluated at LoD. Additionally, a small subset of
432   representative organisms could also be analyzed as intact spiked samples, using all pre-analytical
433   steps.
435   In addition to traditional bench testing, some in silico analysis of organisms that are not readily
436   available could also be used when absolutely necessary. We recognize that the diversity of
437   organisms necessary to validate inclusivity may not be readily available to all developers. In
438   cases where certain targets can not be obtained for “wet-testing”, developers could augment the
439   data using in silico techniques, provided that they could justify the in silico approach, include an
440   explanation of the techniques used and the validation of the analysis.

                                               DRAFT, 9/14/11 

445          Discussion Point: Representative panels, molecular calibration of culture stocks
447   The evaluation of naturally occurring variation found in clinical specimens is critical to establish
448   the analytical performance of a device. The concept being proposed in this document may not
449   fully capture information on matrix interference because of the reduction in the clinical
450   evaluation complexity (mock/archived samples, small number of archived positive/prospective
451   samples). However, enough information would need to be collected on matrix interference by
452   analyzing representative targets from the multiplex menu to ensure device performance.
454   In this type of study, each type of organism included in the device (Gram (+) and Gram (-)
455   bacteria, fungi, or virus, etc.) could be represented on a small testing panel. The representative
456   panel could be composed of a predetermined number of each organism type, and the
457   representative organisms could be spiked into individual negative clinical samples at LoD. All
458   pre-analytical steps could be employed to evaluate the effects of matrix variation on
459   performance. The size and scope of this study would be affected by the pre-analytical processing
460   steps. Devices that have unique pre-analytical processing steps, either as a stand alone device or
461   incorporated into a unitized consumable would require more in depth analysis.
463   In addition to the analysis of the claimed specimen type, the sponsor could also spike purified
464   interferents into negative samples at increasing levels. The final concentration of the interferent
465   might approach calculated worse-case levels and be justified either through a peered literature
466   analysis or Agency feedback.
471          Discussion Point: In silico analysis
473   The purpose of this evaluation is to demonstrate a lack of internal competition due to secondary
474   structure/binding between various oligonucleotides included in the multiplexed assay. In general
475   this information has not been included in current submissions. However, given the increasing
476   complexity of multiplexed devices, FDA is considering that sponsors provide an in silico
477   analysis of all primers, probes, and amplified targets, demonstrate that no adverse binding
478   interactions that result in potential performance erosion occur, discuss all evaluation parameters
479   used for the analysis, and any relevant follow-up validation data. Any potentially negative
480   interaction would need to be justified for inclusion on the assay menu.
485          Discussion Point: Include only relevant co-infection
487   The purpose of this evaluation is to demonstrate simultaneous amplification and detection of all
488   clinically relevant co-infections. The selection of targets to be included in this study would be

                                              DRAFT, 9/14/11

489   justified through a surveillance of the literature, combinations of pathogens that are known to
490   occur, or through Agency feedback. The sponsor could assess competition from simultaneous
491   target amplification using an analysis of spiked clinical samples at the appropriate levels.
492   Specifically, such a study would be designed to demonstrate that when one target is present at
493   high levels (maximum clinical range), amplification and detection of the co-infecting target,
494   present at low levels near the LoD, would not be affected.
497   13.    CONCLUSION
499   This document is meant to provide discussion points for the validation of highly multiplexed
500   microbiology/medical countermeasure devices that will be presented at the public meeting.
501   FDA’s ultimate goal is to advance regulatory science for highly multiplexed devices used in
502   pathogen detection in order to ensure their safety and effectiveness and thereby provide potential
503   clinical and public health benefits.
505   FDA is seeking input from academia, government, industry, clinical laboratories, and other
506   stakeholders regarding the discussion points presented here, and particularly on the following
507   topics:
509   1. Clinical Application of Highly Multiplexed Microbiology Devices: Their clinical application
510      and public health/clinical needs; inclusion of MCM-related pathogens that are expected to be
511      rarely present in the tested specimens; the composition of clinically relevant panels of
512      pathogens; the interpretation of the test results taking into consideration the possible
513      detection of microorganisms that are not clinically relevant, and what is known and unknown
514      about co-infections.
515   2. Device Evaluation: How to evaluate the analytical and clinical performance of highly
516      multiplexed microbiology devices; approaches to device validation when positive specimens
517      are not easily available, which is the case for many MCM pathogens; sufficiency, feasibility,
518      and practicality of the proposed FDA evaluation approach to establish device performance.
519   3. Reference Databases: Quality criteria for establishing the accuracy of reference databases;
520      methods for curating, maintaining, securing and updating these databases; identifying the
521      current practice for creating and maintaining reference databases.


Shared By: