Advancing Regulatory Science for Highly Multiplexed Microbiology/Medical
FDA Discussion of Possible Performance Validation Considerations
FDA is seeking input from academia, government, industry, clinical laboratories, and other
stakeholders regarding the approach towards the science-based validation of nucleic acid based
highly multiplexed microbiology/medical countermeasures (MCM) devices. FDA recognizes
performance validation as the most challenging issue for developers of these tests, especially for
those that are MCM-related.
To this end, FDA is sponsoring a public meeting on October 13, 2011 --- Advancing Regulatory
Science for Highly Multiplexed Microbiology/Medical Countermeasure Devices. The purpose of
this document is to introduce options that are being considered and to help direct discussions at
the upcoming public meeting. The discussion and public input solicited in this meeting may
inform FDA’s approach and, ultimately, help manufacturers overcome some of the current
challenges associated with generating performance data for highly multiplexed
*NOTE: This document contains a description of potential studies that might be
conducted, but in no way constitutes guidance from the Agency on device
development. We are providing this document before the public meeting to obtain
thoughtful feedback from those who have an interest and information to
contribute to this process and to generate fruitful discussions during the meeting.
3 1. BACKGROUND
5 For the purposes of this discussion, highly multiplexed microbiological/medical countermeasure
6 devices are defined as those that are nucleic acid-based and intended to simultaneously detect
7 and identify a large number of organisms (≥20)1 using a single, direct clinical specimen. These
8 devices involve testing multiple targets through a common process of sample preparation,
9 amplification and/or detection, and result interpretation. Organism identification is often based
10 on sequence information compared to databases created either by the device manufacturer or
11 otherwise available publicly.
The selection of multiplexes with ≥20 organisms is based on experience with devices approaching this level of
complexity and the challenges with validation.
13 These diagnostic devices present several advantages, such as simultaneously detecting co-
14 infections and efficiently identifying disease etiology in situations where many different
15 pathogens share a common clinical presentation. However, validating performance of these
16 devices with the confidence needed to inform clinical and public health decisions poses
17 significant scientific challenges. These challenges include:
19 1. Reliability of a positive or negative result. In general, the performance validation of these
20 devices leads to a large amount of analytical and clinical testing because each target analyte
21 or pathogen should be individually assessed as well as in combination with all other analytes.
22 These performance studies are useful to inform laboratorians and physicians about the
23 percentage of false positive and false negative test results that are expected for each analyte
24 and, ultimately, the entire multiplexed device.
25 2. Multiplex assay design (microorganism composition). The assay menu for a particular
26 device is designed in a way that takes into consideration the differential diagnosis for the
27 patient’s symptoms, and the intended specimen type. Inclusion of organisms that are not
28 relevant to the symptoms and the selected specimen type can unnecessarily confound and
29 complicate the validation of a highly multiplexed device and will increase the potential for
30 false positive results.
31 3. Result interpretation. Simultaneous testing for multiple organisms including low and high
32 prevalence pathogens makes result interpretation difficult in part because pathogen
33 prevalence affects corresponding predictive values2 of the test result. Because the prevalence
34 of the tested organisms is different, the interpretation of positive results, which depends on
35 positive predictive values for each organism, can present a challenge. Also, assessing the
36 clinical significance of multiple positive test results from one specimen is difficult as there is
37 no precedent for the clinical application of detecting a large number of organisms
39 4. Specimen type. Each claimed specimen type that is appropriate and clinically relevant for
40 the device indications should be validated, which increases the amount of testing.
41 5. Determination of “Clinical Truth”. “Clinical truth” for each organism tested is defined by
42 the corresponding Comparator Method (CM). For some specimen types, the sample volume
43 available may preclude running all CMs on the same specimen, and thus the determination of
44 clinical truth and determination of performance (especially specificity, negative agreement)
45 for each targeted organism usually leads to more complex study designs.
46 6. Specimen availability. For MCM-targets and emerging infectious diseases, the numbers of
47 prospectively available specimens is unpredictable or extremely limited due to prohibitively
48 low prevalence to conduct effective clinical studies. Likewise, the infrastructure and
49 capability to culture and work with live MCM-pathogens is not widely available.
50 7. Reliability of sequence databases. Test results are often based on sequence information
51 queried against public databases that contain widely accepted information obtained from
52 many sources and may not be appropriately validated or correctly annotated to assure that the
The positive predictive value of a test is the probability that a person with a positive test result
actually has the infection
53 data are accurate. Similarly, many devices may use proprietary databases to evaluate test
54 results, which will also need validation to demonstrate reliability.
55 8. Device modification. When a new target is added or a modification is made to a previously
56 cleared or approved multiplex device, not only the performance of the newly added organism
57 be demonstrated, but also the effect of the new analyte on the performance of the previously
58 validated, cleared or approved analytes must be demonstrated. As the number of targets of
59 the device increases, the amount of validation necessary to modify the existing device also
60 increases. This is an important consideration since some viruses have the potential to mutate
61 with a higher frequency, thus necessitating the ability to modify devices rapidly to detect
62 variable strains.
64 The performance validation studies for highly multiplexed Microbiology/MCM device
65 validation present unique challenges and require innovative approaches and regulatory science
66 solutions to validate their performance. The ultimate goal of any proposed concept for validation
67 of highly multiplexed Microbiology/MCM devices is to balance the need for validation studies to
68 ensure the highest level of performance and protect patient and public health, and the need to
69 avoid expectations that are overly burdensome on developers.
71 To properly evaluate a device, it is important to use appropriately designed assays, including the
72 selection of target signatures, selection of relevant specimen type(s), and incorporation of
73 controls. For a multiplexed device, the reliability of a positive or negative result from the
74 indicated specimen type increases if the analyte composition is clinically relevant and the
75 organism can be found in this specific specimen type. In addition, it is important to select the
76 correct specimen type for the organisms included in the multiplexed device assay menu. For
77 example, assay menus to aid in the diagnosis of an upper or lower respiratory tract infection are
78 significantly different from an assay menu to aid in the diagnosis of gastrointestinal infections.
79 Proper design of the assay menu and selection of the proper sample specimen greatly reduces the
80 potential occurrence of false positive results due to non-specific events. As a result, the overall
81 size of the multiplexed diagnostic device becomes more manageable in size, allows for
82 utilization of the less burdensome analytical and clinical performance validation, and promotes
83 clinically relevant highly multiplexed technologies.
85 The clinical and analytical evaluations discussed below are intended to include the appropriate
86 controls, including any positive and negative controls that are provided with the final device, as
87 well as any appropriate external controls necessary to ensure proper function of the device and
88 all its associated pre-analytical systems. The studies described below delineate fundamental
89 changes in the studies that FDA might consider for validation of highly multiplexed devices.
92 2. CLINICAL EVALUATION
94 FDA is considering evaluating the clinical performance of highly multiplexed
95 Microbiology/MCM devices by evaluating clinical sensitivity, with archived clinical specimens
96 and specificity, through a limited prospective study. Under this framework, clinical sensitivity
97 for each analyte (or positive percent agreement) could be evaluated using well defined
98 retrospective archived specimens confirmed positive by a corresponding comparator method
99 (CM). Clinical specificity (or negative percent agreement) could be determined using
100 prospective samples. Many of the fundamental hallmarks of a traditional clinical study would
101 still be applicable, where feasible, and are briefly discussed in this document.
103 The clinical evaluation of these devices could be conducted using specimens from the intended
104 use patient populations. Specifically, specimens from patients with the signs and symptoms of
105 infection for which the device is indicated could be used to evaluate performance for the claimed
106 specimen type and organisms included on the multiplex assay menu. The clinical samples could
107 be collected from clinical sites in different geographical locations representative of U.S.
108 demographics. If clinical specimens are not available in the United States (U.S.), then the use of
109 samples from outside of the U.S. might be warranted. In specialized cases of localized
110 outbreaks, a single geographic collection site could be considered. If the use of samples from
111 outside of the U.S. is necessary, the relevance of the studies would need to be documented and
112 justification for inclusion of those samples provided.
114 All relevant clinical and laboratory information available for the clinical study patients would
115 need to be collected. This would include age (children, adults, and geriatric population), days
116 since onset of symptoms, gender, patient population (i.e., outpatient, ER, hospitalized, immuno-
117 compromised), signs and symptoms, indications for testing, and any medications taken or
118 administered. The clinical information appropriate for consideration would vary with the study
119 group of interest.
121 Comparator methods would be FDA-cleared or approved devices, if available (appropriate
122 cleared/approved multiplexed devices would be encouraged). When FDA cleared or approved
123 devices are not available, a composite reference method of two well-validated amplification-
124 based assays followed by bi-directional sequencing could be used. Information regarding the
125 minimal validation steps for PCR based assays and sequencing criteria could be obtained through
126 Agency interaction.
129 3. CLINICAL SENSITIVITY (POSITIVE PERCENT AGREEMENT)
131 Discussion Point: Determination through use of positive archived specimens
133 Clinical sensitivity is the probability that a person infected with a certain pathogen will be
134 correctly identified by the device. Generally, clinical sensitivity is established through a
135 prospective clinical study. The number of patients enrolled in a prospective study is based on the
136 predicted prevalence and the requirement to achieve a predetermined point estimate of
137 performance and its associated measure of confidence (length of confidence interval). This
138 approach provides robust clinical evaluation for many types of diagnostic devices; however,
139 there are many considerations that preclude this format to be used for evaluating clinical
140 sensitivity for highly multiplexed devices.
142 Many pathogens, especially those associated with deliberate biological threats, have extremely
143 low prevalence; therefore prospective studies driven by disease prevalence become excessively
144 burdensome or impossible to conduct. In order to address this challenge FDA is considering the
145 use of retrospective archived clinical specimens as the sole source of positive specimens to
146 evaluate clinical sensitivity. The analysis would include a minimum of fifty positive specimens
147 per claimed pathogen (as determined by the CM) and negative specimens to eliminate bias.
149 The final number of positive samples for each pathogen included in the device might be driven
150 by the point estimate of sensitivity and the lower bound of 95% two-sided confidence interval.
151 These values would vary depending on the intended use of the device and would require
152 discussions with the FDA to determine the appropriate clinical sensitivity levels for each
153 pathogen of the multiplexed device. For example, a multiplexed device with an assay menu
154 composed of upper respiratory organisms could include a sufficient number of
155 archived/retrospective samples for each claimed analyte to generate a result with at least 90%
156 sensitivity (46/50) with a lower bound of the two-sided 95% confidence interval (CI) greater than
157 80% (e.g., for 46/50 the lower bound of the 95% CI is 81.2%)3. All positive archived samples
158 could be verified by the corresponding CM to ensure samples were properly archived, no
159 specimen degradation had occurred during storage, and that the samples were properly
160 identified4. In addition, any positive determination by the multiplex device could also be verified
161 by the CM as this provides additional information about multiplexed device performance.
162 Alternative approaches to establishing the positive disposition of samples could be considered,
163 however, and FDA would encourage discussion with the review division for interactive feedback
164 before executing the study. In this case, positive samples could be of the same specimen type,
165 collected from the appropriate intended use population. The samples selected for inclusion in
166 such a study could represent the clinically relevant range of concentrations for the particular
167 pathogen and not be biased to preferentially include samples with a higher level of target
168 pathogen. Archived nucleic acid specimens could be considered for inclusion in the analysis
169 provided that the original specimen was collected from the intended use population, the indicated
170 specimen type was collected and processed using the indicated pre-analytical steps, and
171 confirmation by the corresponding CM was done.
173 For MCM-related pathogens, mock positive clinical samples could be used to evaluate clinical
174 sensitivity since actual clinical human specimens, archived or otherwise, may not be available.
175 The mock clinical samples could be prepared by spiking cultured pathogen into individual
176 negative clinical specimens. For this analysis, fifty percent of the spiked specimens could be
177 made at the limit of detection (LoD) concentration, while the remaining fifty percent could span
178 the expected clinical range of pathogen concentrations. Justification of the expected clinical
179 range of pathogen concentrations for the indicated sample type would be provided by the
182 A multiple testing site approach could be used to evaluate clinical sensitivity of most highly
183 multiplexed microbiology devices. The archived samples (positive and negative) could be
184 randomly distributed evenly among three testing sites for analysis. Given the restrictions
185 associated with handling many of the MCM-related pathogens, arrangements to validate the
If the point estimate of 90% and lower bound of the 95% CI of 80% is not met with the minimum number of 50
specimens then additional archived samples would be tested (e.g., 50/55 = 90.9% (95% CI:80.4% to 96.1%) 54/60 =
90.0% (95%CI: 79.9% to 95.3%)).
Analysis with the CM can be conducted using a single reference site
186 clinical performance at qualified institutions with the capability to conduct the proper studies
187 would need to be made. Due to the logistical issues with this facet of the clinical sensitivity
188 validation, testing for MCM-related pathogens would need to be conducted at a single site.
191 4. CLINICAL SPECIFICITY
193 Discussion Point: Determination using a reduced number of prospective specimens
195 Clinical specificity is the probability that a person uninfected with a certain pathogen will be
196 correctly identified by the device. Generally, clinical specificity is evaluated through a
197 prospective study and comparison to “clinical truth” as determined by the corresponding CM.
198 There are many considerations that preclude the use of traditional clinical evaluation formats for
199 the evaluation of clinical specificity for highly multiplexed microbiology/MCM devices. For
200 example, current practice requires sufficient specimen volume for all CMs to be run on each
201 clinical specimen. This approach becomes problematic as the number of CMs necessary to
202 establish clinical truth increases with multiplexing. Likewise, some clinical specimens may not
203 have adequate volume to run all CMs (e.g., upper respiratory swabs). To overcome the
204 foreseeable challenges in evaluating clinical specificity due to increases in the number of CMs
205 and the limited available volume of certain sample types, the following study design might be
208 Evaluation of clinical specificity could take place using prospective collected specimens and
209 analysis at a minimum of three clinical sites in the U.S. All patients enrolled in the study would
210 need to have signs and symptoms and meet any additional inclusion criteria for the study.
212 Depending on the number of organisms of the multiplexed device and specimen type, the
213 following approaches might be considered suitable under such a framework:
215 1. Assuming all CMs could be run on each clinical specimen for the indicated sample type, a
216 minimum of 400 prospective samples would be needed for any prospective clinical study to
217 ensure a robust analysis of the general performance of the multiplex device in the hands of
218 the end user and in the intended environment. For example, a multiplexed device with an
219 upper respiratory menu would establish clinical specificity with the lower bound of the 95%
220 CI to exceed 90%5. For select agents, clinical specificity would be demonstrated to achieve a
221 point estimate of 99.5% with a lower bound of the 95% CI greater than 99%.
222 2. If sample volume becomes prohibitive to run all comparator tests, then a randomized
223 approach could be taken such that a minimum of 100 of each comparator test for each analyte
224 would be analyzed (see Figure 1 for additional details). In addition, provisions could be
225 made so that an adequate number of specimens could be analyzed for select agents in order to
226 meet the specificity performance criteria. Using the example of a 20-plex device where each
227 CM needs equal test volumes and allows five CM tests, the first sample could be tested with
228 comparative methods (CM1, CM2, CM3, CM4, CM5), the second sample tested with CMs as
For example, using the minimum number of 400 prospectively collected specimens the clinical specificity point
estimate would be 93.0% (372/400) with a 95% CI of 90.1% to 95.1%.
229 (CM6, CM7, CM8, CM9, CM10) and so on. After testing four samples, each CM would have
230 been applied one time. After testing the first four samples, a new array of integer numbers
231 from 1 to 20 in a random order could be generated and the next four samples could be tested
232 with comparative methods according to this new array. In this example, to generate 100 CM
233 tests, a minimum of 400 prospectively collected samples would be needed.
235 It is expected that during the analysis of prospectively collected specimens some may be positive
236 for pathogens by the multiplexed device. Therefore, in addition, if a sample has positive results
237 by the multiplexed device for a pathogen, this sample would need to be tested also by the
238 corresponding CM. Information about CM results that were driven by a positive result obtained
239 by the multiplex device would not be used directly in calculation of sensitivity and specificity as
240 it introduces bias into estimation of the multiplexed device performance. However, this
241 information would be useful to understand the multiplexed device overall performance.
243 Comparative performance of the multiplex device could be established using FDA-cleared or
244 approved devices, if available. Use of cleared multiplexed devices, when appropriate, would be
245 encouraged. When FDA cleared or approved devices are not available, a composite reference
246 method of two well validated PCR based assays followed by bi-directional sequencing could be
247 used. Information regarding the minimal validation steps for PCR based assays and sequencing
248 criteria could be obtained through Agency interaction.
Comparator Method (CM) Sufficient Test Volume (20-plex device)
CM1 CM2 CM3 CM4 CM5 CM6 CM7 CM8 CM9 CM10 CM11 CM12 CM13 CM14 CM15 CM16 CM17 CM18 CM19 CM20
… Specimen 400
Comparator Method (CM) Proposed Concept - Insufficient Test Volume (20-plex device)
CM1 CM2 CM3 CM4 CM5 CM6 CM7 CM8 CM9 CM10 CM11 CM12 CM13 CM14 CM15 CM16 CM17 CM18 CM19 CM20
251 … Specimen 400
255 Figure 1. “Clinical Truth” Determination.
257 The tables above depict a simplification of two potential scenarios for the determination of
258 “clinical truth.” The upper table, labeled Current Practice/Sufficient Test Volume, depicts 20
259 comparator methods (CM1-CM20). The left column depicts the specimens being tested. In this
260 case, 6 samples are being tested, and the specimen type has sufficient volume to do complete
261 testing with all comparator assays as indicated by the color shading. In the lower table, labeled
262 Proposed Concept/Prohibitive Test Volume, the specimen volume is limiting, thus not allowing
263 for all comparator tests to be run. In this example, only 5 comparator tests can be run and are
264 indicated by the color-coded regions on the lower half of the table forming a checkered pattern.
As shown above, when specimen volume precludes testing with all comparators, Sample 1 was
tested with comparators CM1-CM5, Sample 2 with CM6-CM10, etc. In this scenario, using the
study size criteria as proposed in the clinical specificity (negative agreement) section of this
document, the minimum total number of prospective samples needed to be collected and analyzed
would be 400, thus providing a minimum of 100 prospective samples tested by each comparator
method for each analyte on the multiplex assay menu.
265 5. MULTISITE REPRODUCIBILITY
267 Discussion Point: Analysis using representative organisms and pooling
269 While reproducibility studies are usually conducted in three different sites and include all
270 organisms detected by the device, we are considering that for MCM-targets that are select agents,
271 reproducibility could be evaluated using a single site due to organism handling restrictions.
272 Arrangements to validate reproducibility at qualified institutions with the capability to handle
273 select agents and conduct the proper studies would have to be made. Testing at the single site
274 could be conducted with multiple operators and instruments to assess reproducibility between
275 operators, runs and days at a site that represents the intended use environment. For organisms
276 that are not MCM-related, the reproducibility might be evaluated using a multisite approach, as
277 is usually done for singleplex assays.
279 The operator conducting the reproducibility study could carry out all pre-analytical steps in
280 addition to the testing. Intact cultured organisms could be spiked into the negative clinical
281 sample matrix.
283 In order to reduce the overall size of the study, while still collecting sufficient information to
284 determine the reproducibility of the multiplexed device, representative panels of organisms
285 might be used. Panels could be composed of representative organisms selected from the
286 multiplex menu and agreed upon through Agency feedback prior to initiating the evaluation.
287 The representative panel could be tested at the following proposed concentrations and multiple
288 pathogens could be pooled:
290 – Negative clinical matrix - If negative matrix is not available, the sponsor could
291 consult with FDA regarding the use of an artificial matrix
292 – LoD or cut-off (C95, positive approximately 95% of the time) - Cultured organisms
293 could be spiked into negative matrix at a determined LoD level.
294 – 2-3x LoD (C100, positive 100% of the time) - Cultured organisms could be spiked
295 into negative matrix at 2-3x LoD level (not 10x LoD)
297 The study factors that are held constant could be clearly identified in the study design and all
298 computation and statistical analysis would need to be indicated. The following variables also are
299 examples of those that would need to be assessed and included in such a study design:
301 – Extraction variability: samples used in reproducibility testing could be processed
302 from contrived spiked clinical specimens at the test site, including all pre-analytical
304 – Site and operator variability: the sponsor might include three or more sites (at least
305 two external and one in-house site) with multiple operators at each site. Operators
306 could reflect the intended end users of the device in terms of education and
308 – Between-instrument and between lot variability
309 – A minimum of five non-consecutive days to cover day-to-day variability of the
310 representative panel on the multiplex device
311 – A minimum of two runs per day (unless the assay design precludes multiple runs per
312 day) and two replicates of each panel member or pool per run to assess between-run
313 and within-run precisions
314 – For each analyte level (regardless of pooling) a total of 96 data points would be
317 This type of evaluation would need to be done for all MCM-related targets using the same test
318 levels outlined above.
321 6. CUTOFF
323 A sponsor would need to explain how the cut-off for each target/analyte was initially determined
324 as well as how it was validated. The cut-off could be determined using appropriate statistical
325 methods. For example, a distribution of the results, the 95th and 99th percentiles, and percentage
326 of the positive results for the clinical samples in a pilot study could be provided to support the
327 determination of the cutoff. The selection of the appropriate cut-off could be justified by the
328 relevant levels of sensitivity and specificity based on Receiver Operating Curve (ROC) analysis
329 of pilot studies with clinical samples (for details about ROC analysis, see CLSI document GP10-
330 A Assessment of the Clinical Accuracy of Laboratory Tests Using Receiver Operating
331 Characteristics (ROC) Plots; Approved Guideline; 1995). The performance of the candidate
332 multiplexed device using a pre-determined cut-off could be validated in an independent study
333 consistent with the claimed intended use of the device.
336 7. ANALYTICAL LIMIT OF DETECTION
338 Discussion Point: Use of molecular calibration of cultures, pooled measurements, and
339 Probit analysis
341 The Limit of Detection (LoD) provides a measure of the analytical sensitivity of an assay for a
342 particular target and is defined as the lowest concentration of target distinguishable from
343 negative samples and consistently detected in 95% of the samples. Proper determination of the
344 LoD is critical since many of the analytical validation studies, as well as the levels included in
345 the reproducibility analysis, are based on this target concentration. Generally, the LoD is
346 determined on a per analyte basis using cultured and titered stocks expressed as TCID50 or
347 pfu(cfu)/mL. Given many of the inherent problems with non-standardized culture methods along
348 with the variation in preparations, especially with regard to NAAT-based technologies, we are
349 considering the use of a more standardized approach. To this end, we might calibrate cultures
350 using molecular techniques with results expressed in genome equivalents. This approach would
351 provide more standardized information, and allow the FDA to compare analytical sensitivity
352 across submissions, reduce the overall cost investment for developers, especially in cases where
353 a large numbers of fastidious microorganisms need to be titered.
355 Determination of the LoD for each target included in the assay menu and each specimen type
356 would be necessary. This could be accomplished by limiting dilutions of calibrated target
357 material into negative (non-infected) clinical matrix. The target material could be made from
358 isolated culture material and could be calibrated using acceptable molecular approaches and
359 expressed as genome equivalents/mL. A preliminary evaluation of LoD could be done
360 individually and then substantiated with multiple replicates using pools of multiple targets. One
361 approach would be to prepare serial dilutions of these evaluation pools using appropriate pooled
362 negative sample matrix as a diluent. These pools could include sufficient volume for 3-5
363 replicates of each dilution to establish the preliminary LoD range. After the preliminary range is
364 established, the preliminary LoD could be confirmed by demonstrating a detection rate of 95%
365 using a minimum of 20 samples. Alternatively, sponsors could pool targets for the preliminary
366 estimation and final confirmation of LoD. This approach could be taken with the understanding
367 that developers provide justification for the number of targets included in the evaluation pool. It
368 is important to note that pooling of multiple targets may negatively affect the LoD; therefore, the
369 justification provided by developers would need to indicate what steps were taken to ensure that
370 the LoD obtained in the study is accurate. The use of Probit analysis could also be used to
371 establish LoD, provided that the study is appropriately designed.
373 An additional step for consideration during the determination of LoD could be the inclusion of a
374 preliminary fresh/frozen equivalence analysis. This would provide an early indication of assay
375 performance with samples that have been frozen. If significant differences were observed, frozen
376 specimens could not be used. If no differences in LoD were observed then fresh/frozen
377 equivalence could be further confirmed in other analytical and clinical studies.
379 We have concluded that there could be some risks associated with the changes proposed in the
380 determination of LoD. For example, culture stocks calibrated using molecular methods will
381 eliminate information associated with infectivity (i.e., the number of infectious
382 particles/organisms per volume). This could be an issue for molecular-based assays as they have
383 the potential to detect both intact, infectious organisms and residual nucleic acid. However, the
384 risk is perceived to be minimal as the presence of nucleic acid would be indicative of low
385 organism levels or possibly an infection in the process of being resolved.
388 8. CROSS-REACTIVITY
390 Discussion Point: Use of extracted and calibrated comprehensive nucleic acid panels
392 The purpose of this investigation is to assess potential cross-reactivity in the absence of target
393 organisms. In this study the developer examines the performance of multiplexed devices in the
394 presence of other organisms found in the indicated specimen matrix. Comprehensive panels of
395 organisms are designed to include all closely related organisms and other
396 pathogens/microorganisms that are likely found in similar specimens or known to commonly
397 cause similar symptoms as those included in the assay menu.
399 Under this framework, a developer would test medically relevant levels of viruses and bacteria
400 (106 cfu/ml or higher for bacteria and 105 pfu/ml or higher for viruses). Generally, the bacterial
401 and viral stocks would be cultured and titered using traditional methods; however, for the
402 purposes of highly multiplexed devices and the expanded size of the panels to be used, we are
403 considering the use of cultured stocks that have been calibrated using molecular approaches.
404 From these calibrated stocks, preparations of highly purified and qualified nucleic acids could be
405 made and used to evaluate cross-reactivity at the previously mentioned test levels.
407 Developers could pool targets for the evaluation of cross-reactivity, provided that they justify the
408 number of targets included in the evaluation pool. It is important to note that pooling of multiple
409 targets could mask cross-reactive events due to overloading of nucleic acid; therefore, developers
410 would need to indicate what steps were taken to obviate this potential issue. If a pooling
411 approach was employed, and justified, all cross reactive pools could be further analyzed as
412 individual reactions to determine the cross-reactive organism(s).
415 9. ANALYTICAL REACTIVITY (INCLUSIVITY/EXCLUSIVITY)
417 Discussion Point: Extracted and calibrated nucleic acid panels and culture stocks, in
418 silico analysis
420 A validation of assay analytical reactivity evaluates clinically relevant organisms that represent
421 temporal, geographical, and phylogenic diversity for each claimed target at or near the LoD. For
422 this evaluation, FDA is considering the use of panels designed to include different strains,
423 laboratory isolates, serotypes, and other closely related subspecies relevant to the specimen type.
424 Panel design for inclusivity would need to incorporate a diverse, and clinically relevant, sample
427 Generally, the bacterial and viral stocks would be cultured and titered using traditional methods.
428 However, for the purposes of highly multiplexed devices and the expanded size of the analytical
429 reactivity panels, FDA is considering the use of cultured stocks that would have been calibrated
430 using molecular approaches. From these calibrated stocks, preparations of highly purified and
431 qualified nucleic acids could be made and evaluated at LoD. Additionally, a small subset of
432 representative organisms could also be analyzed as intact spiked samples, using all pre-analytical
435 In addition to traditional bench testing, some in silico analysis of organisms that are not readily
436 available could also be used when absolutely necessary. We recognize that the diversity of
437 organisms necessary to validate inclusivity may not be readily available to all developers. In
438 cases where certain targets can not be obtained for “wet-testing”, developers could augment the
439 data using in silico techniques, provided that they could justify the in silico approach, include an
440 explanation of the techniques used and the validation of the analysis.
443 10. MATRIX INTERFERENCE
445 Discussion Point: Representative panels, molecular calibration of culture stocks
447 The evaluation of naturally occurring variation found in clinical specimens is critical to establish
448 the analytical performance of a device. The concept being proposed in this document may not
449 fully capture information on matrix interference because of the reduction in the clinical
450 evaluation complexity (mock/archived samples, small number of archived positive/prospective
451 samples). However, enough information would need to be collected on matrix interference by
452 analyzing representative targets from the multiplex menu to ensure device performance.
454 In this type of study, each type of organism included in the device (Gram (+) and Gram (-)
455 bacteria, fungi, or virus, etc.) could be represented on a small testing panel. The representative
456 panel could be composed of a predetermined number of each organism type, and the
457 representative organisms could be spiked into individual negative clinical samples at LoD. All
458 pre-analytical steps could be employed to evaluate the effects of matrix variation on
459 performance. The size and scope of this study would be affected by the pre-analytical processing
460 steps. Devices that have unique pre-analytical processing steps, either as a stand alone device or
461 incorporated into a unitized consumable would require more in depth analysis.
463 In addition to the analysis of the claimed specimen type, the sponsor could also spike purified
464 interferents into negative samples at increasing levels. The final concentration of the interferent
465 might approach calculated worse-case levels and be justified either through a peered literature
466 analysis or Agency feedback.
469 11. INTERNAL COMPETITION
471 Discussion Point: In silico analysis
473 The purpose of this evaluation is to demonstrate a lack of internal competition due to secondary
474 structure/binding between various oligonucleotides included in the multiplexed assay. In general
475 this information has not been included in current submissions. However, given the increasing
476 complexity of multiplexed devices, FDA is considering that sponsors provide an in silico
477 analysis of all primers, probes, and amplified targets, demonstrate that no adverse binding
478 interactions that result in potential performance erosion occur, discuss all evaluation parameters
479 used for the analysis, and any relevant follow-up validation data. Any potentially negative
480 interaction would need to be justified for inclusion on the assay menu.
483 12. AMPLIFICATION COMPETITION
485 Discussion Point: Include only relevant co-infection
487 The purpose of this evaluation is to demonstrate simultaneous amplification and detection of all
488 clinically relevant co-infections. The selection of targets to be included in this study would be
489 justified through a surveillance of the literature, combinations of pathogens that are known to
490 occur, or through Agency feedback. The sponsor could assess competition from simultaneous
491 target amplification using an analysis of spiked clinical samples at the appropriate levels.
492 Specifically, such a study would be designed to demonstrate that when one target is present at
493 high levels (maximum clinical range), amplification and detection of the co-infecting target,
494 present at low levels near the LoD, would not be affected.
497 13. CONCLUSION
499 This document is meant to provide discussion points for the validation of highly multiplexed
500 microbiology/medical countermeasure devices that will be presented at the public meeting.
501 FDA’s ultimate goal is to advance regulatory science for highly multiplexed devices used in
502 pathogen detection in order to ensure their safety and effectiveness and thereby provide potential
503 clinical and public health benefits.
505 FDA is seeking input from academia, government, industry, clinical laboratories, and other
506 stakeholders regarding the discussion points presented here, and particularly on the following
509 1. Clinical Application of Highly Multiplexed Microbiology Devices: Their clinical application
510 and public health/clinical needs; inclusion of MCM-related pathogens that are expected to be
511 rarely present in the tested specimens; the composition of clinically relevant panels of
512 pathogens; the interpretation of the test results taking into consideration the possible
513 detection of microorganisms that are not clinically relevant, and what is known and unknown
514 about co-infections.
515 2. Device Evaluation: How to evaluate the analytical and clinical performance of highly
516 multiplexed microbiology devices; approaches to device validation when positive specimens
517 are not easily available, which is the case for many MCM pathogens; sufficiency, feasibility,
518 and practicality of the proposed FDA evaluation approach to establish device performance.
519 3. Reference Databases: Quality criteria for establishing the accuracy of reference databases;
520 methods for curating, maintaining, securing and updating these databases; identifying the
521 current practice for creating and maintaining reference databases.