Get documents about "Choosing and validating new kits and equipment
Shared by: HC111210045951
-
Stats
- views:
- 23
- posted:
- 12/9/2011
- language:
- English
- pages:
- 42
Document Sample
Choosing and validating new kits
and equipment
Keith Perry, Unit Head
Microbiological Diagnostics Assessment Service [MiDAS]
Evaluations & Standards Laboratory
BSMT meeting, Management in Microbiology Laboratories
November 2005
www.hpa-midas.org.uk
Choosing & validating kits & equipment
www.hpa-midas.org.uk
Choosing & validating kits & equipment
Today’s presentation
• Background to MiDAS
• Costs / Procurement
• CE Marking
• Evaluations
• Validations
• Issues common to Evaluations & Validations
• Practical considerations
www.hpa-midas.org.uk
MiDAS
Evaluations & Standards Laboratory [ESL]
Evaluations Unit Quality Control Standards Unit Quality System
[MiDAS] Reagents Unit Unit
Evaluate kits & Provide Write and Advise on quality
equipment and laboratories with co-ordinate the systems and
write reports on virology /serology consultation and audit
findings IQC reagents distribution of
standard
methods and
clinical testing
algorithms
www.hpa-midas.org.uk
MiDAS
Associated organisations / contracts
Centre for National
Evidence-based Blood
Purchasing Service
Input into NBS Kit Evaluation
Evaluation Centre
for Microbiological IVDDs
MiDAS Group committee
[ESL]
World
Health Commercial
Organisation contracts
WHO Collaborating Centre for
Diagnostic and Laboratory Support
www.hpa-midas.org.uk
www.hpa-midas.org.uk
www.hpa-midas.org.uk
Procurement
• Recent formation of Centre for
Evidence-based Purchasing (CEP-PASA)
• Focus on regional procurement hubs
• Improved support for clinical networks by
increasing their involvement in purchasing
• Joining up device performance data with
purchasing cycles
• Strategic role of procurement in supporting
adoption of new technology/ideas
www.hpa-midas.org.uk
www.pasa.nhs.uk/evaluation
www.hpa-midas.org.uk
CE Marking
IVDD Directive – 98/79/EC Common Technical Specifications
www.hpa-midas.org.uk
CE Marking
CE Marking does not cover:-
• Ease of Use
• Monitoring of internal QCs
• External QA
• Use of combinations ie kits/automated platforms
• Comparative data
• Evaluation and validation
www.hpa-midas.org.uk
Evaluations
A generic term for any study that measures the performance
capabilities of an assay.
While the main focus of MiDAS is evaluation of commercial devices,
our processes can also contribute to high quality evaluation of
innovative and in-house assays.
Aims
• To undertake comparative evaluations of devices intended for the
in-vitro diagnosis and management of microbiological infection
• To focus on device performance
• To be independent and unbiased
• To undertake evaluations in an accurate, efficient and timely
manner
• To provide informative reports
www.hpa-midas.org.uk
Evaluations
Why do we need evaluations?
• Best practice
• Relevant to important public health concerns
• Standardisation of methods
• Managed introduction of new technology
• Encourages development
www.hpa-midas.org.uk
Evaluations
www.hpa-midas.org.uk
Evaluations
Choosing HIV kits
Timing of detection of primary HIV following seroconversion
Murex
Vironostika Vitros ECi
HIV 1/2
Murex HIV HIV Uni-form II Ag/Ab anti-HIV
VK84/85
Ag/Ab 1/2
combination Ortho Biotest
Access HIV 1/2 NEW Ab-capt. HIV 1/2
AxSYM HIV ELISA recombinant Innotest
Ag/Ab Combo HIV-1/-2
GENSCREEN Enzygnost
Biotest HIV 1/2
PLUS HIV Ag-
Anti- plus
Ab Biotest Anti-
HIV
HIV TETRA
TETRA
ELISA
ELISA
*Earliest anti-HIV detection
Earliest HIV detection
0 5 10 15 20
Murex days
HIV 1.2.0 Murex ICE
VIDAS GE94/95
HIV HIV 1+2
DUO Abbott
3rd gen
Plus Clonesystems
Enzygnost
Detect-HIV v1
HIV
Vironostika
Integral IMx HIV Uniform II
Wellcozym plus O
HIV1/2
III plus e
Pasteur
Anti-HIV
Genscreen AxSYM HIV 1/2 gO
Version 2
= combined antigen-antibody = immunometric = Class specific antibody capture = antiglobulin / indirect www.hpa-midas.org.uk
Evaluations
Choosing HIV kits
Combined scores for commercial seroconversion panels
Total score
Assay Format (seroconversion Rank
& performance
panels)
Total = 295
®
Seroconversion with HIV p24 antigenæmia (PRB936)
AxSYM HIV Ag/Ab Combo Ag + Ab 204 1 50 AxSYM HIV Ag/Ab
Murex HIV Ag-Ab Combination Ag + Ab 199 2 45
GENSCREEN® Plus HIV Ag-Ab Ag + Ab 190 3
40
VIDAS HIV DUO Ag + Ab 189 4
35 VIDAS HIV DUO
Enzygnost® HIV Integral Ag + Ab 184 5 Abbott HIV antigen
Signal : cut-off value
30
Vironostika® HIV UniForm II Ag/Ab Ag + Ab 172 6
Genscreen® HIV 1/2 EIA version 2 Immunometric 166 7 25
Biotest Anti-HIV TETRA ELISA Immunometric 164 8 20
Genscreen HIV Ag/Ab
Murex HIV-1.2.O Immunometric 163 =9 15 Murex HIV Ag/Ab
Access ® HIV 1/2 NEW Immunometric 163 =9 10 Murex HIV 1.2.0 (Ab-only)
Vitros Eci anti-HIV 1+2 Immunometric 161 11 5
Vironostika HIV Ag/Ab
AxSYM® HIV 1/2 gO Immunometric 160 =12 0
IMx® HIV-1/HIV-2 III plus Immunometric 160 =12 0 5 7 12 14 19 21
Days since first bleed
Ortho® HIV-1/HIV-2 ELISA Immunometric 160 =12
Murex HIV 1+2 Immunometric 160 =12
Abbott HIV-1/HIV-2 3rd Generation Immunometric 157 16
Enzygnost® Anti-HIV 1/2 Plus Immunometric 155 17
Vironostika® HIV Uni-Form II plus O Immunometric 154 18
Murex ICE HIV-1.O.2 Antibody capture 153 19
Biotest Anti-HIV-1/2 recombinant Indirect/recomb 147 20
Clonesystems HIV-1 and HIV-2 EIA Indirect/peptide 117 21
Innotest HIV-1/HIV-2 Ab s p EIA Indirect/peptide 116 22 www.hpa-midas.org.uk
Evaluations
Choosing HIV kits
Sensitivity
Assay Number positive/ Sensitivity 95% confidence Range
number tested % interval S/CO
AxSYM HIV Ag/Ab Combo 501/501 100 99.3 – 100 3.2 – 80.0
Enzygnost HIV Integral 509/509 100 99.3 – 100 2.8 – 7.5
GENSCREEN PLUS HIV Ag-Ab 500/500 100 99.3 – 100 3.2 – 18.6
Murex HIV Ag/Ab Combination 507/508 99.8 98.9 – 100 0.7 – 16.4
PRISM HIV Ag/Ab Combo 161/161 100 97.7 – 100 4.1 – 44.2
VIDAS HIV DUO 485/485 100 99.2 – 100 42.1 – 51.2
Vironostika HIV UniForm II Ag/Ab 669/669 100 99.5 – 100 3.1 – 19.0
Specificity
Assay Number Initially Repeatedly Repeat 95% confidence
tested reactive reactive reactive interval,
rate, % %
AxSYM HIV Ag/Ab Combo 514 5 0 0.00 0 - 0.7
Enzygnost HIV Integral 527 4 0 0.00 0 - 0.7
GENSCREEN PLUS HIV Ag-Ab 8740 a 17 10 0.10 0 - 0.2
Murex HIV Ag/Ab Combination 531 4 1 0.20 0 - 1.0
PRISM HIV Ag/Ab Combo 1800 a 8 1 0.05 0 - 0.3
VIDAS HIV DUO 372 1 1 0.27 0 - 1.5
Vironostika HIV UniForm II Ag/Ab 2578 a 12 1 0.04 0 - 0.2
a = Specificity trial data from UK blood centres
www.hpa-midas.org.uk
Evaluations
Good evaluation practice
www.hpa-midas.org.uk
Evaluations
Defined procedures
Prioritisation system
Literature Review 16 stages
Preparation
Setup 10
11 stages
Technical Assessment 13 stages
Report 13 stages
Post publication 8 stages
Customer Service
& feedback
www.hpa-midas.org.uk
Evaluations
Collaborations
www.hpa-midas.org.uk
Evaluations
Benefits of collaborations
• To incorporate latest developments and test algorithms into
evaluation design
• To maximise mutual benefit
• To avoid conflicts in output
• To bring together information on all evaluations undertaken
• To work to publish results in a timely manner
• To use mechanisms already in place and to avoid
compromising good evaluation practice
www.hpa-midas.org.uk
Evaluations
Collaborations: Chlamydia trachomatis NAATs
Scheme for the evaluation of chlamydia SDA at Portsmouth
Urine
Routine testing
Routine
Report Retest
screen
result positives
by SDA
Aliquot & label
Aliquots & transport
To Portsmouth To Liverpool To UCLH for To STBRL for
for SDA for TMA Cobas PCR QPCR Rotor-Gene
Into TMA tubes
Daily transport
to other labs
To be tested by SDA: Add to
SDA pouch
Specimens initially screened at
Liverpool, UCLH & Portsmouth
Evaluation
MiDAS to identify discordants
Retest initially reactive & discordant results.
Data analysis & report
www.hpa-midas.org.uk
Extent of
NHS use
Evaluations
Prioritisation – CEP ‘Hopper’ system
Anyone can
submit proposals
Adoption/purchase
Evaluation Evaluation
project
proposals
Time
Prioritisation Board
• Chair
• Head of CEP
• Business Planning Manager
• Head of Commissioning and Delivery
• Technical Advisor
• PASA / Hubs (2 reps)
Evaluations
? • NICE (1 rep)
• NIIII (1 rep)
Special Interest Group • HTA (1 rep)
• NPSA (1 rep)
• MHRA (1 rep)
Priorities to be aligned to customer requirements
& peer-organisations www.hpa-midas.org.uk
Evaluations
Prioritisation
Serology Kits
Evaluation priority survey (pilot) – Serology kits
140
120
Antigen
IgM serology
100
total or IgG serology
80
60
40
20
0
Herpes I/II
HHV-6
Parvovirus B19
SARS
Toxoplasma
Measles
Rotavirus
Hepatitis A
Mycoplasma
Mumps
Adenovirus
Varicella Zoster
Other markers
Cytomegalovirus
Epstein Barr virus
Chlamydia
Influenza
Syphilis
Papillomavirus
Respiratory Syncytial Virus
Rubella
Legionella
Chart excludes ongoing evaluations ie HIV, HBV, HCV, syphilis (total), chlamydia NAATs
www.hpa-midas.org.uk
0
10
20
30
40
50
60
Adenovirus
Chlamydia
Cytomegalovirus
Epstein Barr virus
Hepatitis A
Prioritisation
Herpes I/II
HHV-6
Evaluations
Influenza
Legionella
Measles
Mumps
Mycoplasma
Nucleic acid tests
Papillomavirus
Parvovirus B19
Rotavirus
Respiratory Syncytial Virus
Rubella
SARS
Syphilis
Chart excludes ongoing evaluations ie HIV, HBV, HCV, syphilis (total), chlamydia NAATs
Evaluation priority survey (pilot) - Nucleic Acid Tests
Toxoplasma
Nucleic acid tests
Varicella Zoster
Other markers
www.hpa-midas.org.uk
Validation:
with special thanks to Dr. Ian Sharp & Sally-Ann Finn
Definition of validation
Validation is the evaluation of a process to determine its fitness for
a particular use.
Validation is the confirmation by examination and the provision of
objective evidence that the particular requirements for a specific
intended use are fulfilled (ISO 17025:2005)
www.hpa-midas.org.uk
Validation
Evaluation of the process
• It is not the kit or reagent in isolation that is being validated but the
whole process that it is being used in to produce the correct result
• You are validating your ability to achieve acceptable results with the
assay in question
www.hpa-midas.org.uk
Validation
www.hpa-midas.org.uk
Validation
Why do we need to validate assays?
• Good laboratory practice
• Protection from litigation
• CPA requirement
• F 1.2 Examination procedures shall be validated for their
intended use prior to introduction, and the methods used and
results obtained, recorded.
www.hpa-midas.org.uk
Validation
What do we need to validate?
• Commercial assays
–Can use the term commissioning
• In house assays
• Modified commercial assays
• Changes to assays
• Equipment-assay combinations
www.hpa-midas.org.uk
Validation
Commercial assays
• Just because they are CE marked does not mean that they do not
need to be validated (commissioned) to ensure that they are fit for
purpose
• Most assays are CE marked following self-declaration by the
manufacturer
–May be certified to ISO 13485
www.hpa-midas.org.uk
Validation
What’s needed?
Project team
Project lead
Project manager
Molecular skills
Serology skills
Statistical skills
Validation plan
www.hpa-midas.org.uk
Validation
Validation file
Process Commercial Modified New
assays commercial In-house
assays assays
Evidence of CE Marking ●●● - -
Independent Evaluation Reports ●●● - -
Published papers ●●● ● ●
Manufacturer's instructions/literature ●● ● -
In-house R & D - ●●● ●●●
Workbooks ●● ●●● ●●●
Laboratory validation on known samples ●● ●●● ●●●
IQA ●● ●● ●●
EQA ●● ●● ●●
QC data ●● ●● ●●
Relevant SOPs ●● ●● ●●
Error logging ●● ●● ●●
Customer complaints ●● ●● ●●
Estimated level of requirement from ●●● (high) to ● (low)
www.hpa-midas.org.uk
Issues common to Evaluations & Validations
Challenging specimen panel - Numbers
Assay No. of kits Positive Negative Seroconversion /
evaluated Performance panels
HIV kits 30 500* 2000** 20 panels
HCV kits 8 500* 2000** 20 panels
HBsAg kits 30 500* 2000** 20 panels
anti-HBc 6 316 680 5 panels
Syphilis 15 235 250 -
Chlamydia NAATs 4 500 ♀ / 500 ♂ 1500 -
* following introduction of IVDD Directive, numbers reduced to 200
** Number of blood donor specimens tested by the National Blood Service
www.hpa-midas.org.uk
Issues common to Evaluations & Validations
Challenging specimen panel - Numbers
Number of 100% 100%
specimens specificity specificity
lower 95% CI upper 95% CI
50 92.9 100
125 97.1 100
250 98.5 100
500 99.3 100
1000 99.6 100
2000 99.8 100
4000 99.9 100
8000 100 100
Confidence intervals should always be quoted
www.hpa-midas.org.uk
Issues common to Evaluations & Validations
Challenging specimen panel - Numbers
Carley, Dosman S, Jones SR, Harrison M (2005): Simple nomograms to calculate
sample size in diagnostic studies
Emerg Med J 2005; 22: 180-181
www.hpa-midas.org.uk
Issues common to Evaluations & Validations
Challenging specimen panel – Minimising bias
• Compare results with a suitable reference standard / algorithm
• Avoid including specimens pre-screened by kits that are part of
the evaluation
• Avoid including specimens known to share false reactivities with
particular groups of kits
• Be aware of discontinued use of challenging specimens within a
panel (eg due to low volume) which may apparently enhance
sensitivity/specificity calculations of kits tested at a later date.
www.hpa-midas.org.uk
Issues common to Evaluations & Validations
Challenging specimen panel – Representative
types
Specimen category Example
Disease stages syphilis primary, secondary early & late latent
HIV seroconversion
Subtypes HIV HIV-1, HIV-2, HIV-1-0, Major subtypes
Geographical HIV Different continents
Risk groups HIV Sexually active
Injecting Drug Users
Sex chlamydia Male / Female
Include specimens with High, Medium and Low reactivity, and unreactive specimens
www.hpa-midas.org.uk
Issues common to Evaluations & Validations
Challenging specimen panel
– in practice depends on:
• Specimen availability
• Specimen volume
• Number of kits to evaluate
• Specimen quality (eg freeze/thaw cycles)
• Ethical issues
• Access to diagnostic specimens
• Funds eg commercial seroconversion
panels
www.hpa-midas.org.uk
Practical Considerations
• Technical expertise required to perform the assay
• Training
• Test throughput
• Reliability of the supplier
• Service provision & continuity
• Requirement for specialised equipment
• Maintenance contracts
• Laboratory space
www.hpa-midas.org.uk
Challenges
• Responding to the need for a wide range of microbiological
device evaluations
• Joining device performance with procurement
• Earlier assessment and usage of innovative technologies
• Improved access to available evaluation results
eg National Evaluations Register
• Improved sharing of information
www.hpa-midas.org.uk
Acknowledgments
Evaluations & Standards Laboratory:
Ian Sharp
Sally-Ann Finn
Ruhi Siddiqui
Joe Vincini
Valerie Bevan
MiDAS-ESL:
Katrina Barlow
Michelle Cole
Johanna Curtis
Laura Dean
Galit Gonen
Fu Li
www.hpa-midas.org.uk
Can You Help?
Acquiring specimens
www.hpa-midas.org.uk
