Docstoc

Validation of screening methods

Document Sample
Validation of screening methods Powered By Docstoc
					             Validation of
             screening methods
             (2002/657/EC)
              N. Van Wouwe




AFSCA-FAVV
                                 IPH
Definition (2002/657/EC)
   Screening method :
     used to detect the presence of a substance
     or class of substances at the level of interest.
     have the capability for a high sample
     throughput
    => are used to sift large numbers of samples for
     potential non-compliant results.

Exemple: ELISA, plate test, biosensor, receptor test,…
Definition (2002/657/EC)

   Minimum criteria to use an analytical
    method as screening method:

     must   be validated (traceability)

     must  have a false compliant rate of <5% (β-
      error) at the level of interest
  Performance characteristics for method
  validation (screening)
Qualitative method: identifies a substance on basis of its chemical, biological or
physical propriety (binary response: +/-, absence/presence)
Quantitative method: determines the amount or mass fraction of a substance
(response: numerical value of appropriate unit)


                   Detection    Decision   Trueness/   Precision   Selectivity/   Applicability/
                   Capability    Limit     Recovery                Specificity    Ruggedness/
                     CCß          CC                                               Stability

Qualitative    S       +           -           -           -            +               +
methods

Quantitative   S       +           -           -          +             +               +
methods

 + = determination is mandatory
Validation of screening test
   Definition of the scope of the method
             Analyte of group of analytes
             Range of concentration
             List of matrices
   Initial validation with the most often used matrice
    in national monitoring program
             Detection capacity (CCβ)
             Selectivity/Specificity
             Applicability/ Ruggedness/Stability
             Precision (only for semi-quantitative method)

If possible: different sources of blank material, different technicians, different
days on the same spiked sample
Validation of screening test
   Targeted test: for 1 compound
       validation for this compound
   Targeted test: for a family of compounds
       validation for 1 representative molecule of the family (antibody)
   Wide range test: for more than 50 different molecules
       Validation for at least a list of representative compounds
            Common pattern of activity on a specific bacteria?
            Common way of action (acting target)?
            Published reference data on validation available?
Proposition of the CRL for
antimicrobials (in milk)

Representative Compound                      Antimicrobial class
Cefalonium/Cephapirin/Cefquinome             CEPHALOSPORINS
Penicillin G/Cloxacillin                     PENICILLINS
Tetracycline/Doxcycline                      TETRACYCLINS
Gentamicin/Streptomycin/Spectinomycin        AMINOGLYCOSIDES
Enrofloxacin/Flumequine                      QUINOLONES
Sulfathiazole/sulfaguanidine/Sulfamerazine   SULFONAMIDES
Erythromycin/Tylosin                         MACROLIDES
Lincomycin                                   LINCOSAMIDES
Thiamphenicol                                PHENICOLATED
Trimethoprim/colistine                       MISCELLANEOUS
Performance characteristics

 Detection capacity
 Selectivity/Specificity
 Applicability/ Ruggedness/Stability
 Precision (only for semi-quantitative
  method)
Detection capability (CCβ)
   The smallest content of the substance that may
    be detected, identified and/or quantified in a
    sample with an error probability of β
     In case of MRPL, CCβ= lowest concentration at which
      the method is able to detect truly contaminated
      sample with a statistical certainty of 1-β
     In case of MRL, CCβ= concentration at which the
      method is able to detect the MRL concentrations with
      a statistical certainty of 1-β
Detection capability (CCβ)
   No permitted limit
       Analyse 20 blank materials => CCα = 3x signal/noise
        Analyse 20 blank materials fortified at CCα
                => CCβ = CCα + 1.64 x SDRW

       Calibration curve procedure (ISO 11843)
             Analyse of blank material fortified at 0 MRLP, 0.5 MRLP,
              1 MRLP, 1.5 MRLP and 2 MRLP
             Plot analytical results (y-axis) vs concentration(x-axis)

             CCα = y-intercept (blank) + 2.33 x SDRW

             CCβ = CCα + 1.64 x SDRW
Detection capability (CCβ)
   No permitted limit
     If   no quantitative results
              Analyse fortified blank samples at and above CCα
               (n ≥ 20 / concentration level)
              CCβ = concentration level where only ≤5% false compliant
               results remain
Detection capability (CCβ)

        Blank       CC            CCb
           +2.33xSDblank   +1.64xSDRW

                α=1%        β=5%




                                           Signal or
                                         Concentration
Detection capability (CCβ)
   Permitted limit (MRL)
       Analyse 20 blank materials fortified at MRL
                => CCα = MRL + 1.64 x SDRW
        Analyse 20 blank materials fortified at CCα
                => CCβ = CCα + 1.64 x SDRW
       Calibration curve procedure (ISO 11843)
              Analyse of blank materials fortified at 0.5 MRL, 1 MRL, 1.5
               MRL and 2 MRL
              Plot analytical results (y-axis) vs concentration(x-axis)
              CCα = MRL + 1.64 x SDRW
              CCβ = CCα + 1.64 x SDRW
Detection capability (CCβ)

        MRL          CC          CCb
          +1.64xSDMRL   +1.64xSDRW

              α=5%         β=5%




                                          Signal or
                                        Concentration
Performance characteristics
 Detection capacity
 Selectivity/Specificity
 Applicability/ Ruggedness/Stability
 Precision (only for semi-quantitative
  method)
Selectivity/specificity
   Specificity: ability of a method to
    distinguish between analyte being
    measured and other substances
             problem of interference?

       F(measuring technique, class of compounds,
        matrices,…)
Selectivity/specificity
   How to test specificity for qualitative screening method?
         Analyse 20 different blank samples and 20 positive samples
          (blind study, same or different days/technicians)


                                                Specificity= 100* NA/N-
              True positive     True negative
                  (N+)               (N-)
test result     Positive       False positive   Other parameters:
 positive      agreement           (FP)         Accuracy= 100* (PA+NA)/(N- + N+)
                  (PA)
                                                Sensitivity= 100* PA/N+
test result   False negative      Negative
 negative          (FN)        agreement (NA)   False positive= 100* FP/(N- + N+)
                                                False negative= 100* FN/(N- + N+)
Selectivity/specificity
   How to test specificity for semi-quantitative
    screening methods?
       Select potentially interfering substances (metabolites,
        derivatives,…)
       Analyse relevant blank samples (n ≥ 20)
       Analyse fortified blank samples with interfering substances at a
        relevant concentration
       Estimate the effect of the interferences
                  False identification?
                  Influence in quantification?
                  Identification of the target analyte is hindered?
Performance characteristics
 Detection capacity
 Selectivity/Specificity
 Applicability/ Ruggedness/Stability
 Precision (only for semi-quantitative
  method)
Applicability
   Scope of the method must be define in term of :
     Matrix(solid/liquid matrix, type of tissue)
     Animal species

   To introduce a new matrix
       Analyse at least 10 different blank material fortified at level
        of interest for the new matrix (CCβ) + test of interferences
            If 10 positive results => method applicable for the new matrix
            If 1 negative result => 10 additional analyses
                  If 1 negative result=> CCβ must be recalculated for the new
                   matrix
Ruggedness
   Ruggedness: the susceptibility of an
    analytical method to changes in
    experimental conditions
        sample material
        analytes

        storage condition

        environmental condition

        sample preparation condition
Ruggedness
   How to test ruggedness? (during development)
     Identify possible factor that could influence             the results
      (the analyst, solvents, pH, T°, rate of heating,…)
     Vary  each factor slightly
     If one factor is found to influence results of the
      representative molecule, conduct further experiments
               => acceptability limits for this factor
                       (in the method protocol)

Recommendation of CRL: analyses of 10 blank and 10 spiked samples at the
same concentration and with minor change of factor to detect influence on results
Stability
   Test are not necessary if stability data already
    exist (from other lab or from publication)
     To    include in the validation report


   Stability test:
     the  analyte in solution
     the analyte in matrix
    Aliquots of a fresh solution or sample stored under
      different conditions (T° and/or storing time)
Performance characteristics
 Detection capacity
 Selectivity/Specificity
 Applicability/ Ruggedness/Stability
 Precision (only for semi-quantitative
  method)
Precision (for quantitative
screening)
   Precision: the closseness of agreement
    between independent test results obtained
    under predetermined conditions

   Expressed in terms of imprecision /
    standard deviation of test results
Precision (for quantitative
screening)

   How to test precision?
     Repeatability test
     within-laboratory reproducibility test(or intermediate
      precision)
     Reproducibility test (between laboratories:
      interlaboratory studies)

              determination of RSD (%) < Precision criteria
Precision (for quantitative
screening)
   Repeatability                   Within-laboratory
                                     Reproducibility
    3   concentrations:             3   concentrations:
            1x; 1,5x; 2x MRPL               1x; 1,5x; 2x MRPL
            0,5; 1x; 1,5x MRL               0,5; 1x; 1,5x MRL
    6  replicates/level             6  replicates/level
     3 times                         3 times
     same conditions                 different conditions
                                       (analyst, env.
                                       condition,…)
Precision (for quantitative
screening)
   ANOVA treatment of data => RSDr & RSDRW

   Comparison with precision criteria:
        Horwitz equation: RSDR(%) = 2(1-0.5logC)
        Criteria for repeatability: RSDr = 1/2 to 2/3 RSDR
        Criteria for within-lab reproducibility: RSDRW = 2/3 to 1 RSDR




     !     For concentration < 100 µg/kg, RSDR becomes too high!
Other recommendations
   False negative rate <5%: Analyses of 20 negative and 20
    positive samples in order to test the screening method (see
    selectivity).

   One QC sample must be added in routine and results must be
    added to the validation file

   Method transfer/Commercial test
       Bibliographical survey to compil the evaluation of performance of the
        test
       Collection of data from supplier on validation study
       Experimental plan to test skillness of technician to perform the test
       Use of QC sample
       Participation to proficiency test
Exemple: analyse of PCDD/F by
CALUX bioassay

   PCDD/F: 17 toxic congeners to analyse in various
    matrices (TCDD=most toxic dioxin)
   Results expressed in TEQ (=Sum (CCixTEFi)i=1-17)
   MRL for each matrix (milk, meat, egg, fish oil,…)
   MRL expressed in pg TEQ/g fat or ng TEQ/ kg
   Reference method: GC-HRMS
   Screening method: immunoassay, bioassay,…
 Exemple: analyse of PCDD/F by
 CALUX bioassay
 CALUX bioassay= genetically modified cell-based bioassay (luciferase)
 Amount of light produced is proportional to the toxicity (TEQ) of extracts




All substances
fixing the Ah                                        Gene expression
receptor




                                                        LIGHT
Analyse of PCDD/F by CALUX
bioassay
   Advantage:
     Rapid
     Cheaper  than GC-HRMS
     Time for analyses

   Disadvantage:
     Variouscompounds can fix the Ah receptor
      (PAH, PCB, PHDD/F,…)
           specificity!!!!
 Analyse of PCDD/F by CALUX
 bioassay : protocol
                     Extraction of fat


                   Clean-up on silica acid
                   + carbon columns


Fraction with   Fraction             Fraction with
interfering     with PCBs            PCDD/F
compounds
                                    Evaporation


                                                     Reading
                                     Dosing plate
                                                     plate
Analyse of PCDD/F by CALUX
bioassay : validation

 Selectivity/specificity
 Ruggedness/Stability
 Precision
 Detection capability
Analyse of PCDD/F by CALUX
bioassay : selectivity

   Possible interfering compounds?
       PAH : mostly in environmental sample
       PCB: fractionation during clean-up
       Other compounds? (PHDD/F): dependant of the matrix? (matrix
        effect?)

   Results of the selectivity test:
       No interferences for feedstuff, milk, egg, fat
       Interferences for fish oil
         CALUX results = 2 x GC-HRMS results
Analyse of PCDD/F by CALUX
bioassay : selectivity
                                Matrix effect for fish oil
                                 25
CALUX TEQ value (pg TEQ/g oil)




                                 20
                                          y = 1,9161x + 1,999
                                                2
                                               R = 0,9287
                                 15



                                 10



                                  5



                                  0
                                      0    2            4          6           8       10   12

                                                    GC-HRMS TEQ value (pg TEQ/g oil)
Analyse of PCDD/F by CALUX
bioassay : ruggedness
   What are the critical point in the protocol?
     Carbon  column (interferences)
     Solvent (interferences)
     Curve (results)
     Evaporation time (recovery)
     Age of CALUX cell line (RSD)
     Analyse of PCDD/F by CALUX
     bioassay : ruggedness
          Carbon column: amount of carbon used
                                                                       0,7cc XCARB        (Rdt PCDD/F= 60%)                                                         1cc XCARB          (Rdt PCDD/F= 80%)
DX                                  100                                                                                               100

fraction                             90
                                     80
                                                                                                                                      90

                                                                                                                                      80
                                     70                                                                                               70
                       % recovery




                                                                                                                        % recovery
                                     60                                                                                               60

PCB                                  50
                                     40
                                                                                                                                      50
                                                                                                                                      40

fraction                             30                                                                                               30
                                     20                                                                                               20
                                     10                                                                                               10
                                         0                                                                                             0
                                              PCB    PCB   PCB   PCB    PCB   PCB   PCB   PCB   PCB   PCB   PCB   PCB
Not collected                                  81     77   123   118    114   105   126   167   156   157   169   189
                                                                                                                                            PCB
                                                                                                                                             81
                                                                                                                                                  PCB
                                                                                                                                                   77
                                                                                                                                                        PCB
                                                                                                                                                        123
                                                                                                                                                              PCB
                                                                                                                                                              118
                                                                                                                                                                    PCB
                                                                                                                                                                    114
                                                                                                                                                                           PCB
                                                                                                                                                                           105
                                                                                                                                                                                 PCB
                                                                                                                                                                                 126
                                                                                                                                                                                       PCB
                                                                                                                                                                                       167
                                                                                                                                                                                              PCB
                                                                                                                                                                                              156
                                                                                                                                                                                                    PCB
                                                                                                                                                                                                    157
                                                                                                                                                                                                          PCB
                                                                                                                                                                                                          169
                                                                                                                                                                                                                 PCB
                                                                                                                                                                                                                 189


fraction
                                                                       1,4cc XCARB        (Rdt PCDD/F= 80%)                                                         1,9cc XCARB

                              100                                                                                                     100

                                    90                                                                                                 90
                                    80                                                                                                 80

                                    70                                                                                                 70
                                                                                                                         % recovery
                % recovery




                                    60                                                                                                 60

                                    50                                                                                                 50

                                    40                                                                                                 40

                                    30                                                                                                 30

                                    20                                                                                                 20

                                    10                                                                                                 10

                                    0                                                                                                   0
                                             PCB    PCB    PCB   PCB   PCB    PCB   PCB   PCB   PCB   PCB   PCB   PCB                       PCB   PCB   PCB   PCB    PCB   PCB   PCB    PCB   PCB   PCB    PCB    PCB
                                              81     77    123   118   114    105   126   167   156   157   169   189                        81    77   123   118    114   105   126    167   156   157    169    189
Analyse of PCDD/F by CALUX
bioassay : ruggedness
   Evaporation time
                         8

                         7

                         6
        CALUX response




                         5

                         4

                         3

                         2

                         1

                         0
                             0   5       10         15          20       25   30
                                     evaporation time with dry extract
Analyse of PCDD/F by CALUX
bioassay : ruggedness

   Solvent: tested before use on a TCDD solution
    (antagonist/agonist effect)

   Curve: tested with an independant TCDD
    solution

   Age of CALUX cells: new cell every 2 months
Analyse of PCDD/F by CALUX
bioassay : precision

   Validation protocol
                  Day 1   Day 2   Day 3   Day 4   Day 5   Day 6

Blank solvent      1       1       1       1       1       1

Blank sample       1       1       1       1

Sample at MRL/2            3       3       2

Sample at MRL      6       2       3       3

Sample at 2MRL             3       2       3

Quality sample     1       1       1       1       10      10
Analyse of PCDD/F by CALUX
bioassay : precision

   ANOVA results for the TEQ determination of PCDD/F in
    feedstuff by CALUX bioassay
       At MRL (0.75ng TEQ/kg) :   XMRL= 0.751 ng TEQ/kg
                                   Sr= 0.063 => RSDr= 8.4%
                                   SRW=0.073 =>RSDRW = 9.7%


       At MRL/2 (0.376ng TEQ/kg) : XMRL/2= 0.464 ng TEQ/kg
                                   Sr= 0.051 => RSDr= 11%
                                                              RSD < 30%
                                   SRW =0.051 =>RSDRW = 11%   (2002/70/EC)

       At 2MRL (1.5ng TEQ/kg): X2MRL= 1.571 ng TEQ/kg
                                   Sr= 0.107 => RSDr= 6.8%
                                   SRW=0.115 =>RSDRW = 7.3%
Analyse of PCDD/F by CALUX
bioassay : detection capacity
   CCβ for the TEQ determination of PCDD/F in feedstuff
    by CALUX bioassay

       CCα = MRL + 1.64 x SRW
        CCα = 0.75 + 1.64 x 0.073 = 0.87 ng TEQ/kg

       CCβ = CCα + 2.33 x SRW
        CCβ = 0.87 + 2.33 x 0.073 = 1.04 ng TEQ/kg

                              2002/70/EC: false negative rate < 1% !

          => At a concentration of 1.04ng TEQ/kg, we are sure that the
          sample is a positive sample with 99% certainty
  Analyse of PCDD/F by CALUX bioassay
  :confirmatory range

COMPLIANT               SUSPICIOUS
                           ?                       NON COMPLIANT

        CC*         MRL          CC         CCb
            -2.33sMRL     +1.64sMRL +2.33ssample




       * =1%                  β=5%   α=5%               Signal or
                                                      Concentration
Analyse of PCDD/F by CALUX
bioassay :confirmatory range

   Lower limit of the confirmatory range for the TEQ
    determination of PCDD/F in feedstuff by CALUX
    bioassay
       CC*= MRL-2.33 x SDRW
        CC*= 0.75- 2.33 x 0.073 = 0.58 ng TEQ/kg

   Conclusion
       Sample lower than 0.58 ng TEQ/kg are negative with 99%
        certainty (false negative rate < 1%)
       Sample above 0.58 ng TEQ/kg must be confirmed by GC-HRMS

				
DOCUMENT INFO