Concepts in Quality Control Data Management - PowerPoint by malj

VIEWS: 34 PAGES: 118

									Concepts in Quality Control Data
         Management
        Thomasine Newby BS, MT (ASCP)
             Bio-Rad Laboratories
           Quality System Specialist
                Field Marketing
                    Objectives
At the conclusion of this program, participant should
be able to:

 Understand how to plan statistical Quality Control
 Learn how to derive QC Statistics
 Evaluate QC using Westgard rules and graphs
 Discuss the types of errors found when evaluating
  QC and the resolution of these errors.
 Develop a method validation plan
           Reference Material
   Clinical and Laboratory Standards
                 Institute
 Statistical Quality Control for Quantitative Measurement
  Procedures: Principles and Definitions; Approved
  Guideline – Third Edition (C24-S3, Vol. 26 No. 25)
 Method Comparison and Bias Estimation Using Patient
  Samples; Approved Guideline – Second Edition (EP9-
  A2, Vol 22 No. 19)
 Evaluation of the Linearity of Quantitative Measurement
  Procedures: A Statistical Approach; Approved Guideline
  (EP6-A, Vol 23 No. 16)
        Quality
  Laboratory Quality means achieving
positive patient outcomes through control
     of pre-analytical, analytical, and
          post-analytical error.
             Pre-Analytical Errors
   Sample taken from wrong patient
   Wrong sample analyzed
   Incorrect test ordered/performed
   Patient sample not properly preserved
   Patient sample not properly collected – IV arm
   Incorrect or incomplete patient instruction and
    preparation
              Post-Analytical Errors
 Wrong result reported or charted – transcription errors
 Computer error – LIS integrity
 Incorrect interpretation of test results provided to
  clinician
 Incorrect patient normal ranges
 Results reported from an instrument not used to
  establish the normal range
                Analytical Errors
   Instrument malfunction
   Lack of maintenance
   Inadequate environmental control
   Inadequate electrical supply
   Inadequate water supply
   Inadequate operator training
   Incorrect calibration
   Use of compromised reagents and/or calibrators
  Purpose of Statistical Quality
            Control
    Statistical quality control procedures are intended
to monitor the analytical performance of a measurement
   procedure and alert analysts to problems that might
    limit the usefulness of a test result for its intended
                      medical purpose.
       Statistical Quality Control
 Performed by analyzing stable specimens (external QC)
  and comparing the observed values to the distributions
  expected under stable conditions.

 Involves calculating means and SDs for the QC to set
  appropriate limits and identifying situations that
  represent unstable operation.
        Statistical Quality Control
Derived statistics are used to make judgments
concerning:

 Quality of analytical results

 Whether system correction is necessary

 Whether patient tests are accepted or rejected

 Estimating performance parameters that can be
  compared to analytical and medical goals
Planning a Statistical QC
      Procedure
          Planning a Statistical QC
                 Procedure
   Define the quality specifications for the test.
   Select control materials.
   Determine method performance
   Identify QC strategies
   Predict QC performance
   Set goals for QC performance
   Select appropriate QC rules
  Define the Quality Specifications
 Total allowable error – the magnitude of measurement
  error that if exceeded would cause a test result to be of
  unacceptable quality (imprecision and bias).

 Medical important changes in test results (imprecision,
  bias, and pre-analytical variables).

 Biological variation (imprecision and bias)
        Selecting Control Materials
   Menu
   Shelf life/Open-vial Stability
   Vial to vial variability
   Medical relevant levels
   Challenge range of instrument
   Matrix/preservatives
   QC material different from calibrator material
   Number of levels and concentration – sufficient to
    determine proper method performance over the
    measuring range of interest.
  Determine Method Performance
 Performance characteristics – bias and imprecision

 Imprecision – estimated by repeated measurements
  of QC

 Bias – comparison with proficiency testing or
  interlaboratory QC program
          Identify QC Strategies
 What control materials used?

 How many control samples analyzed?

 Where the QC are placed on a run?

 What QC rules are applied?

 When the QC rules are evaluated?
         Predict QC Performance
 Direct indicator of QC performance – the expected
  number of unacceptable patient test results reported
  when the process is out of control
   - Depends on type and magnitude of
     out-of-control error
   - When the error occurs
   - How long the error lasts
 All depends on QC strategy and the probability that
  the QC rules and number of control measurements
  will detect the error.
Frequency Of Occurrence
   Set Goals for QC Performance
 Maximum allowable number of unaccepted results due
  to out-of-control error
 Maximum allowable probability of reported unacceptable
  results
 Minimum acceptable probability of detecting an error
 Maximum acceptable probability of false rejections.
 Overall goal for QC performance – to maximize the
  detection of an out-of-control condition for a
  procedure while minimizing the probability of false
  QC alerts.
   Select Appropriate QC Rules

   Various combinations of QC rules and
parameters can be selected to satisfy the QC
            performance goals.
QC Applications
                    QC Strategy
 Define QC material
 Decide the number of measurements and the
  location of QC
 Set the control limits
 Decide on the QC rules for acceptable run

 Outline the response to the data acceptance
  decision (out-of control)
         Frequency and Location
 Frequency of QC
   - QC must be analyzed once during a run
   - Run is defined as interval that the method is
     considered stable
 Location of QC
   - QC results evaluated before reported patients
   - Placement immediately after calibration may give
     false low imprecision and shift during run
Set Control Limits
Establishing Data Parameters
    Establishing Data Parameters

Determination of valid mean and standard deviation values
 are crucial to successful data rejection or acceptance by
                  error detection schemes.
 Establishing Data
    Parameters
Parallel Testing vs. Product Inserts
             CLIA Regulations
“The stated values of an assayed control material may be
   used as the target values provided the stated values
    correspond to the methodology and instrumentation
     employed by the laboratory and are verified by the
                        laboratory.”

                                        CLIA 88, Final Rule
                         Federal Register, February 2, 1992
                                        Standard 493.1218
      CLSI Recommendations
“If assayed controls are used, the values stated on the
  assay sheets provided by the manufacturer should be
    used as guides in setting the initial control limits for
  testing new materials. Actual values for the mean and
 standard deviation must be established by serial testing
 in the laboratory. The observed mean should fall within
         the range published by the manufacturer.”

                   Clinical and Laboratory Standards Institute
                                       Volume 26 Number 25
                                                        2006
                     QC Statistics

The first step in defining performance limits
► Mean. . . . . . . . . . . . . . . . . . . x
► Standard Deviation . . . . . . . .s
► Coefficient of variation . . . . . cv
                         Mean
 Arithmetic average of a set of data points

 Sum of values divided by the number of values

                     a + b + c ……+ n
                              n
             Standard Deviation
 A measure of variability of the data points

 The degree of dispersion around the mean

             SD = the positive square root of the
                   population variance
      Coefficient of Variation, CV
 Measure of variability – random or systematic

 Expressed as a percent

                 CV = SD/Mean X 100
     Calculating A
Mean/Standard Deviation

   CLSI Recommendations
        Standard
        Provisional
        Things to Consider When
        Calculating QC Statistics
 Do I have a sufficient number of data points for the
  calculation?
 Have I used a sufficient number of analytical runs to
  account for random variables?
  - Multiple calibrations
  - Multiple reagent lots
  - Maintenance
  - Technologists
  - Environmental conditions
        Things to Consider When
        Calculating QC Statistics

 Did I follow “normal” procedure each analytical run?

 Have I appropriately excluded outliers?
                Test for Outliers
 Suspect low value – arrange values from high to low
  value
 Suspect high value – arrange values from low to high
 Refer to table – “Significance limits for testing extreme
  values of a sample”
 Column – use 0.05 (5% of values may be outside 2sd)
 Row – use the number of data points
 Calculate value – compare to table
 Calculated value > table value – Reject
 Calculated value < table value – Accept
Table Value
     Calculating A
Mean/Standard Deviation

   NCCLS Recommendations
       Standard
       Provisional

   95% Confidence Interval
        Estimation of a 2s Range
    Estimation of a 2s Range for a small sample by
         calculating a 95% confidence interval.
SE=s/ n             SE=Standard Error of the Mean
n = 10                        n=5
x = 398.2                     x = 401.2
s = 5.41                      s = 4.97
SE = 1.71                     SE = 2.22
LL = 398.2 – (1.71)(2.262)    LL = 401.2 – (2.22)(2.776)
UL = 398.2 + (1.71)(2.262)    LL = 401.2 + (2.22)(2.776)
2s Range (est.) = 394.3-402.1 2s Range (est.) = 395.0-407.4
    Standard Deviation Index, SDI
CALCULATION:

       SDI= [Lab Mean – Peer Group Mean]
            Peer Group’s 1 standard deviation

Use to assess bias compared to a peer group

TARGET SDI IS 0.0, INDICATING THAT THE LAB’S
MEAN VALUE IS THE SAME AS THE PEER’S.
           Coefficient of Variation
                 Ratio, CVR
Calculation
  Lab’s Monthly CV
  Peer Monthly CV

IDEALLY, CVR ≤ 1.0, SINCE YOUR VALUES ARE FROM A
SINGLE LAB, WHILE THE PEER CV IS FROM SEVERAL.

IF CVR = 1.5 to 2.0, THE LAB IS 50-100% LESS PRECISE
THAN IT’S PEER GROUP, USUALLY REQUIRING
INVESTIGATION.
Frequency Of Occurrence
Set Quality Control
      Rules
     Westgard Rules
                 Westgard Rules
 Error Detection
  - Random
  - Systematic
 False Rejection
 Multi-rule improves error detection with a low
  probability of false rejection
 Minimum of 2 controls per run
 Not all rules are necessary for all tests
       Types of Analytical Errors
 Random
  - Precision
  - Inherent to the test system
  - Always present
  - Minimize any increase
 Systematic
  - Accuracy
  - Shifts
  - Trends
  Westgard Rules Error Detection

 Random Error – imprecision
   •1 2s, 1 3s, R4s


 Systematic Error – bias
   •1 3s, 2 2s, 4 1s, 10x
               Troubleshooting and Westgard Rules
                Multirule Flowchart- When a rule is violated,
Run Control     stop and investigate.

          N                                                  N
   12S             In Control, Accept Run
          o                            x                     o

 Yes

          N                  N                 N             N
   13     o       22         o         R4S     o       41S   o         10
   S              S                                                    X
         Yes           Yes       Yes               Yes           Yes

          Investigate, Reject              Review Current and Previous Runs
Out of Control
 Situations
  Troubleshooting
   Troubleshooting Random Error
 Was there a deviation from procedure?

 Was the correct equipment used?

 Is there potential for random malfunction in
  the test system?

 Is the test sensitive to technique?
   Troubleshooting Random Error
 Is this error a manifestation of pre-analytical error?

 Is the test system vulnerable to electrical power
  fluctuations?

 Is this truly random error or the initial stages of
  systematic error?
       Sources of Random Error
 Power supply

 Double pipetting of control sample

 Misplacement of control sample within the run

 Air bubbles in water supply

 Random air bubbles in reagent or sample pipette system
     Sources of Systematic Error:
           Shifts & Trends
 Improper alignment of sample or reagent pipettes
 Drift or shift in incubator chamber temperature
 Inappropriate temperature/humidity levels in the testing
  area
 Change of reagent or calibrator lot
 Deterioration of reagent while in use, storage or
  shipment
     Sources of Systematic Error:
           Shifts & Trends
 Deterioration of the calibrator while in use, storage, or
  shipment
 Deterioration of the control product while in use, storage
  or shipment
 Incorrect handling of the control product (e.g., freezing
  when not recommended)
 Inappropriate storage of control products in frost-free
  freezers
     Sources of Systematic Error:
           Shifts & Trends
 Dirty filter wheel

 Failing light source

 Use of non-reagent grade water in the test system

 Recent calibration

 Change in test operator
            Troubleshooting the
              Control Product
 Is the product being aliquoted and frozen?

 Is the product being properly stored before and after
  reconstitution?

 Is the reagent being reconstituted with volumetric
  pipettes?

 Is reagent-grade water used for reconstitution?
Troubleshooting
     With
   Graphics
   Troubleshooting
         Troubleshooting Using
         Graphical Techniques

 Levey-Jennings Charts

 Frequency Histograms

 Youden Plots
                 Action Plan for
                 Rule Violations
 Review last 10 data points (both levels)
 Do not re-run the control
 Consult troubleshooting guides
 Consult your lab’s procedure manual
 Evaluate the test system
 Check maintenance records
 Correct any problems
Method Validation
             Method Validation
 Comparing test method with reference method

 Comparing new test method to current method.
         Overview of Comparison
Evaluating an analytical method requires:
 Time for operators to become familiar with the device
  operation and procedures
 Time for operators to become familiar with the evaluation
  protocol
 Both new and current methods must be in proper control
  over evaluation period
 Enough data to ensure representative results from both
  test and comparative methods
    Device Familiarization Period
 Operators of both test and new methods
  - General operation: set-up, operation, and
    troubleshooting
  - Maintenance procedures
  - Methods of sample preparation
  - Calibration and quality control
                     Samples
 Samples – collected according to the lab
  procedures, properly stored, and no longer than
  analytes are stable.
 Range of measurement – evaluate the test method
  over the clinically significant range
  - Concentrations should cover analytical measurement
    range – the concentration interval claimed by the
    manufacturer to provide acceptable performance
                    Samples
 Number of samples – at least 40 samples meeting
  the criteria

 Duplicate measurements – duplicates can be analyzed
  on both new and current methods

 Sample Sequence – assign positions and reverse on
  duplicate runs (minimize carryover and drift)
              Time and Duration
 Analysis on both methods should occur within a time
  span consistent with analyte stability.

 If possible, use samples the day of draw.

 If stored, use proper storage and store samples for both
  methods the same.
                  Error and QC
 Record any errors (out-of-control) or human errors –
  do not include data in the analysis

 Quality Control – follow procedures for QC and repeat
  any run that is out-of-control.
             Final Assumptions
 Current method (old method) is assumed to be precise
  and accurate, free from interference.

 Variation is due to the new method.
                Evaluating Data
 Scatter Plots – Bias

 Linear Regression
Method Comparison – Bias Plot
Method Comparison – Bias Plot
Method Comparison – Bias Plot
                    Linear Regression
X Axis
    Old, Reference, “A”
Y Axis
    New, Comparative, “B”
 New, Comparative




                       Old, Reference
                              Linear Regression
Constant Error
 Affects accuracy (bias) throughout range,
 seen as change in Y-Intercept

                   1000
   ACTUAL VALUES




                                             Actual results
                    750
                                                              Line of identity
                                                              Slope, m = 1.00
                    500                    }Constant
                                             Error
                    250


                      0
                          0    250         500           750            1000

                                     EXPECTED VALUES
                             Linear Regression
Proportional Error
   Varies the Y value as the concentration of X increases. The results
   are seen as a change in the slope from 1.00
        1000         Actual results
   ACTUAL VALUES




                   750
                                     Proportional      Line of identity
                   500


                   250
                                }       Error          Slope, m = 1.00




                     0
                         0    250           500       750          1000

                                    EXPECTED VALUES
                 Linear Regression

Y = mX + b                         Error

 Y = 1.00(x) + 0                  None
   - Then Y = X, identical
 Y = 1.05(x) + 0                  Proportional
   - Then Y is 5% greater than X
 Y = 1.00(x) + 5                  Constant
   - Then Y = X + 5
r = correlation between methods
                   Linear Regression
Test Yields: Line equation Y = mX + b
    r = Correlation           N, number of
       coefficient            tested pairs
Method Validation
      Linearity
                      Linearity
 Device familiarization period

 Duration of experiment – short a time interval as
  possible (preferably one day)

 Specimen requirements – enough specimen to
  prepare dilutions and test each of 5 – 11 concentrations
  in duplicate
                       Samples
 Matrix – similar to specimens used in testing

 Patient pools – ideal matrix with analyte near the upper
  reportable range

 Patient pools – supplemented with analyte

 Commercial – controls, calibrators, linearity material
         Patient Pooled Samples
 Low concentration pool – near and within the lower limit

 High concentration pool – the highest concentration
  tested

 Intermediate concentration pools – made by dilutions in
  constant relationship to each other and the low and high
              Linearity Purposes
 To establish linear range – 9 to 11 levels with 2 – 4
  replicates at each level

 To validate “in-house” or modified methods – 7 to 9
  levels with 2 – 3 replicates at each level

 To confirm linear range is valid – 5 to 7 levels with 2
  replicates
                      Testing
 Analytical sequence – random

 Analyte range
  - Concentration levels should be equal to the minimum
    and maximum values in the performance claims
  - When establishing linear range – the levels should
    include a range 20% to 30% wider than anticipated
    measuring range (plan to eliminate nonlinear
    points to establish the widest possible range of
    acceptable linear response)
             Linearity Data Exam
 Plot results
  - Y = testing results
  - X = sample concentrations or relative concentrations

 Linear pattern – segment slopes will be approximately
  equal

 Nonlinear pattern – increasing or decreasing trend

 Outlier – any result far from other Y values - eliminate
                                  Linearity Graphs
Instrument Limitations
       Allow all values to be reported but eliminate outliers
                 1000                                    Line of identity
 ACTUAL VALUES




                            S
                            e
                              Replicate
                            r1
                  750       i
                              Replicate
                            e
                            s2
                            1
                  500       S                                  Outlier
                            e
                            r
                            i
                  250       e
                            s
                            2
                            S
                    0       e
                        0   r         250         500         750           1000
                            i
                            e
                            s
                                            EXPECTED VALUES
                            3
Linearity

            Slope
            Intercept
Linearity – Poor Precision & Bias


                             Slope
                             Intercept
Method Validation
     Precision
               Precision Testing
 Familiar with instrument operation

 Familiar with precision protocol

 Maintain instrument with proper QC

 Note any instrument malfunctions

 Collect sufficient data over a long period
  of time to make long term estimates
                     Precision
Within Run Precision, Preliminary

 Minimum of 10 replicates for each level

 Calculate mean, SD, and CV for all levels

 Use for troubleshooting

 May not reflect usual conditions
                      Precision
Day to Day Precision
 Extended testing – 20 days
   - 4 replicates a day – 2 test samples run in duplicate
 Abbreviated testing
   - 3 replicates x 7 runs
   - 4 replicates x 5 runs
 Variables – same as in routine testing
   - Personnel
   - Instrument and test conditions
 Problems – contact manufacturer
                Precision Example
n                       80           80             80
mean                    20.6         51.4           128.7
within run sd           1.1*         1.3            1.4*
within run CV, %        5.5*         2.6            1.1*
Total sd                1.4          1.4            2.8
Total CV, %             6.9          2.8            2.2

                *sd's similar, CV's 5 x different
   Golden Rule of
 Clinical Laboratory
       Science
“No result is better than a wrong result.”
Questions?

								
To top