Concepts in Quality Control Data Management - PowerPoint by malj

VIEWS: 34 PAGES: 118

									Concepts in Quality Control Data
        Thomasine Newby BS, MT (ASCP)
             Bio-Rad Laboratories
           Quality System Specialist
                Field Marketing
At the conclusion of this program, participant should
be able to:

 Understand how to plan statistical Quality Control
 Learn how to derive QC Statistics
 Evaluate QC using Westgard rules and graphs
 Discuss the types of errors found when evaluating
  QC and the resolution of these errors.
 Develop a method validation plan
           Reference Material
   Clinical and Laboratory Standards
 Statistical Quality Control for Quantitative Measurement
  Procedures: Principles and Definitions; Approved
  Guideline – Third Edition (C24-S3, Vol. 26 No. 25)
 Method Comparison and Bias Estimation Using Patient
  Samples; Approved Guideline – Second Edition (EP9-
  A2, Vol 22 No. 19)
 Evaluation of the Linearity of Quantitative Measurement
  Procedures: A Statistical Approach; Approved Guideline
  (EP6-A, Vol 23 No. 16)
  Laboratory Quality means achieving
positive patient outcomes through control
     of pre-analytical, analytical, and
          post-analytical error.
             Pre-Analytical Errors
   Sample taken from wrong patient
   Wrong sample analyzed
   Incorrect test ordered/performed
   Patient sample not properly preserved
   Patient sample not properly collected – IV arm
   Incorrect or incomplete patient instruction and
              Post-Analytical Errors
 Wrong result reported or charted – transcription errors
 Computer error – LIS integrity
 Incorrect interpretation of test results provided to
 Incorrect patient normal ranges
 Results reported from an instrument not used to
  establish the normal range
                Analytical Errors
   Instrument malfunction
   Lack of maintenance
   Inadequate environmental control
   Inadequate electrical supply
   Inadequate water supply
   Inadequate operator training
   Incorrect calibration
   Use of compromised reagents and/or calibrators
  Purpose of Statistical Quality
    Statistical quality control procedures are intended
to monitor the analytical performance of a measurement
   procedure and alert analysts to problems that might
    limit the usefulness of a test result for its intended
                      medical purpose.
       Statistical Quality Control
 Performed by analyzing stable specimens (external QC)
  and comparing the observed values to the distributions
  expected under stable conditions.

 Involves calculating means and SDs for the QC to set
  appropriate limits and identifying situations that
  represent unstable operation.
        Statistical Quality Control
Derived statistics are used to make judgments

 Quality of analytical results

 Whether system correction is necessary

 Whether patient tests are accepted or rejected

 Estimating performance parameters that can be
  compared to analytical and medical goals
Planning a Statistical QC
          Planning a Statistical QC
   Define the quality specifications for the test.
   Select control materials.
   Determine method performance
   Identify QC strategies
   Predict QC performance
   Set goals for QC performance
   Select appropriate QC rules
  Define the Quality Specifications
 Total allowable error – the magnitude of measurement
  error that if exceeded would cause a test result to be of
  unacceptable quality (imprecision and bias).

 Medical important changes in test results (imprecision,
  bias, and pre-analytical variables).

 Biological variation (imprecision and bias)
        Selecting Control Materials
   Menu
   Shelf life/Open-vial Stability
   Vial to vial variability
   Medical relevant levels
   Challenge range of instrument
   Matrix/preservatives
   QC material different from calibrator material
   Number of levels and concentration – sufficient to
    determine proper method performance over the
    measuring range of interest.
  Determine Method Performance
 Performance characteristics – bias and imprecision

 Imprecision – estimated by repeated measurements
  of QC

 Bias – comparison with proficiency testing or
  interlaboratory QC program
          Identify QC Strategies
 What control materials used?

 How many control samples analyzed?

 Where the QC are placed on a run?

 What QC rules are applied?

 When the QC rules are evaluated?
         Predict QC Performance
 Direct indicator of QC performance – the expected
  number of unacceptable patient test results reported
  when the process is out of control
   - Depends on type and magnitude of
     out-of-control error
   - When the error occurs
   - How long the error lasts
 All depends on QC strategy and the probability that
  the QC rules and number of control measurements
  will detect the error.
Frequency Of Occurrence
   Set Goals for QC Performance
 Maximum allowable number of unaccepted results due
  to out-of-control error
 Maximum allowable probability of reported unacceptable
 Minimum acceptable probability of detecting an error
 Maximum acceptable probability of false rejections.
 Overall goal for QC performance – to maximize the
  detection of an out-of-control condition for a
  procedure while minimizing the probability of false
  QC alerts.
   Select Appropriate QC Rules

   Various combinations of QC rules and
parameters can be selected to satisfy the QC
            performance goals.
QC Applications
                    QC Strategy
 Define QC material
 Decide the number of measurements and the
  location of QC
 Set the control limits
 Decide on the QC rules for acceptable run

 Outline the response to the data acceptance
  decision (out-of control)
         Frequency and Location
 Frequency of QC
   - QC must be analyzed once during a run
   - Run is defined as interval that the method is
     considered stable
 Location of QC
   - QC results evaluated before reported patients
   - Placement immediately after calibration may give
     false low imprecision and shift during run
Set Control Limits
Establishing Data Parameters
    Establishing Data Parameters

Determination of valid mean and standard deviation values
 are crucial to successful data rejection or acceptance by
                  error detection schemes.
 Establishing Data
Parallel Testing vs. Product Inserts
             CLIA Regulations
“The stated values of an assayed control material may be
   used as the target values provided the stated values
    correspond to the methodology and instrumentation
     employed by the laboratory and are verified by the

                                        CLIA 88, Final Rule
                         Federal Register, February 2, 1992
                                        Standard 493.1218
      CLSI Recommendations
“If assayed controls are used, the values stated on the
  assay sheets provided by the manufacturer should be
    used as guides in setting the initial control limits for
  testing new materials. Actual values for the mean and
 standard deviation must be established by serial testing
 in the laboratory. The observed mean should fall within
         the range published by the manufacturer.”

                   Clinical and Laboratory Standards Institute
                                       Volume 26 Number 25
                     QC Statistics

The first step in defining performance limits
► Mean. . . . . . . . . . . . . . . . . . . x
► Standard Deviation . . . . . . . .s
► Coefficient of variation . . . . . cv
 Arithmetic average of a set of data points

 Sum of values divided by the number of values

                     a + b + c ……+ n
             Standard Deviation
 A measure of variability of the data points

 The degree of dispersion around the mean

             SD = the positive square root of the
                   population variance
      Coefficient of Variation, CV
 Measure of variability – random or systematic

 Expressed as a percent

                 CV = SD/Mean X 100
     Calculating A
Mean/Standard Deviation

   CLSI Recommendations
        Standard
        Provisional
        Things to Consider When
        Calculating QC Statistics
 Do I have a sufficient number of data points for the
 Have I used a sufficient number of analytical runs to
  account for random variables?
  - Multiple calibrations
  - Multiple reagent lots
  - Maintenance
  - Technologists
  - Environmental conditions
        Things to Consider When
        Calculating QC Statistics

 Did I follow “normal” procedure each analytical run?

 Have I appropriately excluded outliers?
                Test for Outliers
 Suspect low value – arrange values from high to low
 Suspect high value – arrange values from low to high
 Refer to table – “Significance limits for testing extreme
  values of a sample”
 Column – use 0.05 (5% of values may be outside 2sd)
 Row – use the number of data points
 Calculate value – compare to table
 Calculated value > table value – Reject
 Calculated value < table value – Accept
Table Value
     Calculating A
Mean/Standard Deviation

   NCCLS Recommendations
       Standard
       Provisional

   95% Confidence Interval
        Estimation of a 2s Range
    Estimation of a 2s Range for a small sample by
         calculating a 95% confidence interval.
SE=s/ n             SE=Standard Error of the Mean
n = 10                        n=5
x = 398.2                     x = 401.2
s = 5.41                      s = 4.97
SE = 1.71                     SE = 2.22
LL = 398.2 – (1.71)(2.262)    LL = 401.2 – (2.22)(2.776)
UL = 398.2 + (1.71)(2.262)    LL = 401.2 + (2.22)(2.776)
2s Range (est.) = 394.3-402.1 2s Range (est.) = 395.0-407.4
    Standard Deviation Index, SDI

       SDI= [Lab Mean – Peer Group Mean]
            Peer Group’s 1 standard deviation

Use to assess bias compared to a peer group

           Coefficient of Variation
                 Ratio, CVR
  Lab’s Monthly CV
  Peer Monthly CV


IF CVR = 1.5 to 2.0, THE LAB IS 50-100% LESS PRECISE
Frequency Of Occurrence
Set Quality Control
     Westgard Rules
                 Westgard Rules
 Error Detection
  - Random
  - Systematic
 False Rejection
 Multi-rule improves error detection with a low
  probability of false rejection
 Minimum of 2 controls per run
 Not all rules are necessary for all tests
       Types of Analytical Errors
 Random
  - Precision
  - Inherent to the test system
  - Always present
  - Minimize any increase
 Systematic
  - Accuracy
  - Shifts
  - Trends
  Westgard Rules Error Detection

 Random Error – imprecision
   •1 2s, 1 3s, R4s

 Systematic Error – bias
   •1 3s, 2 2s, 4 1s, 10x
               Troubleshooting and Westgard Rules
                Multirule Flowchart- When a rule is violated,
Run Control     stop and investigate.

          N                                                  N
   12S             In Control, Accept Run
          o                            x                     o


          N                  N                 N             N
   13     o       22         o         R4S     o       41S   o         10
   S              S                                                    X
         Yes           Yes       Yes               Yes           Yes

          Investigate, Reject              Review Current and Previous Runs
Out of Control
   Troubleshooting Random Error
 Was there a deviation from procedure?

 Was the correct equipment used?

 Is there potential for random malfunction in
  the test system?

 Is the test sensitive to technique?
   Troubleshooting Random Error
 Is this error a manifestation of pre-analytical error?

 Is the test system vulnerable to electrical power

 Is this truly random error or the initial stages of
  systematic error?
       Sources of Random Error
 Power supply

 Double pipetting of control sample

 Misplacement of control sample within the run

 Air bubbles in water supply

 Random air bubbles in reagent or sample pipette system
     Sources of Systematic Error:
           Shifts & Trends
 Improper alignment of sample or reagent pipettes
 Drift or shift in incubator chamber temperature
 Inappropriate temperature/humidity levels in the testing
 Change of reagent or calibrator lot
 Deterioration of reagent while in use, storage or
     Sources of Systematic Error:
           Shifts & Trends
 Deterioration of the calibrator while in use, storage, or
 Deterioration of the control product while in use, storage
  or shipment
 Incorrect handling of the control product (e.g., freezing
  when not recommended)
 Inappropriate storage of control products in frost-free
     Sources of Systematic Error:
           Shifts & Trends
 Dirty filter wheel

 Failing light source

 Use of non-reagent grade water in the test system

 Recent calibration

 Change in test operator
            Troubleshooting the
              Control Product
 Is the product being aliquoted and frozen?

 Is the product being properly stored before and after

 Is the reagent being reconstituted with volumetric

 Is reagent-grade water used for reconstitution?
         Troubleshooting Using
         Graphical Techniques

 Levey-Jennings Charts

 Frequency Histograms

 Youden Plots
                 Action Plan for
                 Rule Violations
 Review last 10 data points (both levels)
 Do not re-run the control
 Consult troubleshooting guides
 Consult your lab’s procedure manual
 Evaluate the test system
 Check maintenance records
 Correct any problems
Method Validation
             Method Validation
 Comparing test method with reference method

 Comparing new test method to current method.
         Overview of Comparison
Evaluating an analytical method requires:
 Time for operators to become familiar with the device
  operation and procedures
 Time for operators to become familiar with the evaluation
 Both new and current methods must be in proper control
  over evaluation period
 Enough data to ensure representative results from both
  test and comparative methods
    Device Familiarization Period
 Operators of both test and new methods
  - General operation: set-up, operation, and
  - Maintenance procedures
  - Methods of sample preparation
  - Calibration and quality control
 Samples – collected according to the lab
  procedures, properly stored, and no longer than
  analytes are stable.
 Range of measurement – evaluate the test method
  over the clinically significant range
  - Concentrations should cover analytical measurement
    range – the concentration interval claimed by the
    manufacturer to provide acceptable performance
 Number of samples – at least 40 samples meeting
  the criteria

 Duplicate measurements – duplicates can be analyzed
  on both new and current methods

 Sample Sequence – assign positions and reverse on
  duplicate runs (minimize carryover and drift)
              Time and Duration
 Analysis on both methods should occur within a time
  span consistent with analyte stability.

 If possible, use samples the day of draw.

 If stored, use proper storage and store samples for both
  methods the same.
                  Error and QC
 Record any errors (out-of-control) or human errors –
  do not include data in the analysis

 Quality Control – follow procedures for QC and repeat
  any run that is out-of-control.
             Final Assumptions
 Current method (old method) is assumed to be precise
  and accurate, free from interference.

 Variation is due to the new method.
                Evaluating Data
 Scatter Plots – Bias

 Linear Regression
Method Comparison – Bias Plot
Method Comparison – Bias Plot
Method Comparison – Bias Plot
                    Linear Regression
X Axis
    Old, Reference, “A”
Y Axis
    New, Comparative, “B”
 New, Comparative

                       Old, Reference
                              Linear Regression
Constant Error
 Affects accuracy (bias) throughout range,
 seen as change in Y-Intercept


                                             Actual results
                                                              Line of identity
                                                              Slope, m = 1.00
                    500                    }Constant

                          0    250         500           750            1000

                                     EXPECTED VALUES
                             Linear Regression
Proportional Error
   Varies the Y value as the concentration of X increases. The results
   are seen as a change in the slope from 1.00
        1000         Actual results

                                     Proportional      Line of identity

                                }       Error          Slope, m = 1.00

                         0    250           500       750          1000

                                    EXPECTED VALUES
                 Linear Regression

Y = mX + b                         Error

 Y = 1.00(x) + 0                  None
   - Then Y = X, identical
 Y = 1.05(x) + 0                  Proportional
   - Then Y is 5% greater than X
 Y = 1.00(x) + 5                  Constant
   - Then Y = X + 5
r = correlation between methods
                   Linear Regression
Test Yields: Line equation Y = mX + b
    r = Correlation           N, number of
       coefficient            tested pairs
Method Validation
 Device familiarization period

 Duration of experiment – short a time interval as
  possible (preferably one day)

 Specimen requirements – enough specimen to
  prepare dilutions and test each of 5 – 11 concentrations
  in duplicate
 Matrix – similar to specimens used in testing

 Patient pools – ideal matrix with analyte near the upper
  reportable range

 Patient pools – supplemented with analyte

 Commercial – controls, calibrators, linearity material
         Patient Pooled Samples
 Low concentration pool – near and within the lower limit

 High concentration pool – the highest concentration

 Intermediate concentration pools – made by dilutions in
  constant relationship to each other and the low and high
              Linearity Purposes
 To establish linear range – 9 to 11 levels with 2 – 4
  replicates at each level

 To validate “in-house” or modified methods – 7 to 9
  levels with 2 – 3 replicates at each level

 To confirm linear range is valid – 5 to 7 levels with 2
 Analytical sequence – random

 Analyte range
  - Concentration levels should be equal to the minimum
    and maximum values in the performance claims
  - When establishing linear range – the levels should
    include a range 20% to 30% wider than anticipated
    measuring range (plan to eliminate nonlinear
    points to establish the widest possible range of
    acceptable linear response)
             Linearity Data Exam
 Plot results
  - Y = testing results
  - X = sample concentrations or relative concentrations

 Linear pattern – segment slopes will be approximately

 Nonlinear pattern – increasing or decreasing trend

 Outlier – any result far from other Y values - eliminate
                                  Linearity Graphs
Instrument Limitations
       Allow all values to be reported but eliminate outliers
                 1000                                    Line of identity

                  750       i
                  500       S                                  Outlier
                  250       e
                    0       e
                        0   r         250         500         750           1000
                                            EXPECTED VALUES

Linearity – Poor Precision & Bias

Method Validation
               Precision Testing
 Familiar with instrument operation

 Familiar with precision protocol

 Maintain instrument with proper QC

 Note any instrument malfunctions

 Collect sufficient data over a long period
  of time to make long term estimates
Within Run Precision, Preliminary

 Minimum of 10 replicates for each level

 Calculate mean, SD, and CV for all levels

 Use for troubleshooting

 May not reflect usual conditions
Day to Day Precision
 Extended testing – 20 days
   - 4 replicates a day – 2 test samples run in duplicate
 Abbreviated testing
   - 3 replicates x 7 runs
   - 4 replicates x 5 runs
 Variables – same as in routine testing
   - Personnel
   - Instrument and test conditions
 Problems – contact manufacturer
                Precision Example
n                       80           80             80
mean                    20.6         51.4           128.7
within run sd           1.1*         1.3            1.4*
within run CV, %        5.5*         2.6            1.1*
Total sd                1.4          1.4            2.8
Total CV, %             6.9          2.8            2.2

                *sd's similar, CV's 5 x different
   Golden Rule of
 Clinical Laboratory
“No result is better than a wrong result.”

To top