VIEWS: 34 PAGES: 118 POSTED ON: 4/22/2012 Public Domain
Concepts in Quality Control Data Management Thomasine Newby BS, MT (ASCP) Bio-Rad Laboratories Quality System Specialist Field Marketing Objectives At the conclusion of this program, participant should be able to: Understand how to plan statistical Quality Control Learn how to derive QC Statistics Evaluate QC using Westgard rules and graphs Discuss the types of errors found when evaluating QC and the resolution of these errors. Develop a method validation plan Reference Material Clinical and Laboratory Standards Institute Statistical Quality Control for Quantitative Measurement Procedures: Principles and Definitions; Approved Guideline – Third Edition (C24-S3, Vol. 26 No. 25) Method Comparison and Bias Estimation Using Patient Samples; Approved Guideline – Second Edition (EP9- A2, Vol 22 No. 19) Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach; Approved Guideline (EP6-A, Vol 23 No. 16) Quality Laboratory Quality means achieving positive patient outcomes through control of pre-analytical, analytical, and post-analytical error. Pre-Analytical Errors Sample taken from wrong patient Wrong sample analyzed Incorrect test ordered/performed Patient sample not properly preserved Patient sample not properly collected – IV arm Incorrect or incomplete patient instruction and preparation Post-Analytical Errors Wrong result reported or charted – transcription errors Computer error – LIS integrity Incorrect interpretation of test results provided to clinician Incorrect patient normal ranges Results reported from an instrument not used to establish the normal range Analytical Errors Instrument malfunction Lack of maintenance Inadequate environmental control Inadequate electrical supply Inadequate water supply Inadequate operator training Incorrect calibration Use of compromised reagents and/or calibrators Purpose of Statistical Quality Control Statistical quality control procedures are intended to monitor the analytical performance of a measurement procedure and alert analysts to problems that might limit the usefulness of a test result for its intended medical purpose. Statistical Quality Control Performed by analyzing stable specimens (external QC) and comparing the observed values to the distributions expected under stable conditions. Involves calculating means and SDs for the QC to set appropriate limits and identifying situations that represent unstable operation. Statistical Quality Control Derived statistics are used to make judgments concerning: Quality of analytical results Whether system correction is necessary Whether patient tests are accepted or rejected Estimating performance parameters that can be compared to analytical and medical goals Planning a Statistical QC Procedure Planning a Statistical QC Procedure Define the quality specifications for the test. Select control materials. Determine method performance Identify QC strategies Predict QC performance Set goals for QC performance Select appropriate QC rules Define the Quality Specifications Total allowable error – the magnitude of measurement error that if exceeded would cause a test result to be of unacceptable quality (imprecision and bias). Medical important changes in test results (imprecision, bias, and pre-analytical variables). Biological variation (imprecision and bias) Selecting Control Materials Menu Shelf life/Open-vial Stability Vial to vial variability Medical relevant levels Challenge range of instrument Matrix/preservatives QC material different from calibrator material Number of levels and concentration – sufficient to determine proper method performance over the measuring range of interest. Determine Method Performance Performance characteristics – bias and imprecision Imprecision – estimated by repeated measurements of QC Bias – comparison with proficiency testing or interlaboratory QC program Identify QC Strategies What control materials used? How many control samples analyzed? Where the QC are placed on a run? What QC rules are applied? When the QC rules are evaluated? Predict QC Performance Direct indicator of QC performance – the expected number of unacceptable patient test results reported when the process is out of control - Depends on type and magnitude of out-of-control error - When the error occurs - How long the error lasts All depends on QC strategy and the probability that the QC rules and number of control measurements will detect the error. Frequency Of Occurrence Set Goals for QC Performance Maximum allowable number of unaccepted results due to out-of-control error Maximum allowable probability of reported unacceptable results Minimum acceptable probability of detecting an error Maximum acceptable probability of false rejections. Overall goal for QC performance – to maximize the detection of an out-of-control condition for a procedure while minimizing the probability of false QC alerts. Select Appropriate QC Rules Various combinations of QC rules and parameters can be selected to satisfy the QC performance goals. QC Applications QC Strategy Define QC material Decide the number of measurements and the location of QC Set the control limits Decide on the QC rules for acceptable run Outline the response to the data acceptance decision (out-of control) Frequency and Location Frequency of QC - QC must be analyzed once during a run - Run is defined as interval that the method is considered stable Location of QC - QC results evaluated before reported patients - Placement immediately after calibration may give false low imprecision and shift during run Set Control Limits Establishing Data Parameters Establishing Data Parameters Determination of valid mean and standard deviation values are crucial to successful data rejection or acceptance by error detection schemes. Establishing Data Parameters Parallel Testing vs. Product Inserts CLIA Regulations “The stated values of an assayed control material may be used as the target values provided the stated values correspond to the methodology and instrumentation employed by the laboratory and are verified by the laboratory.” CLIA 88, Final Rule Federal Register, February 2, 1992 Standard 493.1218 CLSI Recommendations “If assayed controls are used, the values stated on the assay sheets provided by the manufacturer should be used as guides in setting the initial control limits for testing new materials. Actual values for the mean and standard deviation must be established by serial testing in the laboratory. The observed mean should fall within the range published by the manufacturer.” Clinical and Laboratory Standards Institute Volume 26 Number 25 2006 QC Statistics The first step in defining performance limits ► Mean. . . . . . . . . . . . . . . . . . . x ► Standard Deviation . . . . . . . .s ► Coefficient of variation . . . . . cv Mean Arithmetic average of a set of data points Sum of values divided by the number of values a + b + c ……+ n n Standard Deviation A measure of variability of the data points The degree of dispersion around the mean SD = the positive square root of the population variance Coefficient of Variation, CV Measure of variability – random or systematic Expressed as a percent CV = SD/Mean X 100 Calculating A Mean/Standard Deviation CLSI Recommendations Standard Provisional Things to Consider When Calculating QC Statistics Do I have a sufficient number of data points for the calculation? Have I used a sufficient number of analytical runs to account for random variables? - Multiple calibrations - Multiple reagent lots - Maintenance - Technologists - Environmental conditions Things to Consider When Calculating QC Statistics Did I follow “normal” procedure each analytical run? Have I appropriately excluded outliers? Test for Outliers Suspect low value – arrange values from high to low value Suspect high value – arrange values from low to high Refer to table – “Significance limits for testing extreme values of a sample” Column – use 0.05 (5% of values may be outside 2sd) Row – use the number of data points Calculate value – compare to table Calculated value > table value – Reject Calculated value < table value – Accept Table Value Calculating A Mean/Standard Deviation NCCLS Recommendations Standard Provisional 95% Confidence Interval Estimation of a 2s Range Estimation of a 2s Range for a small sample by calculating a 95% confidence interval. SE=s/ n SE=Standard Error of the Mean n = 10 n=5 x = 398.2 x = 401.2 s = 5.41 s = 4.97 SE = 1.71 SE = 2.22 LL = 398.2 – (1.71)(2.262) LL = 401.2 – (2.22)(2.776) UL = 398.2 + (1.71)(2.262) LL = 401.2 + (2.22)(2.776) 2s Range (est.) = 394.3-402.1 2s Range (est.) = 395.0-407.4 Standard Deviation Index, SDI CALCULATION: SDI= [Lab Mean – Peer Group Mean] Peer Group’s 1 standard deviation Use to assess bias compared to a peer group TARGET SDI IS 0.0, INDICATING THAT THE LAB’S MEAN VALUE IS THE SAME AS THE PEER’S. Coefficient of Variation Ratio, CVR Calculation Lab’s Monthly CV Peer Monthly CV IDEALLY, CVR ≤ 1.0, SINCE YOUR VALUES ARE FROM A SINGLE LAB, WHILE THE PEER CV IS FROM SEVERAL. IF CVR = 1.5 to 2.0, THE LAB IS 50-100% LESS PRECISE THAN IT’S PEER GROUP, USUALLY REQUIRING INVESTIGATION. Frequency Of Occurrence Set Quality Control Rules Westgard Rules Westgard Rules Error Detection - Random - Systematic False Rejection Multi-rule improves error detection with a low probability of false rejection Minimum of 2 controls per run Not all rules are necessary for all tests Types of Analytical Errors Random - Precision - Inherent to the test system - Always present - Minimize any increase Systematic - Accuracy - Shifts - Trends Westgard Rules Error Detection Random Error – imprecision •1 2s, 1 3s, R4s Systematic Error – bias •1 3s, 2 2s, 4 1s, 10x Troubleshooting and Westgard Rules Multirule Flowchart- When a rule is violated, Run Control stop and investigate. N N 12S In Control, Accept Run o x o Yes N N N N 13 o 22 o R4S o 41S o 10 S S X Yes Yes Yes Yes Yes Investigate, Reject Review Current and Previous Runs Out of Control Situations Troubleshooting Troubleshooting Random Error Was there a deviation from procedure? Was the correct equipment used? Is there potential for random malfunction in the test system? Is the test sensitive to technique? Troubleshooting Random Error Is this error a manifestation of pre-analytical error? Is the test system vulnerable to electrical power fluctuations? Is this truly random error or the initial stages of systematic error? Sources of Random Error Power supply Double pipetting of control sample Misplacement of control sample within the run Air bubbles in water supply Random air bubbles in reagent or sample pipette system Sources of Systematic Error: Shifts & Trends Improper alignment of sample or reagent pipettes Drift or shift in incubator chamber temperature Inappropriate temperature/humidity levels in the testing area Change of reagent or calibrator lot Deterioration of reagent while in use, storage or shipment Sources of Systematic Error: Shifts & Trends Deterioration of the calibrator while in use, storage, or shipment Deterioration of the control product while in use, storage or shipment Incorrect handling of the control product (e.g., freezing when not recommended) Inappropriate storage of control products in frost-free freezers Sources of Systematic Error: Shifts & Trends Dirty filter wheel Failing light source Use of non-reagent grade water in the test system Recent calibration Change in test operator Troubleshooting the Control Product Is the product being aliquoted and frozen? Is the product being properly stored before and after reconstitution? Is the reagent being reconstituted with volumetric pipettes? Is reagent-grade water used for reconstitution? Troubleshooting With Graphics Troubleshooting Troubleshooting Using Graphical Techniques Levey-Jennings Charts Frequency Histograms Youden Plots Action Plan for Rule Violations Review last 10 data points (both levels) Do not re-run the control Consult troubleshooting guides Consult your lab’s procedure manual Evaluate the test system Check maintenance records Correct any problems Method Validation Method Validation Comparing test method with reference method Comparing new test method to current method. Overview of Comparison Evaluating an analytical method requires: Time for operators to become familiar with the device operation and procedures Time for operators to become familiar with the evaluation protocol Both new and current methods must be in proper control over evaluation period Enough data to ensure representative results from both test and comparative methods Device Familiarization Period Operators of both test and new methods - General operation: set-up, operation, and troubleshooting - Maintenance procedures - Methods of sample preparation - Calibration and quality control Samples Samples – collected according to the lab procedures, properly stored, and no longer than analytes are stable. Range of measurement – evaluate the test method over the clinically significant range - Concentrations should cover analytical measurement range – the concentration interval claimed by the manufacturer to provide acceptable performance Samples Number of samples – at least 40 samples meeting the criteria Duplicate measurements – duplicates can be analyzed on both new and current methods Sample Sequence – assign positions and reverse on duplicate runs (minimize carryover and drift) Time and Duration Analysis on both methods should occur within a time span consistent with analyte stability. If possible, use samples the day of draw. If stored, use proper storage and store samples for both methods the same. Error and QC Record any errors (out-of-control) or human errors – do not include data in the analysis Quality Control – follow procedures for QC and repeat any run that is out-of-control. Final Assumptions Current method (old method) is assumed to be precise and accurate, free from interference. Variation is due to the new method. Evaluating Data Scatter Plots – Bias Linear Regression Method Comparison – Bias Plot Method Comparison – Bias Plot Method Comparison – Bias Plot Linear Regression X Axis Old, Reference, “A” Y Axis New, Comparative, “B” New, Comparative Old, Reference Linear Regression Constant Error Affects accuracy (bias) throughout range, seen as change in Y-Intercept 1000 ACTUAL VALUES Actual results 750 Line of identity Slope, m = 1.00 500 }Constant Error 250 0 0 250 500 750 1000 EXPECTED VALUES Linear Regression Proportional Error Varies the Y value as the concentration of X increases. The results are seen as a change in the slope from 1.00 1000 Actual results ACTUAL VALUES 750 Proportional Line of identity 500 250 } Error Slope, m = 1.00 0 0 250 500 750 1000 EXPECTED VALUES Linear Regression Y = mX + b Error Y = 1.00(x) + 0 None - Then Y = X, identical Y = 1.05(x) + 0 Proportional - Then Y is 5% greater than X Y = 1.00(x) + 5 Constant - Then Y = X + 5 r = correlation between methods Linear Regression Test Yields: Line equation Y = mX + b r = Correlation N, number of coefficient tested pairs Method Validation Linearity Linearity Device familiarization period Duration of experiment – short a time interval as possible (preferably one day) Specimen requirements – enough specimen to prepare dilutions and test each of 5 – 11 concentrations in duplicate Samples Matrix – similar to specimens used in testing Patient pools – ideal matrix with analyte near the upper reportable range Patient pools – supplemented with analyte Commercial – controls, calibrators, linearity material Patient Pooled Samples Low concentration pool – near and within the lower limit High concentration pool – the highest concentration tested Intermediate concentration pools – made by dilutions in constant relationship to each other and the low and high Linearity Purposes To establish linear range – 9 to 11 levels with 2 – 4 replicates at each level To validate “in-house” or modified methods – 7 to 9 levels with 2 – 3 replicates at each level To confirm linear range is valid – 5 to 7 levels with 2 replicates Testing Analytical sequence – random Analyte range - Concentration levels should be equal to the minimum and maximum values in the performance claims - When establishing linear range – the levels should include a range 20% to 30% wider than anticipated measuring range (plan to eliminate nonlinear points to establish the widest possible range of acceptable linear response) Linearity Data Exam Plot results - Y = testing results - X = sample concentrations or relative concentrations Linear pattern – segment slopes will be approximately equal Nonlinear pattern – increasing or decreasing trend Outlier – any result far from other Y values - eliminate Linearity Graphs Instrument Limitations Allow all values to be reported but eliminate outliers 1000 Line of identity ACTUAL VALUES S e Replicate r1 750 i Replicate e s2 1 500 S Outlier e r i 250 e s 2 S 0 e 0 r 250 500 750 1000 i e s EXPECTED VALUES 3 Linearity Slope Intercept Linearity – Poor Precision & Bias Slope Intercept Method Validation Precision Precision Testing Familiar with instrument operation Familiar with precision protocol Maintain instrument with proper QC Note any instrument malfunctions Collect sufficient data over a long period of time to make long term estimates Precision Within Run Precision, Preliminary Minimum of 10 replicates for each level Calculate mean, SD, and CV for all levels Use for troubleshooting May not reflect usual conditions Precision Day to Day Precision Extended testing – 20 days - 4 replicates a day – 2 test samples run in duplicate Abbreviated testing - 3 replicates x 7 runs - 4 replicates x 5 runs Variables – same as in routine testing - Personnel - Instrument and test conditions Problems – contact manufacturer Precision Example n 80 80 80 mean 20.6 51.4 128.7 within run sd 1.1* 1.3 1.4* within run CV, % 5.5* 2.6 1.1* Total sd 1.4 1.4 2.8 Total CV, % 6.9 2.8 2.2 *sd's similar, CV's 5 x different Golden Rule of Clinical Laboratory Science “No result is better than a wrong result.” Questions?