Jack Montgomery, MLS(ASCP)cm
October 2017—The challenge for all clinical laboratories is to produce the highest quality in vitro diagnostic results in the most efficient manner. Fortunately, high quality and high efficiency are not mutually exclusive, and the direct correlation between the two is well documented.1,2 As the quality of processes increases, so does process efficiency, which ultimately drives down costs.
Since the modern clinical laboratory is often perceived as a factory of test results, it is no surprise that sigma metrics—widely used in manufacturing processes as a measure of defects per million (dpm)—are often used in laboratory quality control and quality assurance processes. In this context, the sigma metric of an assay is based on the precision (CV%) and accuracy (bias%) of an assay against a predefined percent total error allowable (TEa%) specification:
The sigma metric of an assay reflects the probability of an erroneous test result. A sigma metric of six (3.4 dpm) or above is considered “world class,” and a sigma metric below three (66,807 dpm) means an assay “needs improvement.” The goal is to have as many five and six sigma processes as possible. These values can thus be used to set up assay-specific QC protocols, according to Westgard rules.3
Proposed TEa values for different assays can be obtained from a number of recognized sources,4–8 while the Clinical Laboratory Improvement Amendments specify the minimum regulatory compliance requirement in the U.S.5 The error allowed for regulated analytes mandated by CLIA can be expressed in terms of a percentage, absolute concentration, or both and can vary greatly between measurands. For example, the extent of this variation can be seen in the TEa values of alanine aminotransferase (ALT) at 20 percent, calcium at 1.0 mg/dL, cholesterol at 10 percent, and glucose at 6.0 mg/dL or 10 percent. In addition to those of CLIA, some of the commonly cited TEa sources are from Dr. Carmen Ricós and colleagues (Spanish consensus using biologic variation),6 German Guidelines for Quality (RiliBÄK),7 and Australian RCPA.8 The often significant variation in recommended TEa among the sources—as well as the lack of a unified protocol for measuring the values of bias and precision—introduces uncertainty in interpretation and generalizability of the published sigma metrics for various assays, which may confound head-to-head comparisons of assays from different manufacturers. Examples of such variations can be seen between published TEa source limits other than CLIA of the same analytes mentioned earlier: ALT (Ricós: 27.48 percent, RiliBÄK: 21 percent, RCPA: 12 percent); calcium (Ricós: 2.55 percent, RiliBÄK: 10 percent, RCPA: four percent); cholesterol (Ricós: 9.01 percent, RiliBÄK: 13 percent, RCPA: six percent); and glucose (Ricós: 6.96 percent, RiliBÄK: 15 percent, RCPA: eight percent). In addition to selecting the appropriate TEa to maintain regulatory compliance, it is also wise to consult clinicians to ascertain the quality necessary to drive appropriate medical decisions.
Sigma metric analysis is only one of the tools in the laboratory QC/QA arsenal. While each sigma calculation can define an individual assay’s performance at one point in time, the clinical laboratory is a dynamic entity with an array of moving parts, and metrics often move over time. Whatever metrics are used to guide laboratory operations, they must remain monitored, which requires constant and careful consideration.
The laboratory. Asante Rogue Regional Medical Center (RRMC), Medford, Ore., is the 378-bed flagship referral and trauma center for Asante, a growing three-hospital health system incorporating a group of medical professionals, Asante Physician Partners, serving southern Oregon and northern California. RRMC was named a Truven Health Analytics Top 100 hospital in 2014, 2015, 2016, and 2017. It was also recently deemed a Top 100 cardiovascular hospital and Top 15 health system. The core laboratory resides within RRMC and performs more than 3 million laboratory tests per year.
QC ramifications of chemistry system upgrade. In 2015, RRMC laboratory embarked on a project to upgrade its nearly 10-year-old Beckman Coulter Power Processor laboratory automation system with Beckman Coulter’s Power Express total laboratory automation. The upgrade would allow RRMC to boost the throughput of the automation line from 700 to 1,200 tubes per hour and interconnect four major laboratory disciplines.
The four-lane track, designed with T-lanes, was set up with a dynamic inlet, three centrifuges, dual decappers, dual aliquoters, a labeler, dual recappers, an outlet, and a 5K storage unit. The final instrument configuration is composed of two new modular chemistry analyzers (Beckman Coulter AU5812 primary chemistry instruments, each with two ISE units), three immunochemistry analyzers (two Beckman Coulter DxI 600s and one Siemens Centaur XP), two hematology workcells with a slide-staining module (Beckman Coulter DxHs), and one coagulation analyzer (Instrumentation Laboratory Top 700). The introduction of the new chemistry system (AU versus incumbent DxC) required evaluation of the system’s biases, coefficients of variation, decision levels (Xc), and other metrics (e.g. AMR, reference intervals) in accordance with the RRMC quality system.
The options for evaluating sigma metrics during the system-to-system transition included method validation studies or the use of external QC. The former is easier to implement but suffers from two theoretical limitations:
- Most method validation studies use a comparison of field methods rather than a comparison with a standardized reference method. Therefore, a calculation of bias in the relationship between the candidate method (y) and the comparative method (x) might not be the best possible measure.
- Imprecision experiments performed during method validation studies might not offer sufficient duration to best capture an individual assay’s CV% over time for the sigma calculation.
Therefore, to maximize the rigor of the evaluation, RRMC opted for an external QC technique, using two levels of QC material (Bio-Rad Liquichek Unassayed Chemistry Control) as the basis to determine each assay’s bias and CV%. The data were accumulated over four months, yielding more than 100 data points per assay level with a comparative peer group often exceeding 100 laboratories. CLIA-defined TEa values were used to maintain regulatory compliance.
RRMC examined 22 CLIA-regulated general chemistry analytes across both AU5812 platforms, calculating sigma values at two levels of concentration each, which then became the de facto “decision limits” (Xc) for each analyte. Each analyzer’s sigma values were plotted on discrete, normalized operational process specification (OPSpec) charts to clearly show the distribution of sigma values ranging from six to three.
One AU5812 (nicknamed Zeus and maintained by the day shift) was evaluated for 22 assays (see list of analytes evaluated, Fig. 1) at two QC levels each. The level two QC data showed 15 assays at six sigma or greater. The remaining were four assays at five sigma, two at four sigma, and one at three sigma or greater, as shown in Fig. 1. When evaluated for the level one QC, the results were similar. All TEa values used in calculation of sigma values were CLIA specifications per published sources.5
Of note, the sigma values of the assays obtained on the Zeus AU5812 analyzer were comparable to those published by the manufacturer on the basis of method validation studies and bias/variability data published in the system’s instructions for use.9
In comparison, the second AU5812 system (nicknamed Apollo and maintained by the night shift) had 30 of 44 control levels at six sigma or greater. The remaining levels were six at five sigma, three at four sigma, and five at three sigma.
In studying the performance between the two AU instruments, with particular focus on cholesterol, we found the Zeus analyzer had smaller CV percentages (2.0, 1.8) and higher sigma values (4.3, 5.1) than the Apollo’s higher CVs (2.7, 2.8) and lower sigma values (3.5, 3.5). This observation led to an increase in the frequency and monitoring of the Apollo’s sample probes, syringe replacement, and instrument maintenance schedules. For example, rather than replace individual sample and reagent syringes on an as-needed basis whenever a failure occurred (e.g. visible leaking, discoloration), new preventive measures dictated that all syringes be replaced on a universal six-month replacement cycle. CV% limits were also set for indicator assays such as cholesterol that might cause an assay’s sigma value (with negligible bias) to fall below five as a stopping point to further examine the system for causes of random vari-ation. As a result, the next month’s quality assurance program data indicated significant improvement in the Apollo analyzer’s cholesterol CV percentages (1.3, 1.4) and sigma values (7.4, 7.1), thus raising the cholesterol sigma level above six for the entire month. Continued monitoring of performance was warranted, and the improvements continued in subsequent months (Fig. 2).
An additional result of this increased monitoring and maintenance was an increase in the metrics of other analytes. See Fig. 3 for an example of accompanying improvements in the calcium assay.
Observation of the sigma metric variation between identical analyzers and assay chemistries—and its apparent dependence on routine maintenance procedures—highlights the directional utility of the sigma metric as a periodic indicator of overall system performance and a guide for the appropriate QC protocols in the clinical laboratory. Assays with sigma levels of three and four should be examined for ways to improve, perhaps in terms of more maintenance, calibrations, or personnel training, or all three. Some assays, owing to the strict CLIA-mandated TEa, may be difficult to raise above four sigma.
For example, at an Xc of 150 mmol/L, a sodium level must have no bias and a CV of 0.67 percent (TEa 4 mmol/L) or better to achieve four sigma or higher. Simplistic comparison of absolute values of sigma metrics for a specific analyte between different analyzers and assay chemistries without careful consideration of the confounding factors—such as the selection of TEa, study protocol, comparative method, reagent lot-to-lot variability, calibrator lot-to-lot variability, and maintenance schedule—may lead to erroneous conclusions about assay performance.
The periodic nature of common QC protocols creates the constant risk of within-run system issues not being detected until the next scheduled QC event. For a high-volume laboratory like RRMC, this means potential reruns performed on many hours of test results, as well as incremental risk of the erroneous clinical test results propagating to the point of affecting patient care. Moving averages (or, better yet perhaps, moving medians) might be best used to supplement traditional QC in critical assays where they can detect shifts and trends in real time. One limitation of moving averages is that they are specific to the patient population undergoing testing. For example, electrolytes in outpatients analyzed in the afternoon might be quite different than electrolytes measured in ICU or critical care inpatients in the morning, though today’s algorithms can be configured to account for these variables. Examples that demonstrate the effect of using moving averages or medians have been published in multiple articles.10,11
Given the potential negative impact on patient care, the quality of laboratory test data must remain of paramount concern at all times. Experience suggests the need for holistic approaches to laboratory quality rather than relying on a single parameter such as sigma metrics. Assays and instrument systems should be assessed and monitored on multiple measures, including but not limited to precision, accuracy, standardization/traceability, reagent and calibrator stability, ease of use (and resistance to operating errors), and analytical measuring range. Other important metrics are lack of interferences, commonality of reagents and results across platforms, optimized workflow, daily setup time/FTE needs, preventive maintenance time/FTE needs, system uptime metrics, and scalability.
The real-world analysis of Beckman Coulter AU5812 analyzers in a high-volume central laboratory revealed excellent sigma metric performance, consistent with the data provided by the manufacturer, through the method validation studies. Detected dependence of the sigma metrics on routine maintenance procedures highlights the dynamic nature of the metric with both bias and precision changing over time, necessitating periodic monitoring and incremental QC techniques such as moving averages. While sigma metrics can help guide analyte-specific QC rules, as well as the number and frequency of QC observations used each day, a holistic approach to QC protocols is needed. This includes the skill and training of operators, instrument performance directly related to maintenance schedules and procedures as described in the preceding results section, reagent and calibrator quality and stability, and preanalytic sample handling. A total quality program is a multifaceted approach requiring the training, participation, and cooperation of the entire laboratory staff.
- Clinical and Laboratory Standards Institute. QMS20-R: Understanding the Cost of Quality in the Laboratory; A Report. May 2014.
- Clinical and Laboratory Standards Institute. C24: Statistical Quality Control for Quantitative Measurement Procedures: Principles and Definitions, 4th ed.; 2016.
- Westgard JO. Six Sigma Quality Design and Control, 2nd ed. Madison, Wis.: Westgard Quality Corp.; 2006.
- Clinical and Laboratory Standards Institute. EP21: Evaluation of Total Analytical Error for Quantitative Medical Laboratory Measurement Procedures, 2nd ed.; 2016.
- U.S. Code of Federal Regulations 42 CFR §493.909–942 CFR §493.959.
- Ricós C, Alvarez V, Cava F, et al. Current databases on biological variation: pros, cons and progress. Scand J Clin Lab Invest. 1999;59(7):491–500.
- Richtlinie der Bundesärztekammer zur Qualitätssicherung laboratoriumsmedizinischer Untersuchungen. Gemäss dem Beschluss des Vorstands der Bundesärztekammer vom 1.04.2014 und 20.06.2014. Deutsche Ärzteblatt. 2014;111(38):A1583–A1618.
- Allowable limits of performance. Royal College of Pathologists of Australasia. www.rcpaqap.com.au/docs/2014/chempath/ALP.pdf. Accessed Dec. 13, 2016.
- Shih J. Using six sigma metrics to monitor the quality of results. Publication BR-51862. Beckman Coulter Inc.; 2016.
- Wilson A, Roberts WL, Pavlov I, Fontenot J, Jackson B. Patient result median monitoring for clinical laboratory quality control. Clin Chem Acta. 2011;412(15–16):1441–1446.
- Van Rossum HH, Kemperman H. Optimization and validation of moving average quality control procedures using bias detection curves and moving average validation charts. Clin Chem Lab Med. 2017;55(2):218–224.
Jack Montgomery is the chemistry technical specialist, Asante Rogue Regional Medical Center, Medford, Ore.