November 2018—In The Music Man, the four members of the local school board, notorious for their squabbling, are assigned to investigate the credentials of “professor” Harold Hill, a charming con man who has come to town to sell band uniforms and instruments. Instead, through clever pitches and flattery, Hill swiftly diverts the four to singing in harmony, they become an ad hoc barbershop quartet, and their bickering comes to an end.
Such sudden transformations happen in musicals rather than real life. But the hope of bringing a similar end to disagreement is what inspires laboratory medicine’s long-term, complex quest for another kind of harmonization: that of clinical laboratory results. “The fact that methods are not harmonized is probably one of the most critical problems facing the field of laboratory medicine today,” says Lorin Bachmann, PhD, co-director of the clinical laboratory and of point-of-care testing at Virginia Commonwealth University.
It’s an under-appreciated problem, says colleague Greg Miller, PhD, co-director of clinical chemistry and director of pathology information systems at VCU. “A large number of tests are not standardized and yet we have clinical guidelines that say, ‘If the value is greater than x, then you do this or that.’ This situation can lead to errors in medical judgment.”
Through harmonization, laboratories performing a test achieve equivalent results and the same interpretation irrespective of the test method used and when or where the measurement is made. “Harmonization simply means that labs’ methods agree with each other even though there may not be a higher order reference method to relate the methods to,” says Anthony A. Killeen, MD, PhD, professor of laboratory medicine and pathology at the University of Minnesota. Avoiding misinterpretation of results, wrong treatments, and adverse patient outcomes is one of the main purposes of harmonization.
A standardized method is the ideal. Standardization is a stricter process to achieve equivalent results by having calibration traceable to a reference system component; it requires a reference method and reference material, which labs are unlikely to have for most assays. But harmonization can essentially finesse the problem of a lack of standardization.
“If all of the methods are providing the same results, the question of whether they are reporting the ‘true’ result—a result equivalent to that produced by a reference method—may be less important,” Dr. Bachmann explains. “If the clinical decision thresholds associated with patient outcomes have been established for several of those representative methods, results from different methods can be used to guide patient care. If all methods for the same test are harmonized, results from different laboratories, using different manufacturers’ methods, can be used in context of the clinical decision thresholds and patients will be treated appropriately.”
This year, the harmonization campaign is getting a boost from the laboratory community and the offices of Congress. In response to a request from the AACC, the International Consortium for Harmonization of Clinical Laboratory Results (ICHCLR), a coalition of laboratory groups (of which the CAP is a council member), has set priorities for methods that need harmonization. And for the first time, the AACC and other laboratory organizations including the CAP were successful in lobbying Congress to earmark funding for the Centers for Disease Control and Prevention in support of harmonization.
Aside from providing $2 million in support in the 2018 and 2019 budgets, Congress’ decision to allocate more frontline funding for the CDC to expand its capabilities in the area of harmonization “gives recognition to the fact that this is an important problem in clinical medicine today,” Dr. Killeen says. “And obviously we hope it will continue.”
“It’s a major goal in medicine to get harmonized results for a patient sample regardless of which laboratory it’s sent to, so there is little to no bias between different manufacturers’ methods in terms of the results. The goal has been accomplished for some analytes. But there’s a long, long way to go in this field,” he cautions.
Cholesterol, HbA1c, and creatinine, having been standardized as well as harmonized, have demonstrated how harmonization leads to better patient care. “Because of the improvement in the creatinine methods, we now have a much better understanding of an individual patient’s renal disease status,” Dr. Killeen says. “Improvement in HbA1c has given us a much better understanding of patients’ diabetes control and more confidence and certainty in making adjustments to insulin medication.” But these benefits have been realized thanks to sustained efforts over many years, he adds. “Creatinine took probably a good seven years from the realization that we needed harmonization to the point where we are today.”
The lack of harmonization of most other laboratory results is a problem worldwide, he notes. “In laboratory medicine, a handful of manufacturers provide the bulk of equipment to labs all over the world. They recognize the problem with harmonization, and this is a problem they need to solve. An end-user lab like mine purchases FDA-cleared reagents and methods from the companies. We have very little possibility of modifying methods.”
Nothing has happened on a grand scale so far, but Dr. Killeen sees harmonization gaining momentum. “The sweet spot happens when an authoritative body like the National Institutes of Health gets involved, alongside manufacturers and professional organizations like the CAP.” When those three groups come together, you can drive progress, he says. “That’s what happened several years ago when the NIH founded the National Kidney Disease Education Program,” which promoted the standardization of creatinine methods. “Much of the NKDEP’s initial work was driven by CAP data from proficiency testing, which confirmed there were significant differences among different manufacturers’ creatinine methods.”
For now, the ICHCLR’s list of priorities may give manufacturers and potentially the CDC guidance in deciding where to focus their efforts. The top eight U.S. tests that should be harmonized first, the ICHCLR has said in response to an inquiry from the AACC, are B-type natriuretic peptide, prostate specific antigen, human chorionic gonadotropin, D-dimer, CA 125 to monitor ovarian cancer, CA 15-3 to monitor breast cancer treatment, CA 19-9 to monitor pancreatic cancer treatment, and carcinoembrionic antigen to indicate tumor burden in monitoring cancer treatment.
The ICHCLR chose those tests as the most pressing and feasible candidates for harmonization. The AACC has asked other laboratory industry stakeholders to identify tests they believe should be addressed in the near term.
The authors of a recent study on D-dimer concordance illustrate the dangers of following a clinical decision rule based on results of tests that are not harmonized (Rodger MA, et al. Thromb Res. 2018;169:82–86).
Used to identify low-risk women who can safely discontinue anticoagulants after completing treatment, the HERDOO2 rule was derived and validated using the Vidas D-dimer assay, and researchers at Ottawa Hospital Research Institute wanted to determine whether other commercial D-dimer assays used at a corresponding cut-point would categorize patients at high concordance with the Vidas D-dimer assay.
The study found that at the Vidas D-dimer cut-point of 250 µg/L, the concordance of all other D-dimer assays tested was poor. The authors concluded: “[T]he HemosIL, Innovance, Liatest, and Tina-quant DD assays should not be used in the ‘HERDOO2’ rule due to poor concordance with the VIDAS DD assay and unacceptable misclassification of women at high and low risk of recurrent venous thrombosis.”
Dr. Killeen’s interest in harmonization stems from his involvement in CAP creatinine proficiency testing 15 years ago. As a member then of the CAP Chemistry Resource Committee, he participated in the CAP Survey that sent a fresh frozen sample to laboratories using creatinine methods. Using fresh frozen samples allowed valid comparisons of results—laying the groundwork for the NKDEP standardization of creatinine.
“Generally, the PT material exhibits matrix effects. So if there are differences between results, one doesn’t know if they’re getting two different results in patient samples or if the differences are simply a reflection of the artificiality of the PT material. In 2003, we were able to show there are significant differences between different manufacturers’ test results.”
Understanding what actually is the analyte is critical to harmonization, Dr. Killeen points out. “Especially for proteins, they can exist in different forms within the body. Which one is being analyzed?” For simple molecules like creatinine, obtaining scientific agreement on the primary target for measurement is no problem. “But for larger proteins, it’s not always clear what should be measured. Different manufacturers may be detecting slightly different variants or may have different degrees of ability to pick up variant forms of proteins.”
The CAP accuracy-based Surveys use native human serum, which is from human donors and not adulterated like the artificial material, Dr. Killeen says. At least eight such Surveys today use material that is free of matrix effects for lipids, HbA1c, creatinine, vitamin D, testosterone, estradiol, thyroid hormones, urine albumin, creatinine, and other analytes. “That’s how we can assess harmonization in the field.”
The trouble is, “we can only collect samples from people with relatively normal levels of most of these lab analytes. It’s very tough to get disease-level material. You have to collect it from people with pathological conditions, and that’s a tall order for obvious reasons.” But, he adds, the CAP is interested in harmonization and taking steps in the right direction.In a session at the 2018 AACC meeting, speakers on the panel on test harmonization found, through an audience response tool, that understanding of harmonization’s significance is not that deep even within the laboratory community. When Dr. Bachmann, one of the speakers, asked a few multiple-choice questions during her presentation, “The audience showed a general familiarity with the concepts of harmonization and standardization. But I don’t think there was a good appreciation of the extent to which methods disagree and how that affects the clinical management of patients.”
In her experience, Dr. Bachmann says, endocrinologists as a specialty are particularly savvy about the types of methods used to obtain their test results. But, in general, clinicians have a tendency to be unaware that methods for many analytes are not harmonized. “When a laboratory reports a test result, clinicians typically believe the result is transferable among different manufacturers’ methods or can be interpreted according to published clinical decision thresholds. In addition, there is often no consideration of which method was used to establish the clinical decision thresholds.” Moreover, those methods may no longer be in the marketplace. “Some of those thresholds may have been established 10 or 20 years ago—or more. The methods used to establish the thresholds are now obsolete and different technologies have replaced them. That’s a serious problem because those clinical decision thresholds may no longer be appropriate when testing is performed using newer technologies.”
Dr. Bachmann was more surprised when she accompanied a group from the AACC to Capitol Hill to discuss the need for federal funding for harmonization and standardization. “That experience had a significant impact on my own recognition of the general lack of awareness about how lack of harmonization can affect patient care. Legislators and their staff were surprised when we explained that a patient could receive different results for a test depending on the specific method used by the laboratory where testing was performed.”
The funding granted ($2 million for two years) was far less than the $9.2 million the laboratory groups requested. But Dr. Bachmann applauds the increased attention to harmonization and standardization Congress has shown. “It’s very difficult to obtain funding for standardization efforts. There’s no designated party that is taking on the expense.”
For a manufacturer, re-standardizing a lab test now requires resubmission through the FDA, a lengthy and expensive process. “Even if we have a standardization program for a particular analyte such as TSH or urine albumin, and the manufacturers want to reassign their calibrator values so that they are obtaining results through the traceability chain so patient results are standardized, all those manufacturers may have to resubmit significant amounts of analytical performance data to regulatory agencies before standardized methods are approved. That cost involved for significant additional regulatory submissions is a barrier to accomplishing harmonization and standardization,” Dr. Bachmann says.
Given the modest funding available, only some of the most critical tests will be led through harmonization in the short term, leaving a raft of tests out there that are not harmonized. That’s why Dr. Bachmann thinks the interaction of laboratorians with Congress to explain why harmonization and standardization should receive federal funding needs to be ongoing, and the ICHCLR, the CDC, and professional practice societies need to continue to exert their combined influence on the diagnostics industry to facilitate standardization and harmonization efforts.
“It’s also up to the laboratory community to inform the physician groups about the need for standardization on a much larger scale. That would help raise the profile of this problem and increase the pressure on Congress or other funding agencies to make harmonization a greater priority.”What focused laboratory medicine on the need to develop more reference system components for standardization and harmonization of laboratory results was the implementation of the in vitro diagnostic medical devices directive by the European Council in 1998, says Gary L. Myers, PhD, a consultant and chair of the Joint Committee for Traceability in Laboratory Medicine (JCTLM) and formerly chief of the CDC’s Clinical Chemistry Branch where the primary work was in lab improvement and standardization. “An essential element of the EC directive was that IVDs be traceable to higher-order reference methods and reference materials where available.”
Most IVD companies, although they may be U.S. or European based, serve laboratory medicine worldwide and have the responsibility to meet whatever regulatory requirements exist where they market their tests. In response to the European IVD directive, the JCTLM was formed in 2002 by the International Bureau of Weights and Measures in France, the International Federation of Clinical Chemistry and Laboratory Medicine, and the International Laboratory Accreditation Cooperation. The goal of the JCTLM, says Dr. Myers, is to provide a worldwide platform to promote and give guidance on internationally recognized and accepted equivalence of measurements in laboratory medicine and traceability to appropriate measurement standards.
To support IVD manufacturers in meeting the IVD directive essential requirement for traceability to higher order standards, the JCTLM provides an online database (www.bipm.org/jctlm) of reference methods and materials as well as labs that provide reference measurement services. “These are all evaluated in accordance with ISO standards,” Dr. Myers says, “and the ones found acceptable are made part of this database” that IVD companies can consult to find reference methods and materials to use to trace their methods, as many countries now require. ISO 17511, currently under revision, describes principles for metrological traceability of values assigned to calibrators and controls, which form the basis for approaches establishing traceability to higher order reference systems.
The ICHCLR’s full list of priority tests was compiled from a number of different sources of lab tests ordered by physicians, says Dr. Myers, who also serves as chair of the ICHCLR council. “Our harmonization oversight group, made up of experts in laboratory medicine, has been systematically looking at the clinical data, performance data, and need for use in managing and diagnosing patients’ conditions to prioritize which ones need to be harmonized.” The organization’s website (www.harmonization.net) shows which tests have been designated high, low, and medium priority.
Although industry perspectives are represented, “we’re not looking necessarily at harmonization from an industry standpoint but from a medical need standpoint—how the tests are used to diagnose patient conditions and which are higher priority.” Historically, manufacturers have lacked the resources to do all of their harmonization internally, so industry has played a key role in selecting the analytes targeted for harmonization by looking at the competition for resources internally and the external forces pushing harmonization.
“For example, when the National Cholesterol Education Program came along and said, ‘Know your number,’ the laboratory industry said, ‘We can’t measure cholesterol reliably enough for “know your number” to make sense,’” Dr. Myers says. “So manufacturers lined up with the lab community to improve the methods and develop and use standards in the calibration process.” For other methods, the usual approach is to establish a working group or committee to work on developing the reference system components for a particular analyte.
Currently 26 analytes have complete reference systems (a reference method, material, and certified laboratory that can provide measurement services) in the JCTLM database. A total of 293 reference materials, including a combination of pure material and matrix-based materials, are listed in the JCTLM database as well as 184 reference methods; 161 reference measurement services are listed as being delivered by 17 certified reference laboratories.
Harmonization of all lab tests will be no small task, Dr. Myers says. “There’s a group in the Netherlands that estimates there are several thousand lab tests commonly measured in the lab, maybe not routinely. But if you look at it from that standpoint, there’s still a lot of work to do. The 80 or so analytes that have been harmonized or standardized are the tip of the iceberg. The majority of tests that haven’t been touched, because of technical difficulty or another reason, is the mass of the iceberg below the surface of the water.”
In contrast with the EU, the FDA does not require traceability of an in vitro device to a reference system as part of its approval process. And that is a concern, Dr. Myers says. Since long before the EU adopted its standard, “the FDA has required that a method be equivalent to what’s considered a predicate device that has already been through the approval process. They can’t require, as the EU IVD regulations do now, that IVD companies provide documentation indicating their method is standardized or traceable to a higher order standard.”
However, the FDA does consult the JCTLM database for a more limited purpose. “If an IVD company says in its product literature that it has established traceability to a reference method and reference material, the FDA will look to make sure that what the company is stating it has established traceability to is the appropriate standard of a reference material and reference method.”
Dr. Myers would like to see the EU requirement of calibration traceability to a higher order reference system become adopted worldwide by regulators, including the United States. “If we’re going to have standardization and have IVD companies work for more equivalency across different platforms that measure the same analyte, it’s going to have to be done either through traceability to a standard that’s out there or a harmonization process that may use patient samples to do the same thing.”
“But there has to be a process. It doesn’t seem reasonable that a technology to measure something that’s developed in 2017 may be okay as long as it’s traceable to something that was developed in the 1960s or 1970s, especially where there may have been no standards available back when that initial technology was evaluated by the FDA.”
Under the EC directive, an estimated 80 percent of devices will have to go through certification. Only 20 percent, those that fall into a category A in terms of low risk, will be able to be self-declared by the IVD manufacturers themselves, Dr. Myers notes. Self-declaration is practiced in the U.S. now when an IVD company intends to seek a CE mark allowing its products to be sold in Europe.
What manufacturers are doing with harmonization and standardization is changing their calibration, which is going to change the numeric value of results. “But they’re not making a change in the method itself,” Dr. Myers notes. “You’re not changing the precision or reaction conditions or anything like that, so there should be a more streamlined process to allow these IVD companies to submit their work and get approval based on how they changed their calibration to be consistent with particular standards.”
“If an IVD company has to make modifications to a method—for example, to make it more selective for the particular test they are trying to measure—obviously they will need to follow the FDA guidelines that are in place for a major change in the method.”
“So we’re saying that if they recalibrate and just move the curve up, say, five percent, and nothing else changes other than the numeric value—the methods still measure cholesterol, for example, the same way we measured it before—they’re just recalibrating to a new standard that should not require major submissions and new studies that are very costly.”The laboratory medicine community does recognize and has been working on standardization schemes for quite a while, Dr. Miller of VCU notes. “But it’s very slow going, and it hasn’t been until the last 20 years that its importance has been appreciated. Physicians who are community and family practitioners probably make the assumption that the lab tests are pretty equivalent.”
“That’s an educational challenge for pathology,” he says, adding that the CAP’s resource committees are well aware of these problems. When he asked the audience during his own AACC presentation on harmonization whether they have received a call from a physician asking why one result was different from another when the answer was because it was from a different method, nearly 100 percent answered yes.
One of the key things identified as missing in the 2010 conference that led to the creation of the ICHCLR, Dr. Miller says, was a process to harmonize analytes when no reference method or certified reference material exists. “The classical way to approach harmonization and standardization is to develop a reference method and reference material, and the IFCC is the main sponsor of such projects. But for many analytes, those are not available and are not likely to become available due to technical constraints.”
He estimates that of the roughly 1,500 analytes in a typical clinical laboratory test catalog, only about 100 have had reference systems developed to support standardization. “That leaves a very large number of tests, and there needs to be something developed to help them meet the needs of standardization.” Among those 100 analytes, not all calibration hierarchies use reference materials that are commutable with patient samples, which means they are not suitable for use as calibrators. “Before 2010 there was not general recognition and agreement that commutability with patient samples should be required for reference materials,” Dr. Miller says.
To fill the gap, a new process built around a harmonization protocol has been evolving. The IFCC committee working on standardization of thyroid tests has developed a harmonization protocol for TSH, based on using a panel of patient samples as calibrator materials, Dr. Miller says. The ISO Technical Committee 212 for laboratory medicine is currently developing an international standard for harmonization protocols, expected to be published in 2019 or 2020.
He agrees that the laboratory medicine community has moved beyond striving for perfection where perfection is not necessary, and that has opened the door to more progress in harmonization. As a sign: “We now agree that the new harmonization protocol is an acceptable approach. Ten years ago people would have said, ‘You can’t use that—it violates scientific principles.’”
Now it’s better understood that the assays do not need to be perfect but “fit for purpose,” Dr. Miller says. “They just have to be good enough to be able to get reliable results that physicians can use to make medical decisions about patient care.” He commends the educational efforts of the AACC and the CAP’s strong support of harmonization through articles and expansion of its accuracy-based Surveys. “That is essential to understanding what needs to be harmonized and how well harmonized or not harmonized results are.”
Still more outreach is needed to bring harmonization into the spotlight, however. “There are many people who haven’t thought about it yet and don’t really understand that harmonization is critically important and there are now tools to achieve it,” Dr. Miller says. “More education will help our profession understand that harmonization is important and can be achieved with the new tools being developed.”n
Anne Paxton is a writer and attorney in Seattle.