Kevin B. O’Reilly
February 2015—When you go looking for problems, you’re bound to find them. That truism is especially pertinent in the arena of interface validation, as the team at New York’s North Shore-LIJ Health System discovered recently.
The laboratory professionals there were charged with helping to implement the first phase of a joint venture with New York City’s Health and Hospitals Corp. (HHC), in which North Shore-LIJ would serve as the massive public health system’s primary reference lab. Their job was to interface through middleware and validate, in just eight months, 840 HHC tests coming from 21 of that system’s sites. Meanwhile, the North Shore-LIJ team has had to contend with validating the plethora of electronic health record systems at more than 300 physician clients’ offices, as well as a customized smartphone application. North Shore-LIJ Laboratories has long performed interface validations, but they learned just how arduous it can be when so many have to be performed at once.
The essential task of interface validation is incontrovertible, required by CLIA regulations and accrediting bodies such as the CAP. Laboratories must ensure that ordered tests are correctly submitted to the laboratory information system and results are accurately sent to the electronic interface at the other end, displayed properly on screen, and able to be faxed or printed without becoming garbled.
The opportunities for potentially patient-endangering fumbles from one end to the other are nearly too numerous to count. A recent Q-Probes study, “Validating Laboratory Results in Electronic Health Records,” found that of the 1,064 results the 45 participating institutions looked at, 99 percent were found to have been transmitted accurately from the LIS to the EHR. But 30 percent of the time the results transmitted were incomplete. And only 91 percent of the test results were formatted appropriately. (See “LIS to EHR: Is results transmission what it should be?” CAP TODAY, January 2015, page 42.)
So the team at North Shore-LIJ, along with its counterparts at HHC, set out to ensure accurate reporting of patient lab results. They worked at multiple levels—LIS, medical leadership, quality management, and laboratory staff—to establish the plan, execute the validations, and provide direction when changes in the validation process became necessary. “Failure wasn’t an option for either party,” says Hannah Poczter, MPH, DLM(ASCP), assistant vice president of laboratory services at North Shore-LIJ Health System Laboratories.
What did they find, despite much planning and discussion? Unflagged abnormal results, improper or illogical user display of results, report formatting problems, truncated test result text or comments, incorrect reference ranges or units of measure, and scrambling of complex reports that had a lot of text and tables.
The team also discovered, however, that the validation process itself had to be fixed if they were to have any hope of finishing the job on time and without spending a fortune. What began under the leadership of the LIS department, Poczter says, came to require a broader view to ensure success and handle a high-tech future that will see an ever-increasing number of interfaces that require validation.
“I’m a little more at ease, knowing that I have a team of people here and that we have created a template for how to do the validations, and what to look for,” Poczter says. “We already know the obstacles and should be able to continue doing the validations in a timely manner, making sure the end user gets an accurate report.
“It’s a very good journey from where we started to where we are right now,” she tells CAP TODAY. “It’s a very good experience to have solidified the teamwork that we have here within our own laboratories and with our partners as well.”
But the experience was difficult at the start, said Carol J. Sien, MS, MT(ASCP), CQA(ASQ), manager of quality management at North Shore-LIJ Laboratories, based in Lake Success, NY.
“There were very long hours. There were days we used to spend 12 to 14 hours, weekends, working into the wee hours of the night” with “excessive rework,” Sien told the crowd at the Dark Report’s Lab Quality Confab last October.
The 21 HHC sites whose tests had to be validated were divided into eight hubs, with each hub differing from the others in some way that required a separate validation process.
“Even though they were on the same computer system, each of the hubs had differences in terms of their test compendium, test name, their order codes or result codes, and even how they navigated through their LIS screen menus,” Sien said. “Because of these differences, and the fact that each of the hubs’ HIS and LIS was not maintained in the same manner, each hub had to be analyzed and mapped separately.”
North Shore-LIJ started working with its HHC partners on the validation initiative in March of last year and had November 2014 as the deadline for completion. But after three months’ work, validation had been completed for only two of the 21 sites, Sien said.
“The turning point was the test validation of that first hub,” Sien tells CAP TODAY. “The process was disorganized, with a high failure rate of test scripts. Documents were not always provided, and we couldn’t do the comparisons to make sure everything went accordingly. We had a lack of continuity in providing the data.… We knew that we had to stop and really streamline the process.”
In those first three months of work, the validation plan required an eight-step process for each packet of 10 tests being tested, Sien said in her Lab Quality Confab talk.
“You had to place the order in the HHC system. You had to review the orders in the North Shore-LIJ system. You had to result the orders in the North Shore-LIJ system and result the orders at the reference lab if submitted to an outside lab for processing,” she said. “Then you also have to review the results in the HHC LIS system, and then quality management had to review all of the test documents. The medical director and senior management needed to sign off on both the validation plan and the test documents.”
That process, at first, required the attention of 15 full-time-equivalent employees. Meanwhile, an attempt to speed the validation process—running 10 tests through the gauntlet at once—delayed matters.
“There were too many tests assigned to one test script,” Sien said. “It made tracking the completion of a test really a monumental task. And with 10 tests per test script, if any one test failed or was missing a document, we had to fail the entire test script packet.
“Another issue,” she added, “was that the test validation documents weren’t easy to obtain from HHC, North Shore-LIJ, or the reference lab if needed. Also, the test validation plan wasn’t always followed according to plan, and the test script checklist was difficult to keep accurate and up to date. Not everyone was aware of what was going on in the entire process since we had so many people involved. There were misunderstandings about what everyone needed to do and what were the roles of each individual.”
A separate challenge was that the team had to track its progress properly.
“We had multiple issue logs. The issues were not always the same from log to log. The information was either missing, or different, and there was poor or no communication on what the status of a test script was—whether it was completed and ready for review, or whether it was failed and being retested. And when there was a failed test script, it wasn’t always reassigned in a manner that we would identify as being a retest.”
The initial effort “was so convoluted, confusing, and lacking in communication we decided to re-evaluate the process by performing a Lean event,” Sien said. That itself was a major time commitment amid a project already moving at a sluggish pace. Over three months of weekly meetings, the revamped team of lab operations, LIS, technical staff, and quality management professionals restructured and streamlined the validation process.
Instead of attempting to validate 10 tests at a time, the team opted to tackle them one by one. The team also started using a shared Web-based spreadsheet to track the progress of the tests being validated. To prioritize tests for validation, the team obtained a compendium of tests and test volumes from each HHC site. Weekly “touch point” meetings after the restructuring helped keep the lines of communication clear, Sien said. The number of validation documents required per test also was cut.
The validation process that had once required 15 FTEs now needed only five. The three-month validation turnaround for the first hub was cut to a one-month process for the other seven. While 35 percent of test scripts failed validation during the pre-Lean, 10-tests-at-a-time process, only three percent failed after the revamp. Projected across the seven other HHC hubs whose tests the team would go on to validate, the restructuring could save millions, Sien tells CAP TODAY. In all, the team validated 6,120 HHC test results by the November 2014 deadline.
“The success of the HHC validation project can be attributed to the development of a cohesive relationship and open communication between HHC and North Shore-LIJ,” Poczter says. The team continues to work together “to further enhance the level of service and quality provided,” she adds.
The partnership with HHC will involve more work soon, including a jointly controlled central laboratory that will conduct routine tests for the two giant health systems. (North Shore-LIJ performed more than 8 million tests in 2013.) In the meantime, the North Shore-LIJ labs have had their hands full working to validate the increasing number of their physician clients using EHRs. Of its 1,400 clients, about a quarter now have some sort of electronic interface. Another 150 client interfaces are awaiting legal clearances, says Ed Giugliano, PhD, project manager at North Shore-LIJ Health System Laboratories.
Dr. Giugliano said his team’s work in this area benefited greatly from the prior efforts to streamline validation as part of the HHC project. He explained the collaborative validation process during his presentation at the Lab Quality Confab.
“This EMR validation flow process entails the EMR team and the LIS team. The EMR team registers the patient, places the orders, and prints the labels. The LIS team reviews and verifies receipt of this information in the lab information system, then receives the orders and results the tests. The LIS and EMR teams each send their respective screenshots to the quality management team for review and validation.
“Now, if the test script passes and the validation can be signed off, it will eventually go into production,” he added. “However, if the test script fails, the EMR and the LIS teams work in concert to correct the issue, reorder the test, and revalidate it—and the cycle continues.”
While running 10 tests at a time presented big problems for the HHC validation project, Dr. Giugliano explained that his team ran between one and 15 tests at a time through the validation process without much trouble.
“It worked well for us,” he said.
The team looked to ensure that fields containing patient demographics, billing information, test names, results review, comments, reflex orders, and corrected, amended, or appended results were transmitted accurately and displayed properly. And what did they run into? In cases of send-out testing, sometimes multiple labs were listed as performing the test. Sometimes order comments were incomplete, lacking units for gestation age, for example. There were missing reference ranges and units for vitamins D and K testing, calculation problems for other tests, and collection and report time discrepancies.
The biggest impediment to fixing these sorts of problems came not so much from physicians or from the vendors, but from working with multiple parties to ensure the process of transmitting results was safe and error-free, Dr. Giugliano tells CAP TODAY.
“With multiple partners, it was difficult to determine exactly where the ball was being dropped. Eventually, we created an environment of teamwork to remedy the issues.”
Doctors’ offices typically lack access to information technology support, knowledge, and resources, Dr. Giugliano added. There also is a deficit in grasping interface validation as a priority.
“They don’t truly understand the importance of laboratory data integrity and what happens downstream if that data has issues,” he said.
One area in which ordering physicians have been eager to work with North Shore-LIJ labs is in obtaining access to test results on their smartphones.
“We have a very active sales group that sells our services to the physicians in the community,” Dr. Giugliano says. “It became very apparent to us that our competition had a similar mobile app, so that put the pressure on us to start thinking about it. And a number of our physician clients started asking whether or not they could have something like that from us.”
Working with the vendor, physician testers, and others to validate the app for North Shore-LIJ was far from the massive HHC undertaking, but the one-month project involving 50 patients and 85 tests did reveal problems that had to be corrected before going live, Dr. Giugliano said.
For example, there was a special chemistry test result for vancomycin that was critically high. “But the app displayed the red dot over the high end instead of way off to the right end of the range, which would have been the critically high end,” Dr. Giugliano says. “That was on an Android phone. I looked it up with my iPhone and the dot was positioned properly. It’s the phone software that just displays the actual results a little differently.” In another case, an abnormal malaria test result wrongly displayed as a green dot, which was the color assigned to normal results.
The fix was to change the way normal results display.
“Instead of having it appear as a green dot, we just removed the ability of the app to show visually what a normal looks like,” he says. “We changed green dots, which indicate normal results, to a neutral gray color. So the physician now has to pay attention to the lab result values which are simultaneously displayed. We want the physician to make the call, and say, ‘Yes, this result is indeed normal or not.’”
Meanwhile, pediatric patient ages of less than a year, such as five months old, displayed incorrectly as one year. In molecular karyotype results, plus signs were transmitted wrongly as spaces. The physician user sometimes was unable to view the previous version of a corrected report. Foul-ups arose not only due to a problem with the phone itself, but also with the app (as in the case of the pediatric age miscalculations and the inability to recognize a plus sign). Also, issues arose because of something on North Shore-LIJ’s end, Dr. Giugliano said.
“We realized that a number of issues were identified pertaining to the data feed relating to patient care, and these required an LIS fix of our actual feed,” he said. “We had extraneous comments and page numbers present. There were some disclaimers missing, the performing lab, some other items. We’re pleased to see that the website hard copy and phone displays reflect basically the same information because they got the information from the same feed.” A data feed fix corrected downstream data issues as well.
App security was another issue to address. To use the app, physician users must have an alphanumeric password and sign a form promising to adhere to confidentiality and HIPAA requirements. Users can call patients directly from the app and have results emailed or faxed (with a confidential cover page).
Nearly 700 physician clients use the North Shore-LIJ Health System Laboratories’ app, introduced in 2013 and available for Apple iPhones as well as devices running the Android operating system. They now have very few problems or complaints.
“We listened to physicians in our client base and via our physician advisory committee to develop direct patient calling through the app, and to provide fax, printing, and email capabilities,” Dr. Giugliano says. Physicians can be confident that validations were thorough and the result transmission is accurate, he adds. “They appear to be highly satisfied with our product.”
For the professionals involved with validation at North Shore-LIJ Health System Laboratories, the biggest takeaway from their recent experience is that the endeavor can reveal not only problems with how tests are submitted, resulted, transmitted, or displayed but also with a laboratory’s validation process itself.
For Poczter, the North Shore-LIJ assistant VP of laboratory services, the interface validation process requires a division within quality management “solely dedicated to interface validation,” though she notes this may not be the case for other laboratories. The process should not be left entirely to LIS departments that are often short-staffed and may lack some of the knowledge needed, she adds. “Instead, one should engage the expertise of various other laboratory professionals as well,” who can help spot important problems with test results transmission or display.
Such quality management devotion to the validation process “may not be possible for small or medium-sized laboratories that are also faced with the multitude of validations they have to perform,” Poczter says. “But creating an expert team would probably be required with what’s coming up at other laboratories.”
Kevin B. O’Reilly is CAP TODAY senior editor.