November 2015—More than two years ago, when the CAP decided to move forward with proficiency testing for next-generation sequencing, the decision point was modest. “We estimated that about 35 labs would subscribe, based on survey information, and that was sufficient for us to move forward,” says Karl V. Voelkerding, MD, chair of the CAP Next-Generation Sequencing project team.
The first round of PT in a new area, known as the A mailing, is traditionally an educational challenge, with labs not formally graded on their results but instead provided with a summary with which to crosscheck their answers against the CAP answers. But even within this limited framework, signups for the first NGS Survey, covering germline variants, exceeded hopes. “There were quite a bit more in the end than was initially anticipated,” says Jason Merker, MD, PhD, chair of the CAP Molecular Oncology Committee. “We had approximately 130 labs enroll.”
Now the CAP plans to crown that success with a broader menu. For 2016, it will add three more sets of PT challenges in next-generation sequencing: solid tumors, hematologic malignancies, and bioinformatics.
Any time the CAP envisions a new Survey, the first question a team asks is, “Will this be a PT product that a sufficient number of labs will subscribe to, to make this a sustainable effort economically for the College?” says Dr. Voelkerding, professor of pathology at the University of Utah and medical director of genomics and bioinformatics at ARUP Laboratories. To answer that question for NGS PT, the CAP asks accredited labs to list on their activity menu whether they are using NGS.
Proficiency testing for NGS bioinformatics is one of the boldest among the CAP’s PT ventures, Dr. Voelkerding says. “It’s a very important initiative of the College, with the goal of diversifying and augmenting the types of PT challenges we can provide labs that are doing NGS.”
In the NGS world, an important component of the overall testing process is the analysis of sequence data. “It’s all done computationally, and then the summation of that data is interpreted by the laboratory director,” he says. “We foresaw a couple of years ago that we could create additional PT challenges by querying the labs’ ability to analyze the sequence data that we would provide in silico [performed by computer simulation]. We use in-silico computational tools to take a given data set and then artificially introduce into it mutations and genetic variations in areas of interest, then send those data sets to clinical labs to process through their own bioinformatics computational pipeline. And they report back to the CAP what they’ve observed.”
The first bioinformatics PT challenge will have a focus on molecular oncology and the detection of somatic mutations. Twenty-four genes are included in the 2016 NGS Bioinformatics Survey: AKT1, ALK, APC, BRAF, CDKN2A, CTNNB1, EGFR, ERBB2, FBXW7, FGFR2, GNAQ, GNAS, HRAS, IDH1, KIT, KRAS, NRAS, PDGFRA, PIK3CA, PTEN, RET, SMAD4, STK11, and TP53.
“In 2017 and going forward, there will be further emphasis on bioinformatics-based challenges,” Dr. Voelkerding says, specifically in the area of germline variation and genetic disorders.
NGS proficiency testing in bioinformatics is new, he emphasizes. “It’s a response to and reflection of the fact that NGS has a substantial and unprecedented amount of bioinformatics data analysis, and because of that, it really adds a complexity to the overall testing process that needs to be assessed. Interestingly, those testing processes can be assessed as computational bioinformatics-type PT challenges.”
Laboratories that perform NGS take diverse approaches to their information pipelines, employing varying degrees of automation. “In some cases, the way the information is generated and reviewed may involve a series of Excel spreadsheets. Some labs are likely doing the transfer and summation of their data set outputs manually, or the data set may be interfaced into a Web-based program through a commercial vendor or one that is developed internally by the lab.”
“The complexity of NGS results requires a tremendous amount of creativity in how you compile those results in a report that can be reviewed by a laboratory director and then ultimately delivered to the ordering physician,” Dr. Voelkerding says.
So there is great diversity in approach, but it’s important to assess the range of those approaches through proficiency testing of bioinformatics pipelines, Dr. Voelkerding says. In fact, an initial feasibility pilot for the NGS bioinformatics proficiency test signaled there were difficulties in accommodating sequence data formats. “We observed that the format in terms of the sequencing data we provided to the labs could become a barrier. If we provided a data set formatted in a specific way and an individual lab’s bioinformatics computational pipeline uses input data in a different format, then some labs’ pipelines were able to accommodate that, and some labs’ pipelines were not.”
The CAP, guided by the experience of the initial feasibility pilot, conducted a second feasibility pilot based on specific data formats that are generated when using commercially available reagents for molecular oncology testing by NGS for specific instruments. “The second feasibility pilot demonstrated that labs could accommodate that standardized format, because they were using specific commercial reagents and software pipelines and could analyze and process data very efficiently and very accurately,” Dr. Voelkerding says. The bioinformatics PT set to launch in 2016 is based on the approach developed during this second feasibility pilot. (John Pfeifer, MD, PhD, provided collaborative input to the NGS project team members in his role as chair of the CAP Personalized Health Care Committee.)
Over the long term, Dr. Voelkerding projects, “We think the solution—and this is part of our development efforts—is to essentially create customized bioinformatics challenges on individual laboratory tests.” They envision that individual laboratories will provide the CAP with their generated data sets, he says. “We would then use in-silico algorithms to create a series of specific mutations in specific genes in their provided data set, then return that data set to them, without altering the overall format. When the lab receives back the in-silico data set, they would be able to run it through their pipeline and identify what differences we introduced.”
That customized approach, still in the developmental stage, is a reflection of how diverse the technical methods and approaches developed and implemented by clinical laboratories have turned out to be. “In some specific testing areas, there has been more of a movement toward harmonization of methodologies, but overall, we’re continuing to see an expansion of diversity, and that reflects that more labs are performing NGS-based testing, and they are each working through their own approaches.” Secondarily, he adds, commercial vendors are offering more and more technological methods and options to support the various process steps needed to generate an NGS test.
Meanwhile, evaluation of the initial results of the first NGS proficiency test continues, Dr. Voelkerding reports. He expects the bioinformatics and other new NGS PT challenges to be available by the end of the first quarter of 2016. “We’re currently reviewing the A mailing for 2015, then results will be coming from the B mailing in December.”
The other two NGS Surveys to be added in 2016 are for solid tumors and hematologic cancers. “We’re finalizing these now and they will be sent out to labs in early 2016,” Dr. Merker says. The Molecular Oncology Committee he chairs has been collaborating with the NGS project team, the CAP Personalized Health Care Committee, and the CAP/ACMG Biochemical and Molecular Genetics Resource Committee to develop the new NGS Surveys.
Survey development is a process that takes time, says Dr. Merker, co-director of the Stanford Medicine Clinical Genomics Service and assistant professor of pathology at Stanford University School of Medicine. “We needed to identify well-characterized specimens that would be usable for a broad variety of assays, and, consequently, we had to design and manufacture those materials.” Members of the Molecular Oncology Committee tested the materials already available and found there were challenges with those materials for some assay types. “So we specifically redesigned materials to get around those issues.”
The growth in NGS has already been striking, and Dr. Merker expects it to continue. “We’ve seen a dramatic uptake of NGS for heritable disease testing, initially with panel-based testing and more recently with broader-based testing such as exome or genome testing.” Likewise, survey data from clinical laboratories that participate in other CAP molecular oncology PT challenges indicate “there has been and will continue to be a significant increase in the use of NGS-based methods for somatic variant detection, particularly for testing solid tumors,” he says. Factors at play are improvements in sequencing accuracy, decreases in cost, and the availability of smaller, easier to operate, and more affordable desktop sequencers.
The Molecular Oncology Committee has led the effort to design PT for NGS-based testing for detection of somatic variants observed in solid tumors and hematologic malignancies. Says Dr. Merker: “For each Survey mailing, the CAP will provide three engineered specimens, composed of genomic DNA that contains up to six somatic variants in the genes being examined in the respective Surveys. Based on questionnaires included in other molecular oncology Surveys, we selected those genes and variants such that the Survey will be useful for most laboratories performing this type of testing.”
The NGS Solid Tumor Survey will focus on the detection of somatic single nucleotide variants and small insertions or deletions observed in solid tumors. Twenty-eight genes are included in the NGS Solid Tumors Survey: AKT1, ALK, APC, ATM, BRAF, CDH1, CTNNB1, EGFR, ERBB2, FBXW7, FGFR2, GNAQ, GNAS, HRAS, IDH1, KIT, KRAS, MET, NRAS, PDGFRA, PIK3CA, PTEN, SMAD4, SMARCB1, SMO, SRC, STK11, and TP53.
The NGS Hematologic Malignancies Survey will focus on the detection of somatic single nucleotide variants and small insertions or deletions observed in hematologic malignancies. Twenty-four genes are included in that Survey: ASXL1, ATM, BRAF, CALR, CEBPA, CREBBP, CSF3R, DNMT3A, EZH2, FLT3, IDH1, IDH2, JAK2, KIT, KMT2D, MPL, MYD88, NOTCH1, NPM1, SF3B1, SRSF2, TET2, TP53, and U2AF1.
Although Dr. Merker is optimistic about the utility of the new NGS Surveys, he is confident the Surveys will confirm the high quality of laboratories’ performance in NGS. “Our PT experience to date indicates that labs are doing an exceptional job of testing in molecular oncology, and data generated during the piloting of these new Survey materials indicate high performance among pilot laboratories. We look forward to seeing a similar performance in our NGS Surveys next year.”
In the coming years, the CAP hopes to achieve a diversity of PT challenges, Dr. Voelkerding says, and that’s why the NGS project team he leads includes members of a variety of CAP resource committees. That cross-fertilization, or “cross-talk,” among project team members is important, he says. “In essence we’re creating a portfolio and infrastructure that can address not only the PT needed for germline and somatic variants for human diagnostic testing but also leveraged for the development of PT for NGS-based assays for infectious diseases and other clinical needs.”
Anne Paxton is a writer in Seattle. For more information about NGS Surveys, go to “PT Order Supplements” at http://j.mp/cap_catalogs.