A coalition of leaders in the cancer genomics research community has published the first findings from a project to develop robust methods for detecting cancer mutations. The results, published May 18 in Nature Methods, provide an important new benchmark for researchers, helping to define the most accurate methods for identifying somatic mutations in cancer genomes. The study could be the first step in creating a new global standard to determine how well cancer mutations are detected.
Coauthor Josh Stuart, professor of biomolecular engineering at UC Santa Cruz, helped organize the ICGC-TCGA-DREAM Somatic Mutation Calling Challenge, which merged the efforts of the world's largest cancer genome sequencing consortia--the International Cancer Genome Consortium (ICGC) and The Cancer Genome Atlas (TCGA)--with those of Sage Bionetworks and the DREAM project (Dialogue on Reverse Engineering Assessment and Methods). Other project leaders include cancer researchers at the Ontario Institute for Cancer Research (OICR) and Oregon Health & and Science University (OHSU).
"We owe it to cancer patients to interpret tumor DNA information as accurately as we can. This study represents yet another great example of harnessing the power of the open, blinded competition to take a huge step forward in fulfilling that vision," said Stuart, a lead TCGA representative on the project. "We still have important work ahead of us, but accurate mutation calls will give a solid foundation to build from."
Tumor genomes
The SMC Challenge, which was initiated in November 2013, was an open call to the research community to address the need for accurate methods to identify cancer-associated mutations from whole-genome sequencing data. Although genomic sequencing of tumor genomes is exploding, the mutations identified in a given genome can differ by up to 50 percent just based on how the data is analyzed.
Research teams were asked to analyze three in silico (computer simulated) tumor samples and publicly share their methods. The 248 separate analyses were contributed by teams around the world and then analyzed and compared by Challenge organizers. When combined, the analyses provide a new ensemble algorithm that outperforms any single algorithm used in genomic data analysis to date.
The authors of the paper also report a computational method, BAMSurgeon (developed by co-lead author Adam Ewing at UC Santa Cruz), capable of producing an accurate simulation of a tumor genome. In contrast to tumor genomes from real tissue samples, the Challenge organizers had complete knowledge of all mutations within the simulated tumor genomes, allowing comprehensive assessment of the mistakes made by all submitted methods, as well as their accuracy in identifying the known mutations.
Dramatic differences
The submitted methods displayed dramatic differences in accuracy, with many achieving less than 80 percent accuracy and some methods achieving above 90 percent. Perhaps more surprisingly, 25 percent of teams were able to improve their performance by at least 20 percent just by optimizing the parameters on their existing algorithms. This suggests that differences in how existing approaches are applied are critically important--perhaps more so than the choice of the method itself.
The group also demonstrated that false positives (mutations that were predicted but didn't actually exist) were not randomly distributed in the genome but instead they were in very specific locations, and, importantly, the errors actually closely resemble mutation patterns previously believed to represent real biological signals.
"Overall these findings demonstrated that the best way to analyze a human genome is to use a pool of multiple algorithms," said co-lead author Kathleen Houlahan, a junior bioinformatician at OICR working with the Challenge lead, Dr. Paul Boutros. "There is a lot of value to be gained in working together. People around the world are already using the tools we've created. These are just the first findings from the Challenge, so there are many more discoveries to share with the research community as we work through the data and analyze the results."
Team sport
"Science is now a team sport. As a research community we're all on the same team against a common opponent," said Dr. Adam Margolin, director of computational biology at OHSU and co-organizer of the challenge. "The only way we'll win is to tackle the biggest, most challenging problems as a global community, and rapidly identify and build on the best innovations that arise from anywhere. All of the top innovators participated in this Challenge, and by working together for a year, I believe we've advanced our state of knowledge far beyond the sum of our isolated efforts."
Dr. Stephen Friend, president of Sage Bionetworks, said the goal is "no longer one of winning the Challenge but instead of constantly addressing an ever-changing horizon. And given the complex heterogeneity of cancer genomes and the rapid rate with which next-generation sequencing technologies keep changing and evolving, this seems like an ideal approach to accelerate progress for the entire field."
In addition to Ewing and Stuart, UCSC coauthors of the paper include David Haussler, professor of biomolecular engineering and director of the UC Santa Cruz Genomics Institute, and Kyle Ellrott, a software developer at the institute.
DREAM Challenges pose fundamental questions about systems biology and translational medicine. Designed and run by a community of researchers from a variety of organizations, DREAM challenges invite participants to propose solutions--fostering collaboration and building communities in the process.
