Eran Elhaik: ‘Some of the biggest lies are those ‘I took 5 tests and here is what I found’ articles’

Ancestry test results were unreliable

The scientific data collection rate is rising exponentially, leading to massive and highly complex datasets, dubbed the “Big Data revolution.” To make these data more manageable, researchers use statistical methods that aim to compact and simplify the data while still keeping most of the key information.

PCA (principal component analysis) is perhaps the most widely used method. By analogy, think of PCA as an oven with flour, sugar and eggs as the data input. The oven may always do the same thing, but the outcome, a cake, seriously depends on the ingredients’ ratios and how they are combined.

“It is expected that this method will give correct results because it is so frequently used. But it is neither a guarantee of reliability nor produces statistically robust conclusions,” says Dr Eran Elhaik, associate professor in molecular cell biology at Lund University.

According to Elhaik, the method helped create old perceptions about race and ethnicity. It plays a role in producing historical tales of who and where people come from, not only by the scientific community but also by commercial ancestry companies. For instance, a prominent American politician took an ancestry test before the 2020 presidential campaign to support their ancestral claims. Another example is the misconception of Ashkenazic Jews as a race or an isolated group driven by PCA results.

“This study demonstrates that those results were unreliable,” says Eran Elhaik.

These results must be re-evaluated

PCA is used across many scientific fields, but Elhaik’s study centres on its usage in population genetics, where the explosion in dataset sizes is particularly acute, which is encouraged by the reduced costs of DNA sequencing.

The field of paleogenomics, where we want to learn about ancient peoples and individuals such as Copper age Europeans, heavily relies on PCA. PCA is used to create a genetic map that places the unknown sample alongside known reference samples. Thus far, the unknown samples have been assumed to be related to whichever reference population they overlap or lie closest to on the map.

However, Elhaik discovered that the unknown sample could be made to lie close to virtually any reference population just by changing the numbers and types of the reference samples, generating practically endless historical versions, all mathematically “correct,” but only one may be biologically correct.

In the study, Elhaik has examined the twelve most common population genetic applications of PCA. He has used both simulated and real genetic data to show just how flexible PCA results can be. According to Elhaik, this flexibility means that conclusions based on PCA cannot be trusted since any change to the reference or test samples will produce different results.

Authors have employed PCA in from 32,000 to 216,000 scientific articles in genetics alone for exploring and visualizing similarities and differences between individuals and populations and based their conclusions on these results.

“I believe these results must be re-evaluated,” says Elhaik.

He hopes that the new study will develop a better approach to questioning results and thus help to make science more reliable. He spent a significant portion of the past decade pioneering such methods, like the Geographic Population Structure (GPS) for predicting biogeography from DNA and the Pairwise Matcher to improve case-control matches used in genetic tests and drug trials.

“Techniques that offer such flexibility encourage bad science and are particularly dangerous in a world where there is intense pressure to publish. If a researcher runs PCA several times, the temptation will always be to select the output that makes the best story,” adds Professor William Amos, from the University of Cambridge, who was not involved in the study.

“The genetic ancestry industry is built on storytelling”

Exclusively for Realnoe Vremya Doctor Eran Elhaik answered several questions.

Does it mean that most ancestry test results are wrong?

If they were calibrated based on PCA, like 23andme openly admits, then yes. They will be biased. Other companies are not forthcoming with their algorithms. My True ancestry heavily relies on PCA.

Who must be held accountable for these flaws, since so many people around the world have spent huge money on these tests?

The scientific community and the media. Scientists — for using methods without due diligence and critical thinking. Reviewers and editors — for ignoring those problems. Journals — for publishing hyped studies — sometimes, knowing perfectly well that they are wrong. The press — for hyping the research even more. It’s a win-win cycle for everyone involved in the industry.

Those who suffer are the scientists who are interested in the truth — medical research, etc. And the public who relies on these studies to make medical decisions or study their ancestry (see the case of Senator Warren) The public is not expected to know how PCA works, but now that the information is out, the public should demand some answers.

Apparently, this industry is not always about science but making money. How to tell if a lab doing an ancestry test will provide accurate information? Are there any recommendations?

The genetic ancestry industry is built on storytelling and selling dreams with some obvious recommendations (eat healthy, exercise, etc.). Most genetic companies do not use ancient DNA, which should have been used to test the stories that they told people. They don’t do it, because it would conflict with their stories, or because they use ancestry test to lure people to give their genetic data, for some other purpose.

You can never know the truth. Some of the biggest lies are those “I took 5 tests and here is what I found” kind of very popular articles. They tell you nothing about the truth, except how similar the algorithms of those companies. That’s not the truth, that’s how similar the tools are. The public has no tool to reach the truth.

My suggestions are to avoid genetic testing that sells hyped claims and find reliable tests with credible scientists behind them that do not use PCA.

What consequences do you think the results of your study are going to lead to?

Minimal, at best. One paper cannot break the cycle of rapid paper production, hype, high-impact publications, generous grants, and promotions that underlies this community. The reason that we have a replicability crisis, to begin with, is that science is broken.

I am not being a pessimist without a reason, a draft of this paper was available online for a year. During this year, I have seen very few authors replacing PCA with a PCA-like tool that is equally problematic; other authors did not even bother.

As you wrote about an AI-based dating method in the previous study, can AI come in handy again here to avoid a human factor when doing a test? Because I understood that humans still play a big role when interpreting the results and can pursue the goal of making the headlines.

The problem is more with the climate in science than with this or other tool. But, your suggestion is in the right direction. This is exactly what we are trying to do now, in combination with ancient DNA.