"No Difference" May Be the Wrong Conclusion, Scientists Warn
Image Credit: Courtesy of Universities of Manchester
Scientific Frontline: Extended "At a Glance" Summary: Equivalence Testing in Scientific Research
The Core Concept: Equivalence testing is a statistical approach that asks whether any observed difference in research data is too small to be meaningful, rather than simply asking if there is evidence of a difference.
Key Distinction/Mechanism: Instead of traditional statistical testing where a "non-significant" result (like a p-value > 0.05) is frequently misinterpreted as proof of "no effect," equivalence testing differentiates between effects that are truly negligible and results that are inconclusive due to insufficient data.
Origin/History: Highlighted in an August 2026 publication in PNAS by researchers from the Universities of Manchester, Oxford, and Arkansas, promoting the two one-sided tests (TOST) procedure.
Major Frameworks/Components:
- P-value misinterpretation: The common error of assuming a p-value greater than 0.05 means "no effect" rather than "insufficient evidence."
- Two one-sided tests (TOST): A specific equivalence testing procedure that is currently used in psychology and medicine but underused in other life and natural sciences.
- Practical equivalence: The requirement for researchers to define, before data collection, how small an effect must be to be considered scientifically or clinically uninteresting.
Branch of Science: Statistics, Data Science, Methodology (applicable across all Life and Natural Sciences).
Future Application: Broader adoption of equivalence testing, aided by newly developed free online calculators, to improve scientific reporting and prevent promising therapies or genuine risks from being incorrectly dismissed due to small sample sizes or noisy data.
Why It Matters: Relying on traditional significance testing can lead researchers to overlook important effects or misrepresent inconclusive findings as evidence of absence, potentially derailing future research.
Researchers from the Universities of Manchester, Oxford, and Arkansas are urging scientists to stop treating "nonsignificant" results as proof that nothing happened.
The common statistical mistake, they warn in a new paper published in PNAS on August 10, 2026—one of the world’s leading scientific journals—could be leading researchers to draw the wrong conclusions from their data.
A p-value—which measures how surprising the observed data would be if there were truly no effect—of greater than 0.05 does not, they say, show there is "no effect."
However, the interpretation remains widespread in about 50% of research papers and conference presentations, according to sources.
Instead, the authors say, a nonsignificant result simply means there is insufficient evidence to conclude a difference exists, not proof that a difference does not exist.
Crucially, the same statistical result can arise either because there is genuinely no meaningful effect or because an important effect is hidden by small sample sizes or highly variable data.
The researchers argue that failing to recognize this distinction risks oversimplifying scientific findings and may cause potentially important effects to be overlooked.
With many studies lacking a large enough sample size to detect a real difference, reporting that there is none can obscure promising therapies or fail to detect genuine risks, derailing subsequent research in the area.
To address the problem, the team is promoting a statistical approach known as equivalence testing. Equivalence testing, they say, guards against declaring an effect unimportant when the data cannot support that claim. A small or noisy study will usually return an inconclusive result rather than a verdict of no difference, which is the honest answer.
Rather than asking whether there is evidence for a difference, equivalence testing asks whether any difference that exists is too small to matter in scientific, clinical, or practical terms.
The method allows researchers to distinguish between effects that are genuinely negligible and results that remain inconclusive because there is not enough reliable evidence.
They focus on the two one-sided tests procedure, or TOST, which has already gained traction in psychology, medicine, and pharmaceutical regulation but remains underused across many areas of the life and natural sciences.
Wider adoption of equivalence testing, they add, could improve the quality of scientific reporting and help prevent nonsignificant findings from being misrepresented as evidence of no effect.
Co-author David Eisner, professor of cardiac physiology at the University of Manchester, said, “Researchers are often interested in whether an effect is absent or too small to be important, but traditional statistical testing cannot answer that question.
“A nonsignificant result is frequently interpreted as proof that nothing happened, when it may simply mean there is not enough evidence to be certain.”
Co-author Jakub Tomek of the University of Oxford said, “Equivalence testing helps separate genuinely trivial effects from unresolved questions, giving scientists a much clearer picture of what their data are actually telling them and improving confidence in the conclusions that are reported.
“We hope the method will encourage more careful interpretation of scientific data and improve the way research findings are reported, understood, and acted upon.”
To make the approach more accessible, the team has also developed a free online calculator that allows researchers to perform common equivalence tests without writing computer code.
The calculator supports a range of widely used statistical comparisons and has been validated against established statistical software.
Co-author Aaron Caldwell of the University of Arkansas for Medical Sciences added, "Equivalence testing forces you to answer a question most studies never ask: how small is small enough to be uninteresting?
"That is a scientific judgment, not a statistical one, and it has to be made before the data are collected. The payoff is that you can finally say something affirmative about an absence of effect rather than just failing to find one."
Published in journal: Proceedings of the National Academy of Sciences
Title: Beyond “non-significant” results: Why and how to test for practical equivalence
Authors: Jakub Tomek, Aaron Caldwell, and David A. Eisner
Source/Credit: Universities of Manchester
Edited by: Scientific Frontline
Reference Number: sn081726_01