Skip to main content
About HEC About HEC
Summer School Summer School
Faculty & Research Faculty & Research
Master’s programs Master’s programs
Bachelor Programs Bachelor Programs
MBA Programs MBA Programs
PhD Program PhD Program
Executive Education Executive Education
HEC Online HEC Online
About HEC
Overview Overview
Who
We Are
Who
We Are
Egalité des chances Egalité des chances
HEC Talents HEC Talents
International International
Sustainability Sustainability
Diversity
& Inclusion
Diversity
& Inclusion
The HEC
Foundation
The HEC
Foundation
Campus life Campus life
Activity Reports Activity Reports
Summer School
Youth Programs Youth Programs
Summer programs Summer programs
Online Programs Online Programs
Faculty & Research
Overview Overview
Faculty Directory Faculty Directory
Departments Departments
Centers Centers
Chairs Chairs
Grants Grants
Knowledge@HEC Knowledge@HEC
Master’s programs
Master in
Management
Master in
Management
Master's
Programs
Master's
Programs
Double Degree
Programs
Double Degree
Programs
Bachelor
Programs
Bachelor
Programs
Summer
Programs
Summer
Programs
Exchange
students
Exchange
students
Student
Life
Student
Life
Our
Difference
Our
Difference
Bachelor Programs
Overview Overview
Course content Course content
Admissions Admissions
Fees and Financing Fees and Financing
MBA Programs
MBA MBA
Executive MBA Executive MBA
TRIUM EMBA TRIUM EMBA
PhD Program
Overview Overview
HEC Difference HEC Difference
Program details Program details
Research areas Research areas
HEC Community HEC Community
Placement Placement
Job Market Job Market
Admissions Admissions
Financing Financing
FAQ FAQ
Executive Education
Home Home
About us About us
Management topics Management topics
Open Programs Open Programs
Custom Programs Custom Programs
Events/News Events/News
Contacts Contacts
HEC Online
Overview Overview
Executive programs Executive programs
MOOCs MOOCs
Summer Programs Summer Programs
Youth programs Youth programs
Instant

Scientific Research: Should Negative Results Be Published?

Decision Sciences
Published on:

Many call for a systematic publication of scientific negative results in order to make the production of scientific knowledge more efficient. Raphaël Lévy, Assistant Professor in the Economics and Decision Sciences Department, explains why such dissemination of knowledge may actually be beneficial, but also points to potential undesired consequences.

old man writing stokkete-AdobeStock_cover

©stokkete on Adobe Stock

A major concern for inventors, scientists and researchers in general is to establish the primacy of their discoveries by being the first to disclose an advance in knowledge, via either a scientific publication or a patent. This race tends to accelerate innovation, at the cost of possible adverse effects on quality.

The dilemma: quality versus speed

During the development phase, researchers and inventors face a dilemma. On the one hand, a longer maturation increases the chance of detecting an anomaly which reveals that their project is a poor lead that should be dropped (1). On the other hand, the fear of being overtaken by a rival may hasten publication or marketing decisions. The tension between these two competing forces (more learning versus the risk of being preempted) is characteristic of any innovation process, and the timing strategy ultimately chosen by scientists determines the final quality of their innovation.

 

The tension between a longer maturation and the risk of being preempted is characteristic of any innovation process.

 

The issue of the publication of negative results in science

In a world where the quality of scientific publications is increasingly challenged, the public debate has focused on the mechanisms that foster high-quality research. The question of whether or not to publish negative results, in particular, has received much attention. Recent studies show that academic journals have a marked bias for positive results (e.g., results that are significant in a statistical sense) (2).

However, the fact that negative results (for instance, results which, conversely, have insufficient statistical significance) are not made public is potentially harmful in two respects. First, a negative result is a valuable source of information for other researchers. For example, if two researchers are trying to prove the same theory, and one of them finds a counter-example that proves the theory wrong, the other researcher could benefit from knowing this information, which could spare him the cost of pursuing the same direction in vain. Second, the non-publication of negative results creates a selection bias that produces a distorted overall picture of the state of the art (3).

scientifics
©kasto on Adobe Stock

This is why many experts have recently spoken out in support of the systematic publication of negative results. For instance, the World Health Organization states that “negative and inconclusive as well as positive results must be published or otherwise made publicly available" (4). In line with this trend, several academic journals have decided to give greater publicity to negative results, either directly by promoting their publication or indirectly by requiring researchers to register their experimental projects in advance (5). 

The publication of negative results does not necessarily increase the quality of publications

What impact should we expect from such a systematic disclosure of negative results on the quality of publications, in particular in a highly competitive research environment? We address this question in an on-going project with Catherine Bobtcheff (Paris School of Economics) and Thomas Mariotti (Toulouse School of Economics), where we model the choice made by a researcher of the duration of the learning phase in two distinct scenarios. 

In the first scenario, negative results are systematically published and thus immediately made public: since the information is shared by the entire research community, each researcher learns faster than otherwise, hence an incentive to shorten the learning phase all the larger as competition is strong.

In the second scenario, negative results are never made public, and researchers respond to this by waiting longer to learn more about their projects. Indeed, they recognize that being the first to publish is now potentially bad news since a rival may have obtained a negative result and kept it private. To take up the earlier example, if a researcher who suspects that a competitor is trying to prove the same theory does not see the latter publish anything, he must take into account the possibility that the rival has found a counter-example that invalidates the theory, without this "negative result" being made public. This phenomenon well-known from the analysis of auctions is called the "winner's curse". It encourages researchers to prolong the learning period to enhance their confidence in their results.

Comparing the merits of these two scenarios highlights two opposite effects that the systematic publication of negative results has on quality. The learning process in the maturation phase is faster when the results are public (everyone benefits from more information), which is good for quality. At the same time, researchers adjust their timing strategies by shortening the learning period, which is detrimental to quality(6).

 

These preliminary findings already demonstrate that one should be careful before endorsing the systematic publication of negative results as a way to foster the quality of publications, especially in a highly competitive research environment.

 

By quantifying these two effects, our analysis shows that, when competition is strong, the second effect may be stronger, hence a lower average quality of publications when negative results are public! These preliminary findings already demonstrate that one should be careful before endorsing the systematic publication of negative results as a way to foster the quality of publications, especially in a highly competitive research environment.

 

(1) During this “learning phase”, a pharmaceutical firm typically performs additional tests on its new molecule to rule out the risks of side effects, or researchers carry out robustness checks on their results.
(2) See Fanelli, D. (2012), "Negative Results are Disappearing from Most Disciplines and Countries", Scientometrics, 90, 891-904.
(3) For example, a study by Turner et al. (2008), "Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy”, The New England Journal of Medicine, 358, 252-260, shows that 94% of the studies on anti-depressants considered by the Food and Drug Administration (FDA) to approve marketing decisions exhibit positive results. But the percentage of positive results falls to 51% when unpublished studies not taken into account by the FDA are included in the panel.
(4) See the results here.
(5) For example, the Journal of Development Economics gives to authors the opportunity to have their potential empirical projects reviewed and approved for publication before the results are known.
(6) For instance, if a drug actually has important side effects, the firm is less likely to find out because the testing phase is shorter, so the risk that the drug is inappropriately brought to the market increases.

 

Related content on Decision Sciences

Decision Sciences

Risking the future? How Delayed Consequences Can Bias the Perception of Risk

By Emmanuel Kemel

Brian Hill GREGHEC
Brian Hill
CNRS Research Professor
Emmanuel Kemel HEC professor
Emmanuel Kemel
CNRS Research Professor
Economics

How Much to Reveal to Persuade a Decision Maker?

By Tristan Tomala, Marie Laclau, Frédéric Koessler

Subscribe button for Knowledhe@HEC newsletter

Newsletter knowledge

A monthly brief in your email box and 3 issues of the book per year.

follow us

Insights @HECParis School of #Management

Follow Us

Support Research

Our articles are produced thanks to our reader's support