Image: Grace Gay for Research Professional News
Author of 2023 Nature paper says analysis and conclusions hold despite peers’ critique
The co-author of a landmark paper on slowing rates of disruption in science has told Research Professional News he stands by its conclusions, despite criticism from other researchers. Those researchers, meanwhile, have maintained their criticisms despite the authors’ pushback.
The 2023 paper—titled Papers and patents are becoming less disruptive over time—by Michael Park, Erin Leahey and Russell Funk was published in the journal Nature and has been cited hundreds of times. It reports an analysis of citations of millions of research papers and patents, which found such works have become “increasingly less likely to break with the past in ways that push science and technology in new directions”.
This led Funk and colleagues to conclude there has been “a fundamental shift in the nature of science and technology”.
Criticism and pushback
On 12 August, Nature published a criticism of the analysis by five researchers, including corresponding authors Vincent Holst and Vincent Ginis of the Free University of Brussels (VUB).
In a press release, VUB said the criticism “shows that [the original] conclusion was largely based on errors in the underlying data”.
It quoted Ginis as saying: “When we correct these errors, almost the entire decline reported in the original study disappears. That does not mean we can now conclude that science is actually becoming more groundbreaking. It does mean, however, that the earlier conclusion does not provide a reliable basis for such a claim.”
Ginis told Research Professional News that after applying a “correction” to the analysis, “the effect sizes are small…and very sensitive to modelling choices”. He added: “We doubt the original work without the error would have been accepted and have the impact it had over the last years.”
Nature simultaneously published a response from Funk and colleagues, which says there were “severe quality issues” in the data used by Holst, Ginis and colleagues. The response also says Funk and his colleagues conducted additional analyses using the methods of Holst and his colleagues, and produced a finding in line with their original one. Both teams criticised each other over the availability of some of their research materials.
Speaking to RPN, Funk said: “Do I stand by the original work? Yes, absolutely, and even more so than ever before.”
He added: “Since our 2023 paper, declining disruptiveness has been documented in nearly 100 independent studies.”
Nature’s processes
VUB and Ginis also criticised the approach Nature took to their contribution. They said the Belgian researchers first contacted the journal in early 2023 with their claims but Nature’s response, in the form of a Matters Arising piece, was not published until three years later.
“In the meantime, the original study continued to exert influence worldwide,” said the VUB. “Whilst the criticism was under review, Nature itself devoted another editorial and a news article to the original paper.”
Ginis said: “The original paper went from submission to cover story in at most 17 months. Our correction of a factual mistake took 32. During that time no notice of any kind appeared on the article, while it grew from 185 to over 1,200 citations and was cited in a US congressional hearing.”
“If science is to be self-correcting, substantiated critiques should be handled faster than original research, not twice as slowly, and articles under substantiated critique should carry a visible flag while the exchange is under review.”
Nature response
A spokesperson for Nature said post-publication commentary plays an “important role” in advancing scientific discourse and that Nature’s Matters Arising publications support this.
They added that Matters Arising articles are sent to the authors of the papers they relate to and that both can then be sent for peer review before publication, which they said was the case here. The length of time this all takes, they said, “varies depending on the complexity of the issues being considered and the review process required”.
“In this case, we published both the Matters Arising article and a reply from the authors of the original paper,” they added. “The publication of both articles ensures that readers have access to the full scientific exchange and can evaluate the arguments presented.”