scientific papers are my archnemesis.
don’t get me wrong. i like reading a good paper as much as the next gal. but papers are the major bottleneck blocking the scientific system from generating useful and transformative discoveries faster.
the paper has become synonymous with scientific output. you’re working ‘on a paper,’ not a discovery or even idea. and you ‘publish a paper,’ not completing your research. and what can you measure about a paper that tells you anything about the merit of the work? you could measure the number of words, or the range of colors used in its figures. it wouldn’t tell you much.
what you can measure is how often a paper gets mentioned by other papers. and it does tell you something. citations are an okay proxy for how relevant the scientific community thinks a given paper is: relevant enough for them to use it in their own work.
it seems natural that we would look towards citations as a quantifiable proxy for the quality of a paper. after all, science is so specialized that there’s really only a handful of people per researcher (their direct colleagues and competitors) who have enough depth of knowledge in the domain to judge the merit of their ideas and work. if they’re all using it in their work, there must be something to it. but this is conflating influence with quality, reference with endorsement.
the problem compounds because the paper is considered the canonical unit of scientific output. once scientific output is represented through papers, citations become the only scalable way to compare output at all. and once citations become the scalable proxy for paper quality, they get mistaken for a proxy for scientific merit.
because of this conflation of proxies, because we think of papers as scientific output, citations have become what really counts. they count towards reputation. they count towards job opportunities. and they count towards funding. being highly cited makes you a winner, and being a winner allows you to win more. and if getting cited makes you a winner, then of course getting cited becomes the goal.
goodhart told us where this story goes: when a measure becomes a target, it ceases to be a good measure.
once papers are treated as the unit of scientific output, citations become the scorecard for scientists. this incentivizes scientists to manufacture citations. direct forms of gaming are well documented but beside the point: even when nobody is cheating outright, citation-mania still warps behavior in arguably more insidious ways. scientists optimize for producing papers that are easy to cite. they have to — because what gets cited gets published gets funded gets jobs. review papers win. positive results win. broad, clean, confident stories win. this is what gets done. researchers have limited resources, and in their own interest are best advised to spend them on things that are likely to produce citations. what doesn’t get cited doesn’t get done, even if it’s crucial to a healthy system of discovery: sharing negative results, conducting replication studies, challenging the status quo in a field, pursuing topics that are not ‘hot.’
once careers, grants, and prestige depend on being cited, the system starts selecting for citeability: flatter the editor, write the review instead of the careful study, split the dataset, drop the outliers, chase positive results, and let flashy claims accumulate citations. the metric stops tracking scientific value and starts actively distorting it.
the obvious and honorable response is to devise new metrics. with new metrics, we could add new reward streams, incentivize some of the good science behavior that currently is not recognized, or disincentivize some of the shady stuff people do today in order to get cited. there’s a whole field of research dedicated to this. there’s challenges dedicated to this. all great.
but there’s a fundamental flaw in thinking here: most “innovation” around metrics revolves around better ways to measure papers. the paper has so deeply seeped into the collective mental model of what science is that it largely goes unquestioned.
read the call for submissions for the Metascience Novelty Indicators Challenge*:
The Metascience Novelty Indicators Challenge is a global initiative to develop and validate scalable indicators that can identify novelty in research. We are challenging academics, data scientists, researchers, and innovators worldwide to create tools that could reshape how we evaluate novelty and thereby support metascience progress.
The Metascience Novelty Indicators Challenge invites individuals or teams who can generate novelty scores for journal publications across a wide range of disciplines.
an initiative to develop indicators that can identify novelty in research that is asking teams to generate novelty scores for journal publications. you see what i mean?
scientific papers as the unit of scientific output have hamstrung the development of new and more accurate and diverse measures of scientific quality, locking incentive and reputation systems into relying on a crude proxy, diverting the attention of our brightest minds from doing what matters to doing what gets cited.
as long as we stick to papers as the ultimate unit of scientific output, nothing will change.
*this is just the most recent example i came across so it was top of mind. not a criticism of the challenge. i’m actually very excited to see the results.
Recommend this episode.
69 | Keith Yamamoto and the freedom to fail - Night science podcast
Keith talks about how prestige journals such as Cell started.
“ [my friend] Benjamin Lewin would come to meetings, he had a Mercedes convertible with several bottles of very nice wine in a trunk. And scientists got this figured out. And so they thought Lewin’s coming to the meeting. And so I want to make my talk really exciting to him. Because if I do, I can get my paper accepted in a day”.
….
“And so this prestige journals evolved not because they were publishing the best science, because they had created this aura of coolness around papers, around science that they considered to be hot. And the scientific community lapped it up.”
That’s why I left academia (30 years ago).