Subscribe now

A trending debate in research circles of late is whether scientific discoveries are getting harder and slower to come by. To be clear, when researchers talk of the decline in scientific breakthrough, they often mean one of two things: (i) a productivity problem (larger inputs producing smaller outputs), i.e. research now requires more teams, time, resources and funding, yet yields fewer high impact breakthroughs compared to previous decades (ii) a scale or impact problem, i.e. the discoveries of recent decades have been more incremental to rather than disrupting the frontier of knowledge. The comparison also often benchmarks current science against work dating back many decades, as far back as the 1920s. One often-cited example of this decline is the slower, more challenging, and more resource-intensive pace of discovering new drugs. Research shows that despite increasing R&D spending in the pharmaceutical industry, the number of new drugs per billion dollars has continually shrunk since the 1950s[1].

While there may be variations in the level of R&D investment by sector and region (for instance, R&D investment in the British pharmaceutical industry has significantly declined since 2018), the global average R&D investment has seen an upward trajectory, at least in the last 30 years[2]. Data in the UK shows a similar trend, with overall private sector R&D spending and R&D employment expenditure (cost of R&D personnel - researchers, engineers, technicians, etc) steadily rising for the past 30 years. In the U.S, there has been a growth in funding and other metrics for science and engineering research in the past 100 years.

UK Private Sector R&D, constant (£ millions, 2020=100)

Source: Author, based on Office for National Statistics [ONS] (2021).

Trends in research funding, PhDs and papers in US’ science and engineering fields

Source: Collison and Nielsen (2018).

Yet these investments barely correspond to breakthroughs in research impact. A paper by Bloom et al. (2020) examines the relationship between R&D inputs and research productivity. The authors match data on health, agriculture, and semiconductor outputs — such as new molecular discoveries, years of life saved from specific diseases, crop yields, etc — with number of publications, clinical trials, researchers, and other research variables. Results show that despite the significant expansion of research inputs, research productivity in the US is falling by more than 50 per cent every 13 years.

The key puzzle, then, is: if R&D, the fundamental engine of research, is constantly running, why have we reached fewer scientific milestones than in earlier decades? There’s no single comprehensive answer to this question; in fact, the idea that scientific discovery is slowing down is contested. Worth mentioning; measuring scientific progress is conceptually difficult, partly because it is hard to estimate the importance of some discoveries over others.

Scientific progress through a publication lens

One important metric researchers often turn to when comparing the scale of scientific progress over time is research publications. Several metrics are relevant here, such as the ratio of researchers per paper publication, share of papers that are patented, or the proportion of recent publications that inform Nobel prize-winning work. One caveat on that statistical metrics is that they do not always give the complete picture of research productivity. For example, recent increase in the number of scientists and research papers might suggest a growth in science, whereas more new papers could also mean the increasing fragmentation of science into too many new subsets, which increases competition among new ideas and creates a deficit of attention for “promising” ones.

There is a ton of studies using publication metrics to measure scientific advancements, many of which are complex to work through (Matt Clancy offers a detailed discussion of these in his Substack, Science is getting harder). I provide a quick overview of below.

Wang and Barabasi (2021) examined more than 53 million authors and 90 million scientific publications over 100 years (1900-2000). They found a steady increase in the number of authors and papers, particularly between 1950 and 2000. Yet, while the average number of yearly publications per scientist was relatively stable throughout this period (and even increased in the last 15 years), there has been a decline in the number of papers per single author and a rise in the number of authors per publication. Of course, this may mean better collaborations among scientists. It is also probable, as Matt Clancy argues, that the number of papers has only grown for existing discoveries, without a proportional increase in discoveries per se.

Wu et al. (2019) offer a slightly more convincing explanation for this phenomenon, using web-scrapped data on scientific papers, software products, and patents spanning 60 years (1954-2014). They suggest that larger teams of scientists tend to build on existing discoveries, but it is the smaller groups that create new disruptive innovations in science and technology. Placing this alongside Bloom et al.’s thesis on the expansion of research teams in science, we might draw a pattern between the paucity of smaller teams and the lack of “high-risk innovation” in today’s science.

There is also the argument that recent scientific papers are less likely to, within 20 years of their publication[3], receive notable awards, such as the Nobel Prize, compared to works published in the early to mid-1900s. This suggests that recent outputs in science might be less phenomenal than works of the early twentieth century. Indeed, as Patrick Collison and Michael Nielsen (2018) show in their survey of the world’s top academics in the physical and biological sciences fields, many contemporary scientists consider works of the early 1900s more important to their fields than those of the later part of the century.

Closely related to this measure is the probability of a recent paper becoming top cited – after all, recent works are more likely to be cited now than older works due to their relevance to contemporary research. Cui et al. (2022) note that, since the 1950s, there has been a decline in the share of citations made to recently published work in the physical and natural sciences, biological sciences, and social sciences. Does this mean that works of the past have a much stronger influence on science today than contemporary works? Cui et al (2022)’s explanation is that science today is composed of an ageing workforce, and older scientists would more likely cite the [older] works they are familiar with. But an even more important point emerges from here: that the mean age of scientific innovation has been on the rise in the last few decades, with the most significant innovations of our time being mostly concentrated among older scientists.

Science and total factor productivity (TFP) decline

As I noted in the introduction, the challenge with contemporary scientific discovery is both of productivity and scale. But the decline in productivity today is everywhere we look, including science; and one can look at it either as a problem that’s not uniquely science or one that slow science contributes to. In the last 30 years, there has been a decline in total factor productivity (a measure economic output relative to the labour, capital and other inputs required to make it happen) in many advanced economies. This is the case in the US, which currently leads the frontier of scientific research. While US firms have overall maintained a relatively stable productivity level compared to its peers, this is driven by large contributions from the ICT industry. Productivity in the US manufacturing sector has been on a slow decline since the late 2000s.

US Total factor productivity growth vs growth in number of researchers

Source: Bloom et al. (2020)

But it matters how we slice this issue. If we say the slow pace of science today is a driver of the productivity slowdown, then we might be suggesting that science has an innate problem that is not simply down to the “input-to-output” levels diagnosis in TFP measures— which is how many economists conceptualised the scientific pace problem. Conversely, looking at science’s productivity problem as a product of the current generation of humans being simply less efficient at producing things allows us to see things differently.

The point I’m making by drawing a link between science and TFP measures is that our ‘scientific productivity’ problem could be a measurement issue. Generally, economic production now requires more inputs than in the past due to several factors — increased complexity of modern goods, greater regulatory and compliance burden, supply chain complexity, higher consumption (e.g greater access and demand for higher education means more doctoral positions and individual incentives to publish, even when the modalities of group discoveries stay the same), and inclusive reforms that have expanded the science workfroce. For research, the accumulation of knowledge from the past makes it possible for more people to build on existing work, which means every new discovery opens up new areas of study and creates more need for more researchers. Each of these inputs will continue to expand despite the efficiency gains from technology.

Hence, a cost-based measure inevitably paints a picture of inefficiency in science, regardless of whether new inventions are being made. Of course, my point here is not to say we should scale down inputs (that would give us much less results than we are seeing today), but that our estimates need to account for the inevitable rise in the aggregate cost and number of inputs required for economic production (including in science). Understandably, economists default to cost-based measures because of the inherent difficulty in quantitatively estimating the importance of certain discovery over others. Our view of science must shift from simply measuring input to output ratio to considering the depth of research outputs; for example, whether an increased number of researchers produce more clinically effective drugs or curative drugs for terminal diseases rather than simply producing more drugs with fewer dollars.

In this sense, scale becomes the most consequential of the two dimensions of scientific stagnation today.

Can AI restore the pace of science?

There is an alternative view that the pace of contemporary science cannot but be slower, since earlier scientists have already accomplished the ‘easy targets’, raising the standards for later discoveries. Consider a fruit-picking scenario, where the probability of being able to pluck a new fruit declines over time as the low-hanging fruits get picked out. The thinking here is, just as high-hanging fruits are harder to get to, contemporary scientists should have a harder time making new discoveries and fewer potentially consequential breakthroughs.

But what if better tools at our disposal — fruit pickers, ladders, and stools — mean that we can reach fruits that are higher up, faster? Researchers today have the advantage of more innovative tools, technologies, organisational processes, and theories that should make research more efficient.

The missing link is: how do we convert these new inputs into consequential scientific breakthroughs? Let’s return to the fruit-plucking scenario: since plucking low-hanging fruits requires much less skill than the higher ones, picking the high-hanging fruits today will not only require better tools at our disposal but better ways of doing things. In the low-hanging fruit scenario, we had acquired a certain typology for plucking the fruits, which may not be compatible with our new tool suite. Said differently, given that we have more inputs to manage (fruit pickers, ladders, stools, etc., compared to bare hands), plucking the high-hanging fruits as fast as we did the low-hanging ones would require a different approach. If discoveries have truly become smaller as a result of the idea generation process becoming more complex and requiring more sophisticated inputs over time, then condensing the idea generation process is one way to do things differently.

Here comes artificial intelligence. One way to think about the potential of AI to speed up science is to return to the earlier point about contemporary research taking more than 20 years to be awarded a Nobel Prize. The 2024 Nobel Prize in Chemistry, awarded to Demis Hassabis, John Jumper and David Baker based on work on computational protein design that was mostly done in the preceding five years, breaks the 30-year average for the field. The invention of AlphaFold by Hassabis and colleagues condensed the laborious process of predicting protein structures through amino acid sequencing, which relied heavily on experimental databases and physics-based energy calculations that took several months, into deep learning-based methods that take a few hours to a day.

The impact of that work in effect is speeding up the idea production process in science. Since its release, AlphaFold has predicted more than 200 million protein structures, which is nearly every known protein in science. In reality, protein structure prediction (sped up process) is not an end goal in itself; the compounded effect of that innovation are, namely accelerated drug discovery, better understanding of disease mechanisms, and engineering of new enzymes.

We must thus think about AI’s potential on science as the “invention of (new) method(s) of invention”. The ability to now automate several stages of the R&D process makes knowledge synthesis faster and less laborious, freeing up time for scientists to do more theory building and ideation. We can think about this in two ways. One is accelerating access to knowledge, e.g. researchers can more easily access the most relevant information to their work without manually sifting through an overwhelming number of publications, conduct meta-analysis for clinical queries substantially faster, and generate novel hypotheses for experimentation. The second is pattern recognition. Research does not have a publication problem; already, we have more than double the number of papers in any given field today than 50 years prior and the number of unique topics being published within a given field has exponentially increased. What is needed to bring about scientific discovery is establishing connections between disparate fields in science. New inventions would not proceed strictly from a single field or even combination of two fields— that way of doing science is dead. Modern problems are more multidisciplinary and would require more collaboration, something researchers are already doing well. However, the outputs of these collaborations must be better synthesised to discover new patterns of inquiry.

There are structural limits to AI’s potential, though. To truly reap the benefits of AI, institutions must think differently about research culture. For example, it might be necessary to prioritise incentives to innovate over demands for higher publication volumes (i.e the stages of R&D that require more hands-on inputs should become more important than research publications, given that papers can now be published with less human inputs). Institutions must also streamline administrative responsibilities to free scientists from dead work, and encourage greater industry collaboration. Good science costs time and money — thankfully, AI is good at saving us those.

[1] In reality, this evidence is about the cost inefficiency rather than a slowdown in drug discovery per se. In fact, barring a momentary boom from the mid to late 1990s, the FDA approval rate for new drugs has relatively remained the same since the 1960s. And earlier research on British pharmaceuticals between 1971 and 2011 shows that overall, there has been a slight increase in the number of new drugs introduced per annum. However, novelty in clinical advantage offered by new drugs has remained the same since the 1990s and recall rates for released drugs have been higher.

[2] The World Bank’s global R&D data only dates to 1996.

[3] Generally, it takes years to observe the impact of scientific discoveries before awarding them prizes, and this can take up to 50 years for certain discoveries. See this interview with Ralf Petterson, former Chair of the Nobel Prize Committee for medicine.