I am grateful for the opportunity to comment on the timely manuscript by my distinguished colleague, David Donoho (2024).

Donoho argues that we are witnessing a fundamental shift in science, and perhaps society more broadly, due to the rapid advancements in data science and computational research. He believes this shift is not accurately captured by the popular concept of an “AI singularity.’’ Instead, he focuses on the idea of “frictionless reproducibility” (FR). The practical implication being that with FR, almost anyone can seamlessly reproduce scientific results with ease, thereby removing barriers, expediting validation, and facilitating building upon past work. It is argued that this transition drastically accelerates the spread of ideas, changes how researchers collaborate, and could ultimately erase the memory of “pre-FR” methodologies as their inefficiencies become starkly apparent.

Donoho paints a compelling, if arguably utopian, picture of a scientific landscape transformed by frictionless computational omnipotence. As a practitioner of conventional “pre-FR” approaches and a more recent machine learning practitioner-convert, I find myself both exhilarated by the potential Donoho outlines and troubled by a certain undercurrent of technological determinism in the article. Let me elaborate further.

The Devil Is In The Details

The pursuit of frictionless reproducibility is an undeniably worthy goal–a cornerstone of robust science. However, Donoho’s framing perhaps downplays the sheer complexity of ensuring true replicability in practice. Seemingly minor deviations in software libraries (e.g., PyTorch vs. TensorFlow), hardware configurations (e.g., numerical precision, GPU, CPU, TPU), or even the temporal order in which large data sets are processed can introduce nontrivial variance that is hard to control post hoc (Stodden & Miguez, 2014). These realities complicate the idyllic vision of effortlessly replaying any scientific computation from the past. Further, data sets in fields dealing with human systems, from social sciences to medical research, are notoriously dynamic (Sculley et al., 2015), with shifting, often tightening concerns around privacy rights. Distributions shift over time as well; behaviors evolve, and what was replicable yesterday might hold less relevance tomorrow. Thus, FR within Donoho’s paradigm needs to be tempered with an appreciation for context and change over time. I might offer a more practical framing: low-friction reproducibility.

The Human Factor

There is a risk in the article’s narrative that computational advancement may overshadow the enduring centrality of human expertise and judgment. Defining the very questions data science tools might answer remains irreducibly human. In fairness, one might argue this view (Holzinger, 2016) is now outdated; but if we stipulate that the goals of scientific progress are not just understanding the world, but principally, to improve the human condition, then neither machine learning nor data science are likely, on their own, to replace the deep domain knowledge and intuition essential for identifying truly impactful research directions.

While ever more powerful machine learning models achieve startling levels of predictive accuracy, they still often function as ’black boxes’ with limited explanatory power. This necessitates researchers capable of critically interpreting results and building trustworthy AI systems (Rudin, 2019).

A Hybrid Future

Donoho’s concept of a stark divide between a ‘backward’ “pre-FR” era and an enlightened post-FR scientific age is perhaps a little too black and white. It seems far more likely that many fields will experience a long transitional period, retaining proven methodological foundations, while selectively integrating computational innovations as needed, and when they are deemed sufficiently mature and reliable (Brynjolfsson & McAfee, 2017). Science has always advanced by leveraging broadly defined new tools (everything from computational paradigms, new theories, and even physical sensors); computation represents a spectacular next chapter, not a wholesale rewrite.

The Ethical Imperative

Donoho wisely avoids the alarmist tone that often marks conversations around advanced AI and its potential dangers. His central thesis–—that computation and data science herald a profound shift in how research is conducted—is unassailable. However, with this acceleration comes a heightened responsibility. Alongside celebrating computational efficiency, every adherent field would be wise to invest equally in critical analysis of how post-FR paradigms could perpetuate data set biases, amplify errors at scale, and sideline vital ethical considerations in the pursuit of faster, lower friction, and better science (Mittelstadt et al., 2016).

Closing

In closing, permit me to offer a personal take on the state of affairs. Those of us old enough to have worked in ‘conventional’ science and the new era of ‘large scale, low-friction’ science find ourselves at once awed, elated, and sometimes a little hurt to see scale alone resolve problems we have diligently worked on for years. In an often cited post Sutton (2019) coined this the “bitter lesson.” But it is more apt to say it is the bittersweet lesson—as we are immensely privileged to be living through a period of profound transformation—one that will be recorded in the annals of history.

Disclosure Statement

Peyman Milanfar has no financial or non-financial disclosures to share for this article.

References

Donoho, D. (2024). Data science at the singularity. Harvard Data Science Review, 6(1). https://doi.org/10.1162/99608f92.b91339ef

Brynjolfsson, E., & McAfee, A. (2017, July 18). The business of artificial intelligence. Harvard Business Review. https://hbr.org/2017/07/the-business-of-artificial-intelligence

Holzinger, A. (2016). Interactive machine learning for health informatics: When do we need the human-in-the-loop? Brain Informatics, 3(2), 119–131. https://doi.org/10.1007/s40708-016-0042-6

Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data and Society, 3(2). https://doi.org/10.1177/2053951716679679

Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215. https://doi.org/10.1038/s42256-019-0048-x

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J., Dennison, D. (2015). Hidden technical debt in machine learning systems. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 28 (NIPS 2015) (pp. 2503–2511). IEEE, NeurIPS. https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html

Stodden, V., & Miguez, S. (2014). Best practices for computational science: Software infrastructure and environments for reproducible and extensible research. Journal of Open Research Software, 2(1), Article e21. https://doi.org/10.5334/jors.ay

Sutton R., (2019) The bitter lesson.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html

©2024 Peyman Milanfar. This article is licensed under a Creative Commons Attribution (CC BY 4.0) International license, except where otherwise indicated with respect to particular material included in the article.