Background and Overview

I am an economist and study poverty and inequality and the role of the social safety net in family and child well-being. I want to start by positioning myself and my background a little bit, as it relates to the theme in these articles. Over the past decade or more, I have engaged in a variety of professional service activities that are linked to reproducibility and open science. First, I am currently a member of the Committee on National Statistics at the National Academies of Science, Engineering and Medicine, where we discuss a range of issues around data, replication, and reproducibility. I was a member of the Commission on Evidence Based Policymaking, and there we made recommendations around administrative data and linkages balancing access and privacy. I mention these experiences because in economics there has been tremendous growth in the use of confidential, proprietary, or administrative data and there are significant challenges in open science goals around these data. Second, I spent a decade as a co-editor of journals under the American Economic Association (AEA), first at the American Economic Journal (AEJ): Economic Policy and subsequently at the American Economic Review (AER), the flagship journal of the association. While a co-editor at the American Economic Review, and also a member of the AEA Executive Committee, I co-chaired the search committee for an AEA data editor.

This article provides a status update on reproducibility undertaken by the American Economic Association. This includes a particular emphasis on quantitative research. And a chronology of the role the American Economic Association and its journals have taken in this progress.

Main Thoughts

One theme of the articles featured in this Reinforcing Reproducibility and Replicability special theme is how differences in support for reproducibility exist across the social sciences. In this column, I focus on the status of reproducibility in economics and what might make it different from other social sciences.

The (Possibly Unique) Role of the American Economic Association

The AEA has a lot of ‘market power’ because it controls a large share of the high-quality journal landscape. The AEA journals currently include the American Economic Review (AER), AER: Insights, and American Economic Journals (there are four of these), as well as the Journal of Economic Literature and the Journal of Economic Perspectives. The AEA started with the American Economic Review and is one of the ‘top 5’ general interest publications in economics. The AER became so clogged with quality submissions and accepted papers (with 2-year waits between acceptance and publication) that the AEA expanded the number of issues that it produced per year. This led the AER to publish a disproportionate share of the articles in the economics top 5 journals.

In light of the success of the AER and the strong budget situation at the association, the AEA decided to expand and add journals to its portfolio. We added AER: Insights, featuring short-format articles with quick turnaround, in an attempt to compete with publications in Science and Proceedings of the National Academy of Sciences (PNAS). We also added four top field journals to try to occupy the space just below the top 5, perhaps wresting some of the market from for-profit journals, such as those published by Elsevier. This became the four American Economic Journals—AEJ: Applied Economics, AEJ: Economic Policy, AEJ: Microeconomics, and AEJ: Macroeconomics.

So, if you put this all together, there is a lot of journal space (and highly ranked journal space) that the AEA runs.

With this ’market power,’ the association made a series of moves in the area of open science. This approach came about through both top-down and bottom-up mechanisms. There was definitely interest among the AEA Executive Committee, including the lead editors of the AER, first Penny Goldberg and followed by Esther Duflo. But there was also a perspective that percolated up from the membership. This may have come in part in response to the replication crises in other disciplines. Additionally, many of us engaged in training graduate students who desired data to get practice and skills.

Overall, I think the AEA views itself as having the potential to take actions that set standards that are adopted more broadly in the profession. (I hope the association takes the same approach to dealing with sexual harassment in economics.)

AEA Actions

So, what did the AEA do?

First, it set up the AEA RCT Registry. As stated on the website: “Randomized Controlled Trials (RCTs) are widely used in various fields of economics and other social sciences. As they become more numerous, a central registry on which trials are on-going or complete (or withdrawn) becomes important for various reasons: as a source of results for meta-analysis; as a one-stop resource to find out about available survey instruments and data. Because existing registries are not well suited to the need for social sciences, in April 2012, the AEA executive committee decided to establish such a registry for economics and other social sciences.”

Second, and more pertinent to this conversation, the AER adopted a requirement to post data and code for all accepted papers. It initially was very much on the honor system but it was felt that this was a good first step. There were some staff who worked on confirming that files were uploaded, but not much beyond that.

After some time, there was an evaluation of how this was going. The answer was, not so well. Often times, the data and code would be incomplete. Documentation would be insufficient. It would be difficult for reuse, and it was not replicable. There was a recognition that you need skilled leadership and staff for this to work.

This led, in 2017, to the next stage of open science at the AEA. We recognized that we had the budget to do more. The AEA compensates all journal editors a pretty fair wage for the work. And the view was perhaps we should take the next step to hire someone who would work with the journal editors as well as with the AEA Executive Committee to create a more robust system for reproducibility. There was a discussion about whether or not we wanted this individual to be a practicing academic who has skills in this area, or whether we just needed to build up more staffing. We reached out to folks in other disciplines to find out what they were doing. We liked the model at the American Statistical Association, which had appointed an academic researcher who also had skills in open science.

In summer 2017 the AEA initiated a search for this new position, which we called the AEA data editor. From the job ad, the data editor would “Collaborate with journal editors and executive committee; design and oversee implementation of strategy for archiving & curating data, promoting reproducible research.” In short, we needed an architect of the new system as well as a manager and implementer.  We were lucky to attract Lars Vilhuber to the position. He is a practicing academic researcher who also has the skills in open science, and could work at the frontier.  His work began in 2018.

Conclusion

I will conclude by providing some thoughts about where the gaps are and where we need to continue to make progress.

First, we need to continue to revisit the staffing of the data editor’s operation. Are the resources sufficient for the needed work?

Second, we need to devise approaches to deal with confidential data. It is increasingly common for AEA journal papers to use proprietary or confidential data. Examples include government administrative data, data from firms, and so on. Researchers typically do not have the right to post this data. Therefore, replication and reproducibility have to get over a very large hurdle with these data.

Third, we need to build on the success of preregistration of RCTs. For example, should the journal require preregistration? Should this be expanded to preregistration of quasi-experimental papers? The use of pre-analysis plans is uncommon in quasi-experimental research and may be controversial. The expansion of these requirements may raise equity concerns—researchers outside top departments may not have the resources to vet the projects prior to registration. Should the AEA create an RCT editor to check compliance with the preregistration plan? These are issues of ongoing conversation and there is a place for the AEA to continue its leadership by advancing these ideas.

Finally, we need do more analysis and landscaping to investigate where the gaps are in the economics discipline more broadly, and what role the AEA can take to continue to move us toward further progress in open science. These efforts at the AEA, particularly the data editor position, are a resource-intensive endeavor. It is unclear whether they can be expanded more widely across journals that have less budget capacity.

Disclosure Statement

The author received financial payments from the American Economic Association while serving as an Editor for the American Economic Journal: Economic Policy (2007–2010) and for the American Economic Review (2011–2016).

This article is an expanded version of remarks given in the Conference on Reproducibility and Replicability in Economics and the Social Sciences (CRRESS) webinar series, which is funded by National Science Foundation Grant #2217493.

©2023 Hilary Hoynes. This article is licensed under a Creative Commons Attribution (CC BY 4.0) International license, except where otherwise indicated with respect to particular material included in the article.