Column Editor’s Note: The quest for transparency, the support of scientific inquiry, is key for credible research, regardless of the discipline. We started the Reinforcing Reproducibility and Replicability column in Issue 5.3 with a heavy focus on economics. In this column, Kim Weeden explores why sociology might be slow to adopt the standards and practices of scientific transparency. She sees one potential cause in the stronger fragmentation and laissez-faire approaches in sociology, in particular between qualitative and quantitative approaches. But she also sees in those same causes the possibility of a stronger bottom-up diffusion of such practices. 

Keywords: open science, scientific transparency, replication, sociology, social science

Background

Sociology has lagged behind economics, political science, and psychology in its recognition of a replication crisis, adoption of scientific transparency practices, and participation in the large and interdisciplinary literature on reproducibility (Ferguson et al., 2023; Freese & Peterson, 2017; Moody et al., 2022). Although some individual researchers have adopted open science practices, they are still very much in the minority (Ferguson et al., 2023, especially Figure 1). Similarly, although it is increasingly common for top sociology journals to recommend that authors share data and code, very few mandate replication packages, explicitly welcome replication studies, or encourage preregistration of experiments. Indeed, of the 11 empirical sociology journals from which Ferguson et al. (2023) drew a sample of authors for their survey, only one currently requires authors to share data and code (with some exemptions), eight encourage it, and two do not mention it at all; only one encourages preregistration (author’s analysis).1 Even fewer journals, if any, have a process to verify data and code or to check preregistration sites, meaning compliance is on the honor system.

Notably, most efforts to nudge the field toward scientific transparency have been ‘bottom up.’ The American Sociological Association (ASA), the discipline’s primary professional and scholarly association and publisher of some of its top journals, is largely silent on the issue of open science. Although the ASA’s code of ethics (1999, section 13.05) includes a statement on data sharing and requires full disclosure of methods and findings, it is not clear how or if this code has ever been applied.2 A previous executive officer of the ASA issued a public statement opposing open access on the grounds that it would harm the ASA’s revenue stream (Hillsman, 2012).

Why has adoption of transparency standards in sociology been so slow and piecemeal? Is the absence of top-down leadership from the discipline’s professional association a standard story of the challenge of coordination, organizational inertia, and resource constraints? Or is there something about sociology as a field that has slowed its adoption of scientific transparency relative to economics, psychology, and political science?

I argue that both factors are at play. In the next section, I will focus on sociology’s internal structure as a field, and how this hampers scientific transparency. I will then turn to the question of why top-down initiatives have failed, and conclude with speculations about the future of scientific transparency in sociology.

Heterogeneity, Fragmentation, and Disagreement Over Scientific Transparency

As many others have observed, sociology is an unusually low-consensus, internally heterogenous field. Abbott (2001) notes that sociology is “irredeemably interstitial” (p. 6) and “the most general of the social sciences, or, less politely, the least defined” (p. 1). Citation data show that sociology articles have a lower density of citations to other sociology articles than economics, psychology, or law; this contributes to its disciplinary weak core, while also making sociology central in the social science co-citation network (Moody & Light, 2006).

The internal fragmentation of sociology emerges along a number of dimensions: epistemic, topical, and methodological. At the most basic level, sociologists disagree about disciplinary goals: Should sociology be a science that strives, however imperfectly, to identify and empirically evaluate hypotheses about general social processes? Is it a normative project in revealing not what is, but what should be? Or is it an interpretivist project more akin to the humanities than to economics, political science, or psychology?

Even within the ‘sociology is a science’ camp, sociologists are fragmented along methodological lines, sometimes within but also between a multitude of distinct and increasingly problem-oriented topical subfields (Schwemmer & Wieczorek, 2020). In the top journals, it is common to see papers using qualitative, ethnographic, historical-archival, experiments, simulations, network analyses, and computational science methods alongside those using the more common econometric approaches.

At the risk of oversimplifying, sociologists who mainly use qualitative methods have been more reluctant to embrace scientific transparency than those who mainly use quantitative methods, and the nature of their concerns differ. Among qualitative sociologists, some of these concerns focus on issues of feasibility (see, e.g., Tsai et al., 2016): for example, will data sharing undermine the necessary trust that develops between researchers and subjects? Will potential subjects who learn in the process of securing informed consent that their data may be shared be reluctant to participate in research, particularly on the sensitive topics that sociologists often study? Will they alter their behavior if they know that others in their community will have access to interview transcripts? How can the principles of data sharing be reconciled with the strong disciplinary norm of monopoly access to researcher-collected data, a monopoly that is arguably necessary to incentivize qualitative research?

More fundamentally, qualitative researchers disagree about the desirability of scientific transparency. The process of producing and analyzing qualitative data is often iterative rather than linear, it is inherently intersubjective, and it relies on nonverbal cues from subjects and the embedded experiences of researchers that cannot be captured in transcripts, field notes, or other data products (Tsai et al., 2016). Sharing qualitative data may thus not only lead to ‘failed’ replications, but to misinterpretation of data if they are taken out of context. More fundamentally, qualitative researchers are more likely to object to the assumption that reliability, replicability, and reproducibility are appropriate yardsticks for assessing research, given that qualitative research is often designed to be generative of theory rather than evaluative of hypotheses.

Some of these concerns could be circumvented by modifying transparency standards to be sensitive to qualitative research (for suggestions, see Tsai et al., 2016) or by exempting qualitative research from them. Even with exemptions for qualitative research, however, not all sociologists are on board. In some cases, resistance seems to reflect a fear that a disciplinary move toward scientific transparency would marginalize qualitative research. In other cases, it reflects the belief that sociology’s efforts to be a ‘real’ science are doomed to fail, and that the discipline would be better off embracing its humanistic roots.

In quantitative sociology, discourse about scientific transparency typically centers around the verifiability of published results, meaning whether the data and code produce the claimed results (Freese & Paterson, 2017). In this context, reluctance tends to emphasize practical issues, such as the lack of resources to do the labor of independent verification of data and code, or the legal, ethical, and normative constraints on sharing data that were purchased from a private company, obtained from a case study organization only under agreement that the data would not be shared (e.g., personnel records from a private organization, enrollment records from a university), or accessible through a special data license after undergoing a security review by a federal agency.

Barriers to other forms of scientific transparency, such as efforts to ensure quantitative analyses replicate with new data, are cultural and hence even stickier. Many reviewers in sociology still place great weight on novelty and ‘surprising’ results, a focus that devalues replication studies in job evaluations and exacerbates the file drawer problem. Although these issues are not unique to sociology, it is plausible that low consensus over evidentiary standards in sociology create greater reliance on ‘interestingness’ as a criterion.

My argument, then, is not that sociology’s comparatively slow adoption of scientific transparency is entirely due to its large share of qualitative research, at least compared to economics and psychology. Rather, sociology’s eclecticism tends to foster a live-and-let-live ethos between various factions within the field. This ethos may be adaptive, in the sense that it allows sociology to muddle along as a discipline despite its weak core and “bandit gangs of positivists, feminists, interactionists, and Marxists” (Abbott, 2001, p. 6). However, ‘live and let live’ can easily morph into ‘don’t ask, don’t tell’ what is under the hood of research.

Why Have Top-Down Initiatives Failed?

Scientific transparency in sociology has also been hampered by the absence of top-down leadership by the discipline’s core association. This is not for lack of trying. Cohen (2021) recounts an effort in which he used his position as an elected member of the ASA’s publications committee to propose that the ASA adopt the Open Science Badge system. That proposal failed. Sixteen months and two ad hoc committees later, the publications committee referred to the ASA’s elected governing body an alternative proposal that would require authors to declare in a footnote whether their data and code were available online and if not, why not. Council rejected the footnote proposal and, after another 4 months, sent it back down to the publications committee. By this time, the members who had pushed hardest for scientific transparency had given up or rotated off the committee.

Without insider information, any autopsy of the proposal’s death by committee is bound to be inconclusive. Still, it seems likely that ASA’s governance structure—committees of elected members on short-term service stints that make recommendations implemented by permanent staff—contributed. Although this governance structure is not unique to the ASA, it presumably interacts with some of the elected members’ (or professional staff’s) ambivalence or resistance to scientific transparency to produce inertia.

In light of sociology’s eclecticism, it is understandable and perhaps even desirable that ASA-led initiatives toward scientific transparency have failed. After all, the ASA ostensibly represents the interests of all sociologists, including those who are indifferent to or oppose scientific transparency. Even so, the ASA’s weak stand has likely slowed the diffusion of open science in the field.

What Is the Future of Scientific Transparency in Sociology?

If my diagnosis of sociology’s slow progress is correct, what are the prospects for the future? I see no signs pointing to a change in the ASA’s position any time soon. Indeed, the elected leadership of the ASA is heavily and increasingly weighted toward qualitative scholars, who on average are less likely to prioritize scientific transparency. Recent changes to the association’s bylaws have reduced the power of rank-and-file members to propose initiatives related to the association’s practices, including its journals’ editorial practices.

The more likely path to scientific transparency is through ‘bottom-up’ processes of social diffusion and social control. These processes may be facilitated by the same disciplinary fragmentation that hampers top-down efforts. Sociology’s centrality among social science co-citation networks (Moody & Light, 2006) implies that sociologists are especially likely to encounter advances in other fields, to publish in interdisciplinary or extra-disciplinary journals, and to collaborate with researchers in other fields. Bridges to other disciplines where scientific transparency is more established will help diffuse the practices within internal research and training networks.

A second path to change originates in individual journals. For example, Sociological Science, European Societies, and Demography recently began mandating replication packages for quantitative articles (with exemptions for articles using restricted access data), and Sociological Science requires experimental research to meet disclosure expectations and indicate whether the experiment was preregistered. Although it is a relatively new journal, Sociological Science has an impact factor in the top 5 or 6 among general interest sociology journals and a reputation for innovative editorial practices (e.g., all open access, 30-day turnaround times, no ‘revise and resubmits’). European Societies is the journal of the European Sociological Association. Demography is the flagship journal of the Population Association of America, and although it is not a sociology journal per se, many of its papers are by sociologists and Demography is highly regarded in quantitative sociology circles. These journals are not the first in sociology or sociology-adjacent fields to mandate replication packages, but they are the first, top general- interest journals to do so.

The upshot is that although sociology’s adoption of scientific transparency will probably never catch up to economics, political science, or psychology, thanks to its internal fragmentation and interstitial position between the social sciences and humanities, the more quantitative subfields may soon reach a tipping point where a large enough share of researchers and journals adopt transparency standards to establish them as a norm. Once established, this norm will hasten the diffusion of scientific transparency, even absent top-down action from the discipline’s professional association.

Acknowledgments

I thank Lars Vilhuber and the three anonymous reviewers for their very helpful comments on earlier versions of this essay. Thumbnail image courtesy of artist ‘geralt’ on pixabay.com.

Weeden’s original contribution grew out of the oral presentation in Session 5 of the NSF-supported Conference on Reproducibility and Replicability in Economics and the Social Sciences. You can find her presentation on Youtube (archived as Weeden et al., 2023).

The Conference on Reproducibility and Replicability in Economics and the Social Sciences (CRRESS) webinar series is funded by National Science Foundation Grant #2217493.

Disclosure Statement

I was a co-founding, volunteer editor of Sociological Science and one of its deputy editors until 2021. I still serve on the board and publication committee of the journal’s governing organization.

References

Abbott, A. (2001). Chaos of disciplines. University of Chicago Press.

American Sociological Association. (1999). Code of ethics. Retrieved October 14, 2023, from https://www.asanet.org/wp-content/uploads/savvy/images/asa/docs/pdf/CodeofEthics.pdf

Cohen, P. (2021, March 28). The American Sociological Association is collapsing and its organization a perpetual stagnation machine. Family Inequality. https://familyinequality.wordpress.com/2021/03/28/the-american-sociological-association-is-collapsing-and-its-organization-is-a-perpetual-stagnation-machine/

Ferguson, J., Littman, R., Christensen, G., Levy Paluck, E., Swanson, N., Wang, Z., Miguel, E., Birke, D., & Pazzuto, J.-H. (2023.) Survey of open science practices and attitudes in the social sciences. Nature Communications, 14, Article 5401. https://doi.org/10.1038/s41467-023-41111-1

Freese, J., & Peterson, D. (2017). Replication in social science. Annual Review of Sociology, 43, 147–165. https://doi.org/10.1146/annurev-soc-060116-053450

Hillsman, S. T. (2012, January 9). Letter in response to Office of Science and Technology Policy’s Federal Register Notice 2011-28623. Orgtheory . Retrieved October 14, 2023, from https://orgtheory.files.wordpress.com/2014/01/open_access_hillsman.pdf

Moody, J. W., & Light, R. (2006). The view from above: The evolving sociological landscape. The American Sociologist, 37, 67–86. https://doi.org/10.1007/s12108-006-1006-8

Moody, J. W., Keister, L. A., & Ramos, M. C. (2022). Reproducibility in the social sciences. Annual Review of Sociology, 48, 65–85. https://doi.org/10.1146/annurev-soc-090221-035954

Schwemmer, C., & Wieczorek, O. (2020.) The methodological divide of sociology: Evidence from two decades of journal publications. Sociology, 54(1), 3–21. https://doi.org/10.1177/0038038519853146

Tsai, A. C., Kohrt, B. A., Matthews, L. T., Betancourt, T. S., Lee, J. K., Papachristos, A. V., Weiser, S. D., & Dworkin, S. L. (2016.) Promises and pitfalls of data sharing in qualitative research. Social Science & Medicine, 169, 191–198. https://doi.org/10.1016/j.socscimed.2016.08.004

©2023 Kim A. Weeden. This article is licensed under a Creative Commons Attribution (CC BY 4.0) International license, except where otherwise indicated with respect to particular material included in the article.