Keywords: replication, errors, error control, testing, p-values

I would like to thank Junk and Lyons (2020) for beginning a discussion about replication in high-energy physics (HEP). Junk and Lyons ultimately argue that HEP learned its lessons the hard way through past failures and that other fields could learn from our procedures. They emphasize that experimental collaborations would risk their legacies were they to make a type-1 error in a search for new physics and outline the vigilance taken to avoid one, such as data blinding and a strict $5\sigma$ threshold.

The discussion, however, ignores an elephant in the room: There are regularly anomalies in searches for new physics that result in substantial scientific activity but don’t replicate with more data. For example, in 2015 ATLAS and CMS showed evidence for a new particle with a mass of about $750\,\text{GeV}$ that decayed into two photons (CERN, 2015). Whilst the statistical significance was never greater than $5\sigma$ (Aaboud et al., 2016; Khachatryan et al., 2016), the results motivated about 500 publications about the new particle, and countless special seminars and talks (Garisto, 2016). The effect did not replicate when the experimental teams analyzed a larger dataset about six months later (Aaboud et al., 2017; The CMS Collaboration, 2017). Although this was a particularly egregious example, experimental anomalies that garner considerable interest before vanishing are annual events (Garisto, 2020).

We are motivated to attempt to control the type-1 error rate because type-1 errors damage our credibility and lead to us squandering our time and resources on spurious effects. Whilst these non-replications aren’t strictly type-1 errors as the statistical significance didn’t reach the $5\sigma$ threshold and no discoveries were announced, we incur similar damaging consequences, so they cannot be ignored. I shall refer to these errors—substantial scientific activity including publicly doubting the null and speculating about new effects when the null was in fact true—as type-1${}^\prime$ errors. Whilst type-1 errors appear to be under control in HEP, type-1${}^\prime$ errors are rampant. In the following sections, I discuss these errors in the context of statistical practices at the Large Hadron Collider (LHC).

Evidence and Error Rates

Searches for new physics at the LHC are performed by comparing a p-value, $p$, against a pre-specified threshold, $\alpha$.

There are two common interpretations of this procedure (Hubbard & Bayarri, 2003):

1. Error theoretic (Neymaan & Pearson, 1933): By rejecting the null if $p < \alpha,$we ensure a long-run type-1 error rate of $\alpha$. The threshold $\alpha$ specified the desired type-1 error rate and the p-value was a means to achieving it.

2. Evidential (Fisher, 1925): The p-value is a measure of evidence of against the null hypothesis. The threshold $\alpha$ specified a desired level of evidence.

Even among adherents of p-values, the latter interpretation is considered unwarranted (Lakens, 2021), and it is almost never accompanied by a theoretical framework or justification, or a discussion of the desired and actual properties of $p$ as a measure of evidence.

Unfortunately, Junk and Lyons repeatedly implicitly switch from one to the other. Indeed, the authors (2020) interpret $p$ as a measure of evidence and $\alpha$ as a threshold in evidence, e.g., justifying $5\sigma$ by “extraordinary claims require extraordinary evidence” and stating that “[$3\sigma$] or greater constitutes ‘evidence’.” We know, however, that interpreted as a measure of evidence, $p$ is incoherent (Schervish, 1996; Wagenmakers, 2007) and usually overstates the evidence against the null (Berger & Sellke, 1987; Sellke et al., 2001). For example, there exists a famous bound (Sellke et al., 2001; Vovk, 1993) implying that under mild assumptions $p = 0.05$ corresponds to about $30\%$ posterior probability of the null. This was in fact the primary criticism in (Benjamin et al., 2017). Consequently, one factor in the prevalence of type-1${}^\prime$ errors may be that:

1. physicists interpret p-values as evidence (as do Junk and Lyons);

2. based on p-values, physicists overestimate the evidence for new effects;

3. substantial scientific activity on what turn out to be spurious effects

Unfortunately, p-values simply can’t give researchers (including Junk and Lyons) what they want—a measure of evidence—leading to wishful and misleading interpretations of $p$ as evidence (Cohen, 1994). This cannot be overcome by better statistical training; it is an inherent deficiency of p-values and no amount of education about them will imbue them with a coherent evidential meaning.

Controlling Errors

Controlling error rates depends critically on knowing the data collection and analysis plan—the intentions of the researchers and what statistical tests would be performed under what circumstances—and adjusting the p-value to reflect that. There are, however, an extraordinary number of tests performed by ATLAS, CMS and LHCb at the LHC and elsewhere. This already makes it challenging to interpret a p-value at all and undoubtedly contributes to the prevalence of type-1${}^\prime$ errors.

Junk and Lyons rightly celebrate the trend in HEP to publicly release datasets and tools for analyzing them. This, however, raises the specter of data dredging. Massive public datasets (CERN, 2020) combined with recent developments in machine learning (Kasieczka et al, 2021) could enable dredging at an unprecedented scale. We must think about what precautions we need to prevent misleading inferences being drawn in the future; e.g., pre-registration of planned analyses as a requisite to accessing otherwise open data. Other more radical proposals, to the problems here and elsewhere, include moving away from an error theoretic approach, or any approach based on p-values.

Disclosure Statement

Andrew Fowlie has no financial or non-financial disclosures to share for this article.

References

The ATLAS collaboration, Aaboud, M., G. Aad, Abbott, B., Abdallah, J., Abdinov, O., Abeloos, B., Aben, R., AbouZeid, O. S., Abraham, N. L., Abramowicz, H., Abreu, H., Abreu, R., Abulaiti, Y., Acharya, B. S., Adamczyk, L., Adams, D. L. Adelman, J., Adomeit, S., . . . Zwalinski, L. (2016). Search for resonances in diphoton events at $\sqrt s = 13$ TeV with the ATLAS detector. Journal of High Energy Physics, 2016(9), Article 1. https://doi.org/10.1007/JHEP09(2016)001

The ATLAS collaboration, Aaboud, M., G. Aad, Abbott, B., Abdallah, J., Abdinov, O., Abeloos, B., Aben, R., AbouZeid, O. S., Abraham, N. L., Abramowicz, H., Abreu, H., Abreu, R., Abulaiti, Y., Acharya, B. S., Adamczyk, L., Adams, D. L. Adelman, J., Adomeit, S., . . . Zwalinski, L. (2017). Search for new phenomena in high-mass diphoton final states using 37 fb^-1 of proton–proton collisions collected at $\sqrt s = 13$ TeV with the ATLAS detector. Physics Letters B, 775, 105–125. https://doi.org/10.1016/j.physletb.2017.10.039

Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., Cesarini, D., Chambers, C. D., Clyde, M., Cook, T. D., De Boeck, P., Dienes, Z., Dreber, A., Easwaran, K., Efferson, C., . . . Johnson V. E. (2017). Redefine statistical significance. Nature Human Behaviour, 2(1), 6–10. https://doi.org/10.1038/s41562-017-0189-z

Berger, J. O., & Sellke, T. (1987). Testing a point null hypothesis: The irreconcilability of p values and evidence. Journal of the American Statistical Association, 82(397), 112–122. https://doi.org/10.1080/01621459.1987.10478397

CERN. (2015). [ATLAS and cms physics results from run 2]. https://indico.cern.ch/event/442432/

CERN. (2020). [CERN announces new open data policy in support of open science]. https://home.cern/news/press-release/knowledge-sharing/cern-announces-new-open-data-policy-support-open-science

The CMS Collaboration. (2017). Search for high-mass diphoton resonances in protonproton collisions at 13 TeV and combination with 8 TeV search. Physics Letters B, 767, 147–170. https://doi.org/10.1016/j.physletb.2017.01.027

Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997–1003. https://doi.org/10.1037/0003-066X.49.12.997

Fisher, R. A. (1925). Statistical methods for research workers. Oliver & Boyd.

Garisto, D. (2020). The era of anomalies. Physics, 13, 79. https://physics.aps.org/articles/v13/79

Garisto, R. (2016). Editorial: Theorists react to the CERN 750 GeV Diphoton data. Physical Review Letters, 116(15). https://doi.org/10.1103/PhysRevLett.116.150001

Hubbard, R., & Bayarri, M. J. (2003). Confusion over measures of evidence (p’s) versus errors ($\alpha$’s) in classical statistical testing. American Statistics, 57(3), 171–178. https://doi.org/10.1198/0003130031856

Junk, T. R., & Lyons, L. (2020). Reproducibility and replication of experimental particle physics results. Harvard Data Science Review, 2(4). https://doi.org/10.1162/99608f92.250f995b

Kasieczka, G., Nachman, B., Shih, D., Amram, O., Andreassen, A., Benkendorfer, K., Bortolato, B., Brooijmans, G., Canelli, F., Collins, J. H., Dai, B., De Freitas, F. F., Dillon, B. M., Dinu, I.-M., Dong, Z., Donini, J., Duarte, J., Faroughy, D. A., Gonski, J., . . . Yunus, M. (2021). The LHC Olympics 2020: A community challenge for anomaly detection in high energy physics. arXiv. https://doi.org/10.48550/arXiv.2101.08320

Khachatryan, V., Sirunyan, A. M., Tumasyan, A., Adam, W., Asilar, E., Bergauer, T., Brandstetter, J., Brondolin, E., Dragicevic, M., Erö, J.,Flechl, M., Friedl, F., Frühwirth, R., Ghete, V. M., Hartl, C., Hörmann, N., Hrubec, J., Jeitler, M., König, A., . . . Woods, N. (2016). Search for Resonant Production of High-Mass Photon Pairs in Proton-Proton Collisions at $\sqrt s = 8$ and 13 TeV. Physical Review Letters, 117(5), Article 051802. https://doi.org/10.1103/PhysRevLett.117.051802

Lakens, D. (2021). The practical alternative to the p value is the correctly used p value. Perspectives on Psychological Science, 16(3), 639–648. https://doi.org/10.1177/1745691620958012

Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 231(694–706), 289–337. https://doi.org/10.1098/rsta.1933.0009

Schervish, M. J. (1996). P values: What they are and what they are not. American Statistics, 50(3), 203–206. https://doi.org/10.1080/00031305.1996.10474380

Sellke, T., Bayarri, M. J., & Berger, J. O. (2001). Calibration of p values for testing precise null hypotheses. American Statistics, 55(1), 62–71. https://doi.org/10.1198/000313001300339950

Vovk, V. G. (1993). A logic of probability, with application to the foundations of statistics. Journal of the Royal Statistical Society, B55(2), 317–341. https://doi.org/10.1111/j.2517-6161.1993.tb01904.x

Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779–804. https://doi.org/10.3758/BF03194105

©2021 Andrew Fowlie. This article is licensed under a Creative Commons Attribution (CC BY 4.0) International license, except where otherwise indicated with respect to particular material included in the article.