Lost in Stats: Pitfalls of the Scientific Method
Science employs a collection of methods that are designed to increase our knowledge about the real world. However, it is increasingly appreciated that the scientific method as it is actually used will sometimes lead to systematically biased results. This phenomenon is best-understood in the quantitative social sciences, which have faced a replication crisis in recent years -- i.e. a situation where widely-accepted effects have turned out to be spurious. A fairly good theoretical understanding has been achieved of how a field employing state of the art inferential methods can end up accepting non-existing effects as real.
This debate has not been as prominent in physics. There are reasons to believe that physics is less affected (access to more data, quantitative theories, restriction to less complex situations). But there are also signs that we should be more concerned and would benefit from absorbing the lessons learned elsewhere.
In this seminar, we will explore the systematic failure modes of the scientific method and scientific institutions, with particular attention to their applicability to physics.
First meeting is on Wednesday, the 13th of October, 4pm. If you are interested in participating, just show up or write an email to Felipe (email@example.com). Registration via KLIPS is not currently possible. Most topics will be covered by 1h talks by participants, some fundamentals might be covered by the organizers, lecture-style. Update: we are fully booked! Look at the list of presentations and preliminary materials here. (You might want to zoom in into the pdf to see the text.)
Preliminary List of Topics
- Review of inferential methods
- Orthodox & Bayesian statistics, null-hypothesis significance testing, p-values, measures of effect size, uncertainty quantification
- Causation vs correlation: Causal models, randomized designs, observational causal inference, Simpson's and Berksons' paradox
- Pitfalls of empirical science
- Technical accounts: Why most published research findings are false, Why Most Discovered True Associations Are Inflated, ...
- Fun reads: The control group is out of control, Unicorns vs Bigfoot
- The discussion in physics
- Expectation bias in physics. More specific: Bandwagon effects and error bars in particle physics. Less serious: cargo cult science.
- Systematic biases in quantum information experiments: Error bars in tomography, Systematic errors in Bell experiments.
- Biased measurements in astronomy
- We are only interested in systematic biases of the scientific method. That is distinct from hypotheses/theories/programs failing to live up to their hopes. Failure is an expected part of exploration. Systematic biases are not.
- Pseudoscience ("quantum homeopathy...") or crackpots. (Though the typology of crackpots is a fascinating topic, as is its relation to physics)
- Conspiracy theories, nihilism, unfocused cultural criticism
- Fraud. (Fraud is boring).
- "Culture wars" and hot-button political issues
- Activism. Here, we aim to understand, not to reform.