Why publishing everything is more effective than selective publishing of statistically significant results.
paper
Cited
Public
- Authors
- van Assen, Marcel A L M; van Aert, Robbie C M; Nuijten, Michèle B; Wicherts, Jelte M
- Year
- 2014
- Journal
- PloS one
- PMID
- 24465448
- DOI
- 10.1371/journal.pone.0084896
- PMCID
- PMC3894961
BACKGROUND: De Winter and Happee examined whether science based on selective publishing of significant results may be effective in accurate estimation of population effects, and whether this is even more effective than a science in which all results are published (i.e., a science without publication bias). Based on their simulation study they concluded that "selective publishing yields a more accurate meta-analytic estimation of the true effect than publishing everything, (and that) publishing nonreplicable results while placing null results in the file drawer can be beneficial for the scientific collective" (p.4). METHODS AND FINDINGS: Using their scenario with a small to medium population effect size, we show that publishing everything is more effective for the scientific collective than selective publishing of significant results. Additionally, we examined a scenario with a null effect, which provides a more dramatic illustration of the superiority of publishing everything over selective publishing. CONCLUSION: Publishing everything is more effective than only reporting significant outcomes.
Mean (solid lines) and mean plus/minus one meta-analytic standard error (dashed lines) of estimate obtained by a random-effect meta-analysis as a function of the number of publications for both the publishing everything and selective publishing approaches.
1–20 of 23
Next →
No entities extracted from this document yet.
No uploaded files.
Not in any collection.
| Citation | PMID | DOI | Status |
|---|---|---|---|
| AsendorpfJB, ConnerM, FruytFD, HouwerJD, DenissenJJA, et al (2013) Recommendations for increasing replicability in psychology. European Journal of Personality 27: 108–119. | — | — | — |
| BakkerM, van DijkA, WichertsJM (2012) The rules of the game called psychological science. Perspectives on Psychological Science 7: 543–554.2616811110.1177/1745691612459060 | — | — | — |
| Borenstein M, Hedges LV, Higgins JPT, Rothstein HR (2009) Introduction to meta-analysis. Chichester, UK: John Wiley & Sons, Ltd. | — | — | — |
| ButtonKS, IoannidisJPA, MokryszC, NosekBA, FlintJ, et al (2013) Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14: 1–12.10.1038/nrn347523571845 | — | — | — |
| CuijpersP, van StratenA, BohlmeijerE, HollonSD, AnderssonG (2010) The effects of psychotherapy for adult depression are overestimated: a meta-analysis of study quality and effect size. Psychological Medicine 40: 211–223.1949074510.1017/S0033291709006114 | — | — | — |
| De WinterJ, HappeeR (2013) Why Selective Publication of Statistically Significant Results Can Be Effective. PLoS ONE 8: e66463.2384047910.1371/journal.pone.0066463PMC3688764 | — | — | — |
| FanelliD (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS One 5: e10068.2038333210.1371/journal.pone.0010068PMC2850928 | — | — | — |
| FanelliD (2012) Negative results are disappearing from most disciplines and countries. Scientometrics 90: 891–904. | — | — | — |
| FergusonCJ, BrannickMT (2012) Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods 17: 120–128.2178708210.1037/a0024445 | — | — | — |
| FieldAP, GillettR (2010) How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology 63: 665–694.2049762610.1348/000711010X502733 | — | — | — |
| FrancisG (2012) Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review 19: 975–991.2305514510.3758/s13423-012-0322-y | — | — | — |
| FrancisG (2012) Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review 19: 151–156.2235158910.3758/s13423-012-0227-9 | — | — | — |
| GerberAS, GreenDP, NickersonD (2001) Testing for Publication Bias in Political Science. Political Analysis 9: 385–392. | — | — | — |
| Giner-SorollaR (2012) Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science 7: 562–571.2616811310.1177/1745691612457576 | — | — | — |
| HopewellS, LoudonK, ClarkeMJ, OxmanAD, DickersinK (2009) Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews 10.1002/14651858.MR000006.pub3PMC827655619160345 | — | — | — |
| HunterJE, SchmidtFL (2000) Fixed Effects vs. Random Effects Meta-Analysis Models: Implications for Cumulative Research Knowledge. International Journal of Selection and Assessment 8: 275–292. | — | — | — |
| IoannidisJPA (2005) Why most published research findings are false. Plos Medicine 2: e124.1606072210.1371/journal.pmed.0020124PMC1182327 | — | — | — |
| IoannidisJPA (2008) Why most discovered true associations are inflated. Epidemiology 19: 640–648.1863332810.1097/EDE.0b013e31818131e7 | — | — | — |
| IoannidisJPA (2011) Excess Significance Bias in the Literature on Brain Volume Abnormalities. Archives of General Psychiatry 68: 773–780.2146434210.1001/archgenpsychiatry.2011.28 | — | — | — |
| IoannidisJPA, TrikalinosTA (2007) An exploratory test for an excess of significant findings. Clinical Trials 4: 245–253.1771524910.1177/1740774507079441 | — | — | — |
| KraemerHC, GardnerC, BrooksJO, YesavageJA (1998) Advantages of excluding underpowered studies in meta-analysis: Inclusionist versus exclusionist viewpoints. Psychological Methods 3: 23–31. | — | — | — |
| NelsonLD, SimmonsJP, SimonsohnU (2012) Let's Publish Fewer Papers. Psychological Inquiry 23: 291–293. | — | — | — |
| NiemeyerH, MuschJ, PietrowskyR (2012) Publication bias in meta-analyses of the efficacy of psychotherapeutic interventions for schizophrenia. Schizophrenia Research 138: 103–112.2248402410.1016/j.schres.2012.03.023 | — | — | — |
| NiemeyerH, MuschJ, PietrowskyR (2013) Publication Bias in Meta-Analyses of the Efficacy of Psychotherapeutic Interventions for Depression. Journal of Consulting and Clinical Psychology 81: 58–74.2324436810.1037/a0031152 | — | — | — |
| NosekBA, Bar-AnanY (2012) Scientific Utopia: I. Opening Scientific Communication. Psychological Inquiry 23: 217–243. | — | — | — |
| NosekBA, SpiesJ, MotylM (2012) Scientific Utopia: II - Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science 7: 615–631.2616812110.1177/1745691612459058PMC10540222 | — | — | — |
| PashlerH, WagenmakersEJ (2012) Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? Perspectives on Psychological Science 7: 528–530.2616810810.1177/1745691612465253 | — | — | — |
| RosenthalR (1979) The file drawer problem and tolerance for null results. Psychological Bulletin 86: 638–641. | — | — | — |
| RothsteinHR, BushmanBJ (2012) Publication bias in psychological science: comment on Ferguson and Brannick (2012). Psychological Methods 17: 129–136.2236952010.1037/a0027128 | — | — | — |
| SchmidtFL, OhIS, HayesTL (2009) Fixed-versus random-effects models in meta-analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology 62: 97–128.1800151610.1348/000711007X255327 | — | — | — |
| SenaES, van der WorpHB, BathPM, HowellsDW, MacleodMR (2010) Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS biology 8: e1000344.2036102210.1371/journal.pbio.1000344PMC2846857 | — | — | — |
| SimmonsJP, NelsonLD, SimonsohnU (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22: 1359–1366.2200606110.1177/0956797611417632 | — | — | — |
| van Assen MALM, van Aert RCM, Wicherts JM (2013) Meta-analysis using effect size distributions of only significant studies. Tilburg University.10.1037/met000002525401773 | — | — | — |
| Van NoordenR (2013) The true cost of science publishing. Nature 495: 426–429.2353880810.1038/495426a | — | — | — |
| ViechtbauerW (2007) Hypothesis tests for population heterogeneity in meta-analysis. British Journal of Mathematical & Statistical Psychology 60: 29–60.1753557810.1348/000711005X64042 | — | — | — |
| ViechtbauerW (2010) The metafor package: A meta-analysis package for R (Version 1.3-0). | — | — | — |
In this knowledge base
| Title | Year | PMID |
|---|---|---|
| Candidate gene-environment interaction research: reflections and recommendations. | 2015 | 25620996 |
External
| Title | Authors | Journal | Year | Link |
|---|---|---|---|---|
| Beyond the null: Recognizing and reporting true negative findings. | Schweinfurth MK et al. | — | 2025 | → |
| Publication bias is bad for science if not necessarily scientists. | Heesen R et al. | — | 2025 | → |
| Associations Between Cognitive Function and ACL Injury-Related Biomechanics: A Systematic Review. | Bertozzi F et al. | — | 2023 | → |
| Effectiveness of non-pharmacological treatments for vestibular and oculomotor dysfunction in patients with persistent post-concussive symptoms: protocol for a systematic review and meta-analysis. | Sheeba S et al. | — | 2023 | → |
| Positive psychology interventions in the United Arab Emirates: boosting wellbeing - and changing culture? | Lambert L et al. | — | 2023 | → |
| Screen media exposure and young children's vocabulary learning and development: A meta-analysis. | Jing M et al. | — | 2023 | → |
| A parallel architecture perspective on pre-activation and prediction in language processing. | Huettig F et al. | — | 2022 | → |
| Publish without bias or perish without replications. | Ventura R | — | 2022 | → |
| A Bayesian approach to analysing cortico-cortical associative stimulation induced increases in the excitability of corticospinal projections in humans. | Carson RG et al. | — | 2021 | → |
| Competition for priority harms the reliability of science, but reforms can help. | Tiokhin L et al. | — | 2021 | → |
| Negative polarity in quantifiers evokes greater activation in language-related regions compared to negative polarity in adjectives. | Agmon G et al. | — | 2021 | → |
| Do bats use guano and urine stains to find new roosts? Tests with three group-living bats. | Brown BKG et al. | — | 2020 | → |
| Low replicability can support robust and efficient science. | Lewandowsky S et al. | — | 2020 | → |
| The Role of Animal Cognition in Human-Wildlife Interactions. | Goumas M et al. | — | 2020 | → |
| A theory and methodology to quantify knowledge. | Fanelli D | — | 2019 | → |
| Effects of speech rate, preview time of visual context, and participant instructions reveal strong limits on prediction in language processing. | Huettig F et al. | — | 2019 | → |
| Hostile Intent Attribution and Aggressive Behavior in Children Revisited: A Meta-Analysis. | Verhoef REJ et al. | — | 2019 | → |
| Mapping pharmacy journals: A lexicographic analysis. | Mendes AM et al. | — | 2019 | → |
| Recommendations for Increasing the Transparency of Analysis of Preexisting Data Sets. | Weston SJ et al. | — | 2019 | → |
| Clinical trials. | Minneci PC et al. | — | 2018 | → |
| Conditional equivalence testing: An alternative remedy for publication bias. | Campbell H et al. | — | 2018 | → |
| Critical Evaluation of Data Requires Rigorous but Broadly Based Statistical Inference. | Cox NJ et al. | — | 2018 | → |
| Data Sharing in Neurosurgery and Neurology Journals. | Johnson JN et al. | — | 2018 | → |
| Impact of cephalosporin restriction on incidence of infections with extended-spectrum beta-lactamase-producing Klebsiella pneumoniae in an endemic setting. | Nadrah K et al. | — | 2018 | → |
| Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies. | Grant S et al. | — | 2018 | → |
| Making the scientific literature fail-safe. | Casadevall A et al. | — | 2018 | → |
| Simple changes of individual studies can improve the reproducibility of the biomedical scientific process as a whole. | Steinfath M et al. | — | 2018 | → |
| The changing forms and expectations of peer review. | Horbach SPJMS et al. | — | 2018 | → |
| A critique of the "ultra-high risk" and "transition" paradigm. | van Os J et al. | — | 2017 | → |
| A multi-disciplinary perspective on emergent and future innovations in peer review | Tennant JP et al. | — | 2017 | — |
| A multi-disciplinary perspective on emergent and future innovations in peer review. | Tennant JP et al. | — | 2017 | → |
| Conversion of Plastic Surgery meeting abstract presentations to full manuscripts: a brazilian perspective. | Denadai R et al. | — | 2017 | → |
| Dorsal striatum mediates deliberate decision making, not late-stage, stimulus-response learning. | Hiebert NM et al. | — | 2017 | → |
| Human Fear Chemosignaling: Evidence from a Meta-Analysis. | de Groot JHB et al. | — | 2017 | → |
| Indirect evidence of reporting biases was found in a survey of medical research studies. | Albarqouni LN et al. | — | 2017 | → |
| Results Blind Science Publishing. | Locascio JJ | — | 2017 | → |
| The earth is flat (<i>p</i> > 0.05): significance thresholds and the crisis of unreplicable research. | Amrhein V et al. | — | 2017 | → |
| The influence of journal submission guidelines on authors' reporting of statistics and use of open research practices. | Giofrè D et al. | — | 2017 | → |
| The Weak Spots in Contemporary Science (and How to Fix Them). | Wicherts JM | — | 2017 | → |
| Using synthetic biology to explore principles of development. | Davies J | — | 2017 | → |
| Can the behavioral sciences self-correct? A social epistemic study. | Romero F | — | 2016 | → |
| Open data are not enough to realize full transparency. | Bouter LM | — | 2016 | → |
| Publication bias and the canonization of false facts. | Nissen SB et al. | — | 2016 | → |
| Publication outcome of abstracts presented at the AACC annual meeting. | Greene DN et al. | — | 2016 | → |
| Seven Pervasive Statistical Flaws in Cognitive Training Interventions. | Moreau D et al. | — | 2016 | → |
| The academic, economic and societal impacts of Open Access: an evidence-based review. | Tennant JP et al. | — | 2016 | → |
| Candidate gene-environment interaction research: reflections and recommendations. | Dick DM et al. | — | 2015 | → |
| From Protocols to Publications: A Study in Selective Reporting of Outcomes in Randomized Trials in Oncology. | Raghav KP et al. | — | 2015 | → |
| Null hypothesis significance testing: a short tutorial. | Pernet C | — | 2015 | → |
| Refinement of experimental design and conduct in laboratory animal research. | Bailoo JD et al. | — | 2014 | → |
| Why are medical and health-related studies not being published? A systematic review of reasons given by investigators. | Song F et al. | — | 2014 | → |