A meta-epidemiological study found that meta-analyses of the same trials may obtain drastically conflicting results

Research output: Contribution to journalJournal articleResearchpeer-review

Objectives: To assess how much protocols reduce methodological flexibility and variation of results in meta-analyses comparing operative with nonoperative treatments for proximal humerus fractures. Study Design and Setting: A meta-epidemiological study. We searched four bibliographic databases for eligible meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of their protocol. We identified the included studies and extracted intervention effect data for functional outcome, quality of life, and adverse events. Using the same intervention effect data for each outcome domain, we conducted multiple meta-analyses using differing combinations of methodological choices (analytical scenarios) without restricting the available methodological choices (as if no protocol was used). For each protocol, we repeated this process but restricted the available choices to those listed in the protocol. We then compared the distributions of effect estimates with and without protocols. Methodological flexibility was estimated as the difference in number of possible meta-analyses and the variation of results as the difference between the most conflicting effect estimates. Results: We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols. The protocols markedly reduced methodological flexibility (range for functional outcomes 94%–99%; quality of life 58%–76%; adverse events 87%–91%). The corresponding reduction in the variation of the results varied (range for functional outcomes; 33%–78%, quality of life; 10%–24%; adverse events 10%–13%). Conclusion: Although the protocols consistently reduced methodological flexibility, the reduction in the variation (bias) of results varied. Consequently, review authors may obtain conflicting results based on the same data. We advise caution when interpreting the conclusions of meta-analyses with absent or imprecise protocols.

Original languageEnglish
JournalJournal of Clinical Epidemiology
Volume156
Pages (from-to)95-104
Number of pages10
ISSN0895-4356
DOIs
Publication statusPublished - 2023

Bibliographical note

Publisher Copyright:
© 2023 Elsevier Inc.

    Research areas

  • Meta-analysis, Methodological flexibility, Protocol, Proximal humerus fractures, Variation of results, Vibration of effects

ID: 369127326