Evidence synthesis, such as systematic reviews and systematic maps, is the bringing-together of knowledge and findings from research to answer questions about what we know, what research exists and where the critical knowledge gaps lie. These methods use systematic, and ideally reproducible and transparent, approaches to gathering, evaluating and synthesizing research, with an aim of minimizing bias in study selection. The questions asked and answered by these types of approaches are often informed by—and critical to—stakeholders across a wide range of sectors, including policy-makers, clinicians, funding agencies and others. Evidence syntheses can thus critically inform decisions about policy, practice and funding that can have wide implications for society.
Evidence synthesis relies on the ability to access the findings of relevant studies. Barriers to access can prevent comprehensiveness, and missed studies or incomplete data can dramatically impact the conclusions drawn from an evidence synthesis project, and thus the recommendations formed by those findings. These barriers include paywalls (for subscription journals and research databases), a lack of complete reporting of findings or statistics, inadequate information about the study methodology, unavailability of underlying data, and the reluctance to publish non-significant findings.
Fortunately, many of these barriers are addressed by the open science movement. Open science embraces open data, open methods and open access publishing. By openly sharing underlying data, meta-analyses, often a feature of systematic reviews, become more feasible. Meta-analyses statistically combine results across different but similar studies, thus making findings from many small studies more statistically meaningful. More openness in methods, including study context, can help in understanding the generalizability or specific contextual aspects of findings. Open access publishing can make it easier to find and access studies that might address the research question posed by the evidence synthesis project, and allows researchers in under-resourced settings to conduct evidence synthesis that is comprehensive and not constrained by access barriers.
Another important open science practice that contributes to a more open evidence ecosystem is pre-registration or registered reports. This involves the publishing of one's study plan and protocol prior to conducting the research. This can be done either through a formal peer-review process in a journal (registered report) or more informally in a platform such as Open Science Framework (pre-registration). By doing this, researchers make clear the methods they intend to use for gathering and analyzing data, including all outcomes they intend to measure. This can call attention to ongoing, but not yet complete, research that might be relevant to an evidence synthesis question. It also adds transparency that reduces outcome reporting bias, a problem often encountered in evidence synthesis in which researchers only report results for measured outcomes that yielded significant results. Knowing whether something had no effect on an outcome is equally important when making decisions that impact people, communities and society.
So next time you plan a research project, think about the broader implications your work may have when taken into account with similar studies that might be published in the future. Your work may be synthesized with other research to inform critical practice and policy decisions. Using open science practices makes it more likely that your work is discovered and then used appropriately and accurately to inform recommendations and directions in future research.
For more about open synthesis:
Donnelly, C. A., Boyd, I., Campbell, P., Craig, C., Vallance, P., Walport, M., Whitty, C. J. M., Woods, E., & Wormald, C. (2018). Four principles to make evidence synthesis more useful for policy. Nature, 558(7710), 361–364. https://doi.org/10.1038/d41586-018-05414-4
Haddaway, N.R. Open Synthesis: on the need for evidence synthesis to embrace Open Science. Environ Evid 7, 26 (2018). https://doi.org/10.1186/s13750-018-0140-4
Nakagawa, S., Dunn, A. G., Lagisz, M., Bannach-Brown, A., Grames, E. M., Sánchez-Tójar, A., O’Dea, R. E., Noble, D. W. A., Westgate, M. J., Arnold, P. A., Barrow, S., Bethel, A., Cooper, E., Foo, Y. Z., Geange, S. R., Hennessy, E., Mapanga, W., Mengersen, K., Munera, C., … Haddaway, N. R. (2020). A new ecosystem for evidence synthesis. Nature Ecology & Evolution, 4(4), 498–501. https://doi.org/10.1038/s41559-020-1153-2 (open access version: https://pure.bond.edu.au/ws/portalfiles/portal/35767811/AM_.pdf)
For more information about evidence synthesis methods and for guidance on conducting evidence synthesis, visit the Evidence Synthesis service page.