Public health emergencies demand synthesized evidence rapidly. During COVID-19, Ebola outbreaks, and other crises, researchers face pressure to produce guidance within the first week hours, for example, to prevent further spread of infections in the community. Meanwhile, traditional systematic reviews require 6 to 18 months to complete. This temporal mismatch creates an evidence vacuum that decision-makers often fill with preliminary findings or incomplete data.
| Key Takeways |
| Optimize Statistical Methods: Avoid outdated software defaults. Use corrections like HKSJ to improve result reliability without increasing analysis time. |
| Prioritize Clinical Meaning: Routinely report prediction intervals alongside confidence intervals to clearly communicate expected outcomes and account for real-world heterogeneity. |
| Deploy AI Strategically: Use AI to automate repetitive tasks like screening and extraction, but retain human oversight for all synthesis, judgment, and validation. |
Stefano Brini, PhD, a scientific editor at JMIR Publications, knows this pressure intimately. "I worked at the UK Health Security Agency for some time where we responded to public health emergencies," he explains. "We were tasked by commissioners to perform an evidence synthesis within maybe 72 hours and 1 week depending on the emergency in response to a public health emergency. The aim of health security agencies is to save lives. And so for any day that passes by that you don't have the evidence to act on, lives could be lost."
The central question is how to bridge this temporal gap without sacrificing the methodological rigor that makes evidence trustworthy. Recent developments in evidence synthesis methodology, combined with careful application of artificial intelligence, suggest that this is possible – and that many researchers are currently achieving neither speed nor adequate rigor.
Brini recently led a tutorial, “Rapid Evidence to Decision (RED): Modern Meta-Analysis & AI Accelerated Systematic Reviews” at Medical Informatics Europe (MIE) in Genova, Italy, in May 2026, where the event theme was “Opening the Personal Gate between Technology and Health Care." He also is the lead author on a published guide that accompanies the live event topic.
Many published meta-analyses contain statistical limitations that go unrecognized. "There is evidence to suggest that a lot of the evidence that comes from meta-analysis in medical literature, the estimates that they produce are not really close to the true value," Brini notes. "The underlying statistical indices may not be very representative of what the true value is in clinical settings."
Most statistical software defaults to methods developed decades ago without evaluating whether they suit your specific analysis.
The problem: The DerSimonian-Laird (DL) method, the standard random-effects approach since the late 1980s, underestimates variance when meta-analyses include fewer than 10 studies or substantial variability exists across studies. This produces confidence intervals that are narrower than warranted, inflating false-positive risk – particularly problematic in emergency decision-making.
What works instead:
The takeaway: One methodological choice – made during protocol development – substantially improves result reliability with no time penalty.
Heterogeneity – genuine differences in treatment effects across studies – is expected in real-world evidence synthesis. Brini uses a clear conceptual framework to explain why: "A fixed effect assumes there is only one true effect size across all studies. But in a random-effects model, the underlying assumption is that there is some degree of heterogeneity, that means true differences across studies."
He illustrates the point with a practical example: "If I were to measure the length of a table and another person were to measure the same table, my measurement might be that the table is 1 meter long and the other person would measure it at 1.3. The length of the table hasn't changed – that difference is random error. But if you measure something like depression or anxiety, or any construct, a difference between measurements is due to at least two things: random error and true differences among studies."
This distinction matters clinically. "It means that on average, if you were to pick a random person from the population of people with anxiety, an intervention may improve symptoms of anxiety on average, but that doesn't mean that it will work on everyone with anxiety," Brini explains.
The limitation of I²: This statistic describes what proportion of observed variation stems from real differences versus sampling error. An I² of 90% does not indicate whether, or the extent to which,effects are clinically different or in which populations patients benefit or experience harm.
What communicates actual clinical meaning:
The takeaway: Prediction intervals translate statistical heterogeneity into clinically actionable information. Routinely report them when sufficient studies are present.
During the live tutorial at MIE, one participant asked for clarification on the boundary between confidence and prediction intervals. Brini highlighted this as an important distinction that remains relatively obscure even within the meta-analysis community. He expanded on why reporting the prediction interval is essential for revealing the actual range of expected effects in future clinical applications, moving beyond the simple average represented by confidence intervals.
One of the most persistent sources of confusion in meta-analysis reporting boils down to a single phrase. Often, systematic reviews will confidently state that the authors "explored the data for publication bias." According to Brini, this phrasing is fundamentally flawed. "That is, in some sense, incorrect because there is no test for publication bias," he observes. Instead, authors of meta-analyses need to focus on what they can measure: small-study effects. Understanding the distinction between the two is vital for interpreting your data correctly.
When small-study effects are present
If you spot asymmetry in a funnel plot or get a statistically significant Egger’s test, it indicates that small-study effects exist. This can definitely be a symptom of publication bias, such as the file drawer effect or reporting bias. However, it could just as easily be the result of methodological heterogeneity or pure chance.
When small-study effects are absent
Even if your Egger's test is non-significant and your funnel plot is symmetrical, adopt a careful approach in your interpretation of the results. Publication bias is ubiquitous in medical and psychological research. A "clean" test simply means small-study effects aren't present; it does not demonstrate that null findings have not been suppressed in that field.
The takeaway: Terminology matters for transparency. Funnel plots and Egger's tests detect small-study effects – a pattern where smaller studies report larger effects. They do not directly test for publication bias, which remains difficult to conclusively establish.
AI should be deployed strategically in evidence synthesis, with clear understanding of its capabilities and limitations.
Based on currently available evidence:
AI may be useful for:
AI is not appropriate for:
The practical application involves using AI to reduce time spent on repetitive tasks, allowing human expertise to focus on decisions requiring judgment and nuanced interpretation of findings.
During the live MIE tutorial, audience participants explored and prompted generative AI tools in carrying out specified literature review tasks. They shared their experiences testing paid and free versions of various commonly available commercial models during the session and identified important considerations for even basic tasks of evidence synthesis: the quality and detail of their prompts resulted in different outputs from different models, and search strategies suggested also differed in detail and comprehensiveness. Consulting a professionally trained librarian or information retrieval specialist to build a robust search strategy still remains a gold standard.
The essential principle: AI remains a tool requiring human accountability. Researchers must validate AI-generated search strategies, verify data extraction, and critically interpret results. The researcher bears full responsibility for all decisions and conclusions.
You face competing pressures: produce credible evidence efficiently, navigate unfamiliar statistical terrain, and manage tool selection with limited guidance. The good news is that speed and rigor are not opposing forces – they require intentional choices during protocol development.
Brini emphasizes the foundational principle: "The most difficult analysis was already done by the authors of the primary study. You just have to put it all together with the right agreements." This reframing is important. The intellectual heavy lifting in evidence synthesis involves methodological choices and transparent interpretation—not computational complexity.
Step 1: Commit to transparent method selection
Before analysis begins, specify your statistical approach in the protocol and explain your choices. What model are you using (fixed or random effects) and why? What heterogeneity measures will you report? Which software performs these analyses reliably? Documenting these decisions in advance prevents both software defaults and post-hoc adjustments that erode credibility.
Step 2: Plan for prediction intervals and heterogeneity exploration
If your research question expects variation across settings or populations, build subgroup or meta-regression analyses into your protocol. Identify a priori hypotheses rather than data-dredging after results emerge. Commit to reporting prediction intervals when sample size permits. These decisions communicate that you expect and can explain variation in effects.
Step 3: Deploy AI where it accelerates without replacing judgment
Identify the most time-consuming manual tasks in your workflow: initial screening of thousands of titles, de-duplication, or standardized data extraction. It’s possible to use AI for these. Validate the output through spot-checking and maintain human accountability for all synthesis decisions. Document your AI use transparently in the Methods section, including tool name, version, and specific tasks performed. Always check and adhere to journal or publisher policies on disclosing or describing the use of AI tools in your methods or manuscript preparation.
Conducting rigorous evidence synthesis and meeting realistic timelines are achievable simultaneously. The pathway involves examining assumptions, communicating explicitly about uncertainty, and using tools—including AI—strategically rather than by default.
For detailed exploration of these statistical methods, see the published guide on methodological rigor in meta-analysis. The paper includes a detailed checklist of statistical methods and interpretations for rigorous meta-analysis, with guidance on applying different approaches according to specific contexts.
Looking for more guidance? Explore our Early Career Researcher Resources for submission guidelines, author tools, and foundational support for your research career.