MetaReviewer is an invaluable tool for evidence synthesis, but did you know it’s also versatile enough to support a range of research synthesis types? In this latest Meta in Motion: Project Spotlight, we highlight an AIR team conducting a research synthesis project type called a “second-order meta-analysis”.

Examining selective reporting practice in education research using a second-order meta-analysis

Continuing our theme from last month, we highlight our second project, the Consequences of Selective Reporting Bias in Education Research. As the project team summarized:

Selective reporting is a critical concern in the scientific literature. Imagine if a study only reported on the positive effects of an educational intervention, while hiding negative or inconclusive results. This bias would give others an inaccurate understanding of the intervention’s effectiveness. This problem is especially salient in meta-analyses that combine quantitative results across many studies. Meta-analytic conclusions may appear comprehensive but can be biased due to selective reporting in the scientific literature reviewed.

It’s clear that selective reporting is an issue worth diving into, especially for anyone invested in evidence-based decision and policy.

The project has two primary aspects. The first is to develop modern selective reporting models that account for dependent effect sizes. The second is to “empirically evaluate the consequences of selective reporting in education research by reanalyzing data from prior meta-analyses.” It is this second aspect of the project for which the team used MetaReviewer to conduct screening and coding.

The project is funded by the Institute of Education Sciences, the research arm of the U.S. Department of Education (Award Number: R305D220026). Leading this project is Principal Investigator (PI) Martyna Citkowicz, supported by a great team: co-PIs James Pustejovsky (University of Wisconsin-Madison) and Ryan Williams, along with project director David Miller and dedicated AIR analysts and statisticians. I am also a co-PI on the project.

Second-order meta-analysis

A second-order meta-analysis (SOMA) is a type of research synthesis known as an “Overview,” which Polanin, Maynard, & Dell (2017) describe as combining findings from multiple syntheses into a single, cohesive analysis. The basic idea is that the Overview team seeks to combine multiple syntheses into a single product. In a SOMA, the aim is specific: to quantitatively combine effect sizes from individual meta-analyses into one unified synthesis.

The AIR SOMA project team sought to collect effect sizes from all meta-analytic reports included in the SOMA. For each eligible and included meta-analysis, the SOMA project team extracted each primary study effect size that the original meta-analysis team reported. With around 100 meta-analyses in their sample, the SOMA team extracted between 3 and over 100 (!) effect sizes per meta-analysis, each representing findings from individual studies.

Using this wealth of data, the SOMA team will re-estimate meta-analytic models to explore the effects of selective reporting. First, they will re-estimate the meta-analytic model closely resembling what the original meta-analytic team estimated. Then, using the project team’s modern selective reporting formulas, they will re-analyze the meta-analytic data. Finally, by comparing these two sets of results, the team aims to shed light on how selective reporting may be impacting meta-analytic results in the field of education research.

SOMA coding form

As we’ve highlighted in earlier posts, MetaReviewer is designed to support screening and coding for meta-analyses even when they involve complex data. Built on the principles of hierarchical relational databases, MetaReviewer is simultaneously flexible and powerful enough to accommodate any type of data structure.

The SOMA team created a three-part coding form, using MetaReviewer’s customizable template as a starting point. The first section captured report-level details, covering information that applied to the entire meta-analysis, like funding status, search procedures, and publication bias analyses.

The second section was designed to be flexible, with a structure that could handle multiple records/columns when needed. The project team used this second section to capture information about each possible meta-analysis conducted within the report. For instance, if a report analyzed the included primary studies by outcome type, like mathematics and attendance, we would enter two separate records for the report meta-analysis.

Why the SOMA team used excel for effect size data

The third section of the SOMA team’s coding form is where things get interesting. The SOMA project team sought to capture every effect size that the meta-analysts reported for each included primary study. Because of the large number of potential effect sizes per meta-analysis, the team opted to capture effect size information in Excel.

We can hear our long-time readers now: 'See, even the MetaReviewer team is using Excel!' But let’s clear this up – we’re not advocating Excel for everyone. This was a unique case, tailored to very specific needs. The SOMA team’s project had a couple of thorny issues that made the current version of MetaReviewer not ideal for data capture.

First, the team anticipated handling 20–30 effect sizes per study—a level far above the norm, as most primary studies yield only 3–4 effect sizes, with 10–12 at most. MetaReviewer’s current user interface – while great for a few effect size rows – can become unwieldly in cases where more than 20 rows are expected to be extracted per study.

Second, the SOMA project team did not expect to extract information so that they could re-estimate the primary study effect sizes. The vast majority of meta-analytic authors report already-estimated effect sizes (Polanin, Hennessy, & Tsuji, 2020). So MetaReviewer’s built-in effect size field selection and calculation functionality, incredibly useful for evidence synthesis teams, wasn’t of any use to the SOMA project team.

Third, the SOMA team planned to process a significant amount of data in bulk. For this particular project, that entailed(a) extracting effect size values from figures and tables, sometimes by hand and sometimes using artificial intelligence (see below for a brief explainer) and (b) copying and pasting large amounts of values so that they were aligned in a structured way. MetaReviewer’s current configuration, on the other hand, is for individual record-entering.

In the end, the team split tasks: they used MetaReviewer to manage records and capture core information in sections 1 and 2, while Excel handled the effect size data from each meta-analysis. They used MetaReviewer as the primary record-keeper, project management system, and capturing information within sections 1 and 2 of the coding form. They used Excel to capture effect size information from each meta-analysis.

How to connect excel to MetaReviewer

At first, connecting MetaReviewer data with Excel might seem like a daunting or even risky move. If you’re not sure why you’d need to, you’re probably right to hesitate! We – and the SOMA project team – would be the first to advocate for the use of a single data-capture application for most projects.

So, how did the SOMA team pull it off? Like all good magic tricks, the solution is pretty simple.

The SOMA’s coding form in fact did include three sections, each at a different level of a nested hierarchy: meta-analysis-level information, report-level information, and effect size information. Instead of listing each effect size in a separate row, the SOMA team used a single field to display the meta-analysis ID, which connected to the main meta-analysis section. As a double-check with the Excel file, the SOMA team also input the number of effect sizes per meta-analysis.

In each Excel file, the first column was reserved for the meta-analysis ID, matching the IDs generated in MetaReviewer. Therefore, when the MetaReviewer data is exported, the SOMA team can match each Excel effect size row to the appropriate meta-analysis ID. See the first screenshot to view this setup in MetaReviewer, and the second for how it looks in Excel.

IES Selective Reporting - Coding Form IES Selective Reporting - Excel

Conclusion

MetaReviewer supports a wide range of research synthesis project types, each with its own demands. For the SOMA team, this meant finding a system both robust enough for complex data and flexible enough to integrate with tools like Excel when needed. As pragmatic researchers, we believe in using the best tool for the job—even if that occasionally means enlisting help from our old friend, Excel. Ultimately, whether it’s with MetaReviewer alone or blended with Excel, our goal is to make your research process as seamless and impactful as possible.

Post-credits appendix: Using GPT for extracting effect size information from Tables and figures

The MetaReviewer team has been asked more than a few times if we have plans to incorporate artificial intelligence (AI). The short answer is… yes! But it will likely be a while until (a) it is cost-effective to do so, (b) AI models progress enough to be reliable at scale, and (c) various fields determine what AI models and applications truly work best for research synthesis. We are actively working on all these questions and have recently been awarded NSF funding to start the process. We will provide updates on this front as we have more information.

In the meantime, many of us have been conducting our own AI experiments using GPT or its competitors. The SOMA project team, over the past year, used GPT extensively when collecting information from tables and figures.

The basic setup of using GPT – or any relatively new(ish) AI application – for extracting information from figures and tables is relatively straightforward.

  1. Identify the table or figure of interest; either (a) take a clear, zoomed-in screenshot or (b) download the image file, which is sometimes available when viewing the article online.
  2. Write a detailed prompt of what you’d like the GPT to accomplish. The one listed below chunks the work into several tasks, which has been shown to be an effective prompting strategy.
  3. Require that the GPT produce the extracted information as a CSV file or in some other easily copy/past-able format.

Example prompt: You are an expert in extracting data from images and putting the information into tables. I need you to access that skill for the attached screenshot. Take this picture of a forest plot and analyze it. For each row, create a column for the study id (which is the far left column), then the effect size (which is the number on the right hand column before the first parentheses), and the lower and upper confidence limit (which are the numbers inside the parentheses). Output a table I can copy/paste to a spreadsheet.

The screenshot below displays a typical output, which is easily copy/pasted into Excel for further processing.

Meta in Motion: SOMA Table

For this particular task, GPT worked really well. Will it completely remove the need for humans in the process of conducting a research synthesis? Likely not in the short term. But like others have seen, for specific, concrete tasks, AI can be really useful within the larger evidence synthesis process.