With the recent launch of MetaReviewer version 1.2.0, we're seeing lots of new and returning users on our site. Welcome to all! We want to take this opportunity to (re-)introduce ourselves, tell you a bit more about the origins of our product, and explain why we will continue to invest in it.
MetaReviewer officially kicked off in 2021 with funding from the National Science Foundation (EHR-2000672) and we have since received additional funding from the American Institutes for Research's Equity Initiative. Our product designers and developers are all employed by the American Institutes for Research (AIR), in positions ranging from Researcher to Principal Software Architect.
Most of us are seasoned evidence synthesists: we conduct systematic reviews and meta-analyses (primarily in the social and behavioral sciences), we teach meta-analysis workshops, and we develop methods and tools to improve meta-analysis dissemination. We’re well-versed in the process, and we consider ourselves fairly technologically savvy.
Yet time after time, as we would embark on a new evidence synthesis project, we found ourselves re-inventing the wheel – piecing together a hodgepodge of applications and programs to create the full pipeline for our review. Often, we relied on proprietary programs that were only accessible through a subscription – which we found problematic in principle, and also in practice as many of us left academia for applied research. And in many cases, we were co-opting programs like spreadsheet software or relational database systems that weren’t exactly designed for evidence synthesis and therefore didn’t have quite the features and functions we needed.
A brief timeline
MetaReviewer, as a result, is over 12 years in the making. Our challenges to conduct an efficient and smooth synthesis inspired the idea for MetaReviewer: fully customizable, designed for evidence synthesis by evidence synthesists, and building in best practices by default. After years of back-of-the-envelope wireframes over cold beverages, in early 2020, we finally submitted a proposal to NSF for funding to make it happen. And once we won our first award, the idea took off: we teamed up with some of the best developers in the business, created a prototype, and launched our Alpha site (version 0.1.0) in late 2021. Alpha was a hit with our internal project teams, and we finally went public with our Beta launch (version 1.0.0) in fall of 2022. More than 240 users from 23 countries and five continents accessed the site during our Beta launch period from September through November 2022, and they created over 40 projects in topics spanning education, psychology, ecology, climate and environmental science, health, and international development. That November, we closed to new users so that we could gear up for another release.
After our Beta launch, we knew this thing had some legs. Evidence synthesists from all over the world were reaching out to request access and share ideas for new features and functions. However, we knew our work was just beginning. The Beta release only tackled one piece of the pipeline: data extraction, which occurs towards the very end of the synthesis process once all the studies have been screened for eligibility. We opted to begin with data extraction because it is the most tedious and time-consuming phase of the typical evidence synthesis, so we had good reason for starting part-way through the process. Still, the full vision of MetaReviewer had yet to be realized. We had to keep chipping away at the other synthesis phases to make MetaReviewer the one-stop-shop we knew the synthesis community needed.
In December 2023, we launched MetaReviewer version 1.2.0. In response to valuable feedback provided by our Alpha and Beta users, this release added several new features to our platform, including full-text screening, reconciling conflicts, and estimating effect sizes, along with improvements to existing tools for general project management. Over 80 new users have requested access since we announced our most recent launch, and we’re adding more users every day. We’re finally building out both ends of the pipeline, and at this point, we’ve connected many of the pieces. But we won’t stop until the full pipeline is complete.
What's next?
We have many ideas to extend MetaReviewer’s usefulness to the community we serve. Towards the beginning of the pipeline, we want to build out our literature search and citation management functionality to allow for search strategy documentation, reference importation, and duplicate removal. We also want to create an abstract screening interface that incorporates new advancements in machine learning algorithms to reduce project cost and reviewer burden. Most importantly, we will continue to investigate how we can leverage new and emerging artificial intelligence applications and models that enhance the synthesis results. We don’t have any definitive plans yet, but we anticipate that future iterations of MetaReviewer will include these important developments.
These developments will obviate the need for synthesists to transfer data from program to program to get through the early stages of the synthesis. Prior work indicates synthesists rely on no less than four different software programs to assist in the beginning stages of the typical review (Pigott & Polanin, 2020), which presents a multitude of opportunities for loss of information and user error. Transferring information from program to program also limits reproducibility, a crucial best practice in systematic reviewing that all synthesists should do their best to model.
Our long-term vision for MetaReviewer is simple. We want to create a single, simple platform that can encompass every stage of an evidence synthesis project, from start to finish. And we never want to charge our users for access. Because we believe that great software shouldn’t be limited to users with the most resources, especially when its use has the potential to create such impactful and positive change.
So, start your new review project in MetaReviewer today to help us keep up this momentum!
