Shut the (screening) door! Abstract screening has arrived to MetaReviewer. This build has been a labor of love and we're excited to tell you all about this new functionality.
Creating a project
Creating a project with abstract screening is a straightforward endeavor. After logging into MetaReviewer, simply click on the New Project tile and then select Start a synthesis from abstract screening. As always, the MetaReviewer project setup tool will walk you through some key steps for creating your project.
This will feel familiar to experienced users with one key exception: instead of prompting you to create a full-text screening or coding form, the project setup tool will prompt you to configure abstract screening settings. Those settings include:
- The number of abstracts to pilot
- Screening criteria
- Inclusion and exclusion keywords
- Whether abstracts should be single- or double-screened
- Whether and how machine learning should be applied to help label and sort abstracts
Each of these settings will determine how MetaReviewer sets up the workflow and the displays in your abstract screening portal.
Step into the (abstract screening) portal
Once you've created a project, you can enter the abstract screening portal from your project page. You'll see something like this:
The abstract screening portal contains a progress dashboard (more on that later), a paginated list of citations and corresponding abstracts, and a task bar. You'll also see options related to the pilot phase of abstract screening, if you choose to incorporate that as part of your team's training! We encourage you to explore the MetaReviewer User Guide to learn more about piloting. In this post, we'll focus on the basics of screening abstracts in MetaReviewer.
To screen abstracts, all you have to do is click on the blue Continue screening button. This will take you to an abstract page, which looks like this:
Each abstract page contains the citation ID, author list, title, and abstract; a panel where you can see your decisions (and other screeners' decisions, when appropriate); a panel with information to support your decision-making including screening criteria, eligible keywords, and ineligible keywords; a place to take notes; and, if you elect to use it, a predicted screening decision based on a machine learning model.
Most importantly, each page contains icons for making screening decisions. You can click on the icons to either keep, drop, or skip an abstract. You can also click on the icons to go back to a previous abstract in your queue or flag and abstract for discussion with your team. For those who prefer click-free decision-making, we've also provided keyboard shortcuts that appear when you hover over each icon.
As you make screening decisions, MetaReviewer will automatically advance you to the next abstract in your queue. Over time, your abstract screening portal will transform from something dull into a kaleidoscope of decisions:
Of course, there's more to it. But, you'll have to check out the User Guide to get into the nitty-gritty details.
A machine learning boost
Screening abstracts for a synthesis project can be a significant and, at times, tedious undertaking. But, it doesn't need to be! MetaReviewer offers users the option of leveraging machine learning (ML) to accelerate the abstract screening phase. In its simplest form, ML is an algorithm that uses data from human screening decisions to make predictions about future screening decisions based on patterns of words within each abstract.
Project teams can use these decisions to:
- Prioritize abstracts that are most likely to be eligible, helping to save time and effort while maintaining screening quality
- Refine projections about the number of eligible studies to be included in the review, allowing timeline and budget estimates to be updated accordingly
- Determine whether and when to stop manually screening abstracts deemed highly unlikely to be eligible
We know from experience that using ML for abstract screening can be difficult to justify without a full understanding of how the model works. That's why we've provided a thorough explanation of our ML algorithm in our User Guide so review team leadership can make decisions about how to use ML (or not) on their projects.
A sprinkle of project management
As project directors (PDs) ourselves, we know how burdensome tracking progress, managing budgets and timelines, and producing PRISMA flowcharts can be. So, we added a progress dashboard to the abstract screening portal to support all of our fellow PDs with their progress monitoring duties:
In additional to some helpful counts, the abstract screening dashboard contains two data visualizations:
- A line chart that displays how many abstracts have been screened over time, broken down by those that have been single-screened (red line), double-screened but have discrepancies (grey line), and completed (green line)
- A Sankey diagram that gives a real-time snapshot of abstract statuses. Users can hover over each path in the diagram to see how many abstracts are awaiting a decision, have been single- or double-screened, have discrepancies, or have been completed.
While the dashboard was designed with PDs in mind, we hope individual screeners will use it to stay motivated and help their team reach the abstract screening finish line!
All in one place
Finally, it's always been our dream to make MetaReviewer a one-stop shop for evidence synthesists--an application where users can seamlessly transition from one phase of their project to the next., from literature search through analysis. To make this dream a reality, we built a conversion tool that helps teams automatically generate study IDs and individual study pages for each eligible abstract and populate the studies portal. For those connecting the dots: this means you will no longer need to generate study IDs prior to uploading citations to MetaReviewer!
Our conversion tool also allows users to transfer any notes they took on an eligible abstract to its corresponding individual study page. Gone are the days of trying to piece together why a study was deemed eligible--the information will be right at your fingertips (so long as your screeners documented it).
Start screening
While we could go on and on about this new feature, the best way to learn it is to check it out for yourself! We hope you take a moment to set up a new project, upload some citations, and get a feel for abstract screening in MetaReviewer. Don't forget to check out the User Guide and, as always, reach out to metareviewer@air.org with any questions you have about this new tool.
