Why a Decision Tree?
Effect sizes are complicated. We tried to explain how we made things a bit easier for users in a previous post on Calculating Effect Sizes in MetaReviewer. But that post assumed that users know, going into MetaReviewer, what effect size type they need to estimate. But what about times where it’s not clear which effect size to estimate?
Our answer is what we call an Effect Size Decision Tree. We suspect that long-time evidence synthesists use a version of this decision tree – consciously or unconsciously – whenever they read over a study. Once you’ve reviewed a few (thousand) studies, the process might become second nature. But until that point, all that data can be downright overwhelming.
The effect size decision tree is meant to help users determine which effect can be estimated, based off the information available.
Notably, the decision tree is less helpful in determining which effect should be estimated, but we’ll get to that at the end.
What are the Primary Decision Points?
Three primary decisions dictate a large part of the decision-making process:
- What type of effect size data do you have?
- What data are available?
- Does the data derive from a clustered design?
Addressing the first question, we focused the current iteration of the Decision Tree on effect sizes that represent continuous data (like from a test or scale score) and the difference between two groups. MetaReviewer will handle other types of effect sizes, but this decision tree doesn’t yet cover those effects. (We will update the Decision Tree to reflect them once we add more effects to the templates!)
The second question asks about data availability. Numerous decisions within the tree ask a version of this question and two questions prove crucial. The first is asked immediately: are standard deviations (SDs) by condition available? Without SDs, primary authors must provide (a) test statistics (like T or F), (b) an effect size calculated within the study, or (c) the missing SDs via an author query (e.g., from an email request).
The third pivotal question is whether data are from a clustered design. This may be a term that is not commonly used in your field, or you may simply be unfamiliar with it. Simply put, clustered data occur when participants are part of some common structure (like a school, neighborhood, or hospital) and those structures are assigned to a condition. (If you are curious why we care about clustered design, we strongly encourage you read about them here or here. Again, simplifying things, the reason we care is because people from the same structures tend to be more like each other compared to people from different structures.)
We differentiated the clustered nature of the data for a practical as well as statistical reason: MetaReviewer currently calculates effects deriving from non-clustered data. In fact, it will estimate more than 15 types of effect sizes from non-clustered data. We represent each of those in the light purple color.
For all clustered data effect size types, MetaReviewer can still help you. However, instead of estimating the effect size within MetaReviewer, it’ll simply identify the right data for estimating the effect. You will need to use an alternative effect size calculator, like the one from MOSAIC here, to estimate the effect. At some point in the coming year, we will update the templates and application to accommodate these designs.
How Should Teams Use the Decision Tree?
We see the Decision Tree’s usefulness to synthesis team falling into two categories: training and ongoing coding.
For new (and sometimes experienced!) synthesists, it’ll be a great way to learn and get on the same page about effect sizes during training sessions. A framework of any sort can help guide conversation and highlight gaps in knowledge. Researchers who are new to evidence synthesis, in particular, will find the structure a helpful foundation when reading a study for the first time.
As for ongoing coding issues, we suspect teams will use it to prioritize certain effects over others. We said at the top that it’s less helpful to teams in helping determine the effect that should be selected. This is because different teams will make different choices on what effect to prioritize, based on their own circumstances. In our own practice, we use the decision tree to guide users to certain effects. For example, we generally prefer effects that have both pretest and posttest data available, over data that are adjusted for pretest. So those effects will likely have their tree ‘branches’ highlighted.
Need More Info on Effect Sizes?
There’s a lot to explore and think about when it comes to effect sizes. So to end, here are some resources to check out. We’ve sorted them into older (but still relevant!) textbooks and more recent updates.
Classic Textbooks
Practical Meta-Analysis. Lipsey & Wilson, 2001. The appendices are filled with effect size conversions and great introductory information. Also check out Dr. Wilson’s effect size calculator, recently updated.
Introduction to Meta-Analysis. Borenstein, Hedges, Higgins, & Rothstein, 2009. A great primer on why we care about effect sizes and other important meta-analysis information.
Methods of Meta-Analysis. Hunter & Schmidt. Looking for more information on correlation effect sizes, especially when it comes to measurement issues like reliability and bias? This absolute classic text is where to find it.
Modern Resources
Doing Meta-Analysis in R. Harrer, Cuijpers, Furukawa, & Ebert. A great primer on estimating many different types of effect sizes in R.
MOSAIC’s Cluster Design Effect Size Calculator. American Institutes for Research. Another great resource for estimating effect sizes from clustered designs.
Converting between effect sizes. Polanin & Snilstveit. A methods articled commissioned by the Campbell Collaboration to articulate basic conversions across common effect size types.
What Works Clearinghouse, Version 5.0 Procedures and Standards Handbook. U.S. Department of Education, Institute of Education Sciences. Appendix E provides a comprehensive look at findings for two groups and is the most current resource on clustered data effects.