Step 2. Sort

Now the fun and your real training as a synthesis jedi begins (hello to 10,000 hours for mastery). Search your returned hits for appropriate studies that satisfy the criteria that you list a priori. Typically, there are a set of simple assumptions for exclusion criteria for a meta-analysis in evidence-based research. Not a review, includes terms and actually studied the process at hand, and not a duplicate. Those are the three most common reasons for exclusion for the first round of exclusion. The second round shifts from abstract/overview review and sorting of studies to detailed evidence extraction. Within this round, the most common reason for exclusion is lack of open data.

Open science product

The key element to summarize for reproducible science is the criteria. The most typical mechanism to capture the sort workflow is a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement/report, and some disciplines have not been as rapid in adopting this synthesis summary tool. A nice description of its importance here.

Case study

A search for reearch on positive interactions and the niche in deserts.
We will continue with the study of niche within desert ecosystems instance and search matrix. 

library(tidyverse)
library(PRISMAstatement)

#PRISMA####
#use 'prisma' function from PRISMAstatement 
#https://cran.r-project.org/web/packages/PRISMAstatement/vignettes/PRISMA.html
prisma.report <-read_csv("data/prisma.csv")
prisma.report
## # A tibble: 53 x 4
##       ID inclusion                          category
##    <int>     <chr>                             <chr>
##  1     1         N no positive interactions recorded
##  2     2         Y                           include
##  3     3         Y                           include
##  4     4         N       no niche expansion recorded
##  5     5         Y                           include
##  6     6         N no positive interactions recorded
##  7     7         N no positive interactions recorded
##  8     8         N no positive interactions recorded
##  9     9         Y                           include
## 10    10         Y                           include
## # ... with 43 more rows, and 1 more variables: detailed.reason <chr>
exclusions <- prisma.report %>% filter(inclusion == "N") %>% select(ID, category)
dim(exclusions)
## [1] 24  2
prisma(found = 156,
       found_other = 0,
       no_dupes = 53, 
       screened = 53, 
       screen_exclusions = 0, 
       full_text = 53,
       full_text_exclusions = 24, 
       qualitative = 0, 
       quantitative = 29,
       width = 800, height = 800)
#Summary of exclusions
categories <- exclusions %>% group_by(category) %>% tally() %>% arrange(desc(n))
categories
## # A tibble: 6 x 2
##                            category     n
##                               <chr> <int>
## 1                    no nurse plant     9
## 2 no positive interactions recorded     9
## 3       no niche expansion recorded     2
## 4                            review     2
## 5                          language     1
## 6                        not desert     1
ggplot(categories, aes(category, n)) + geom_bar(stat = "identity") + coord_flip() + ylim(0,10)


Exercise

Use your previous search and select a reasonable set of studies to explore the importance of general versus specific exclusion criteria on processing primary research literature as a synthesis substrate. As an important consideration here, please also try the second round, even briefly, for those not immediately excluded by adding another vector in your dataframe indicating whether useable data are available/provided within the publication. For the purposes of this sort step, the data can be a list, table, or even figure that shows means, variances, and sample sizes for every treatment level needed in the synthesis.

Conclusions

  1. A PRISMA statement is similar to a decision tree/workflow diagram and a recommended tool to summarize the synthesis process from primary research objects including publication and datasets (although typically publications).
  2. The sort process is best conceptualized as a two-step process.
  3. The first round of exclusions include reviews, terms listed by not studied, and duplicates.
  4. The second round of exclusions is typically associated with full publication review and included the former two criteria from the previous step but now adds in an estimate of likelihood that data are available for detailed synthesis.
  5. If there are two few studies with available data for synthesis (from within each study), typically defined as < 10 independent studies, then the jedi synthesis researcher can elect to shift to systematic review and summarize the research landscape of studies without detailed analysis of strength of evidence from within studies.

Additional resources

Even if you never want to engage in these forms of synthesis, ensure you primary research publications are available to others by embracing the principles of open science and effective scientific reporting.
A step-by-step guide to the meta versus systematic review decision.
The PRISMA flow diagram is required by many journals when submitting a systematic review or meta-analysis. However, several journals including PLOS ONE and PeerJ also require the PRISMA Checklist (available for donload from site) as a supplemental file.