| 1 |
--- |
| 2 |
title: "Meta-analysis with cluster-robust variance estimation" |
| 3 |
author: "James E. Pustejovsky" |
| 4 |
date: "`r Sys.Date()`" |
| 5 |
output: rmarkdown::html_vignette |
| 6 |
vignette: > |
| 7 |
%\VignetteIndexEntry{Meta-analysis with cluster-robust variance estimation} |
| 8 |
%\VignetteEngine{knitr::rmarkdown} |
| 9 |
%\VignetteEncoding{UTF-8} |
| 10 |
--- |
| 11 |
|
| 12 |
```{r, echo = FALSE, results = "asis", message = FALSE, warning = FALSE} |
| 13 |
robu_available <- requireNamespace("robumeta", quietly = TRUE) |
| 14 |
meta_available <- requireNamespace("metafor", quietly = TRUE) |
| 15 |
|
| 16 |
knitr::opts_chunk$set(eval = robu_available & meta_available) |
| 17 |
|
| 18 |
if (!robu_available) cat("## Building this vignette requires the robumeta package. Please install it. {-} \n") |
| 19 |
if (!meta_available) cat("## Building this vignette requires the metafor package. Please install it. {-} \n") |
| 20 |
``` |
| 21 |
|
| 22 |
This vignette demonstrates how to use the `clubSandwich` package to conduct a meta-analysis of dependent effect sizes with robust variance estimation. |
| 23 |
Tests of meta-regression coefficients and F-tests of multiple-coefficient hypotheses are calculated using small-sample corrections proposed by Tipton (2015) and Tipton and Pustejovsky (2015). |
| 24 |
The example uses a dataset of effect sizes from a Campbell Collaboration systematic review of dropout prevention programs, conducted by Sandra Jo Wilson and colleagues (2011). |
| 25 |
|
| 26 |
The original analysis included a meta-regression with covariates that capture methodological, participant, and program characteristics. |
| 27 |
The regression specification used here is similar to Model III from Wilson et al. (2011), but treats the `evaluator_independence` and `implementation_quality` variables as categorical rather than interval-level. |
| 28 |
Also, the original analysis clustered at the level of the sample (some studies reported results from multiple samples), whereas here we cluster at the study level. |
| 29 |
The meta-regression can be fit in several different ways. We first demonstrate using the `robumeta` package (Fisher & Tipton, 2015) and then using the `metafor` package (Viechtbauer, 2010). |
| 30 |
|
| 31 |
## robumeta model |
| 32 |
|
| 33 |
```{r, include=FALSE} |
| 34 |
options(width = 100) |
| 35 |
``` |
| 36 |
|
| 37 |
```{r, message = FALSE} |
| 38 |
library(clubSandwich) |
| 39 |
library(robumeta) |
| 40 |
data(dropoutPrevention) |
| 41 |
|
| 42 |
# clean formatting |
| 43 |
names(dropoutPrevention)[7:8] <- c("eval","implement") |
| 44 |
levels(dropoutPrevention$eval) <- c("independent","indirect","planning","delivery") |
| 45 |
levels(dropoutPrevention$implement) <- c("low","medium","high") |
| 46 |
levels(dropoutPrevention$program_site) <- c("community","mixed","classroom","school") |
| 47 |
levels(dropoutPrevention$study_design) <- c("matched","unmatched","RCT") |
| 48 |
levels(dropoutPrevention$adjusted) <- c("no","yes") |
| 49 |
|
| 50 |
m3_robu <- robu(LOR1 ~ study_design + attrition + group_equivalence + adjusted |
| 51 |
+ outcome + eval + male_pct + white_pct + average_age |
| 52 |
+ implement + program_site + duration + service_hrs, |
| 53 |
data = dropoutPrevention, studynum = studyID, var.eff.size = varLOR, |
| 54 |
modelweights = "HIER") |
| 55 |
print(m3_robu) |
| 56 |
``` |
| 57 |
|
| 58 |
Note that `robumeta` produces small-sample corrected standard errors and t-tests, and so there is no need to repeat those calculations with `clubSandwich`. |
| 59 |
The `eval` variable has four levels, and it might be of interest to test whether the average program effects differ by the degree of evaluator independence. |
| 60 |
The null hypothesis in this case is that the 10th, 11th, and 12th regression coefficients are all equal to zero. |
| 61 |
A small-sample adjusted F-test for this hypothesis can be obtained as follows. |
| 62 |
The `vcov = "CR2"` option means that the standard errors will be corrected using the bias-reduced linearization estimator described in Tipton and Pustejovsky (2015). |
| 63 |
|
| 64 |
```{r} |
| 65 |
Wald_test(m3_robu, constraints = constrain_zero(10:12), vcov = "CR2") |
| 66 |
``` |
| 67 |
|
| 68 |
By default, the `Wald_test` function provides an F-type test with degrees of freedom estimated using the approximate Hotelling's $T^2_Z$ method. The test has less than 17 degrees of freedom, even though there are 152 independent studies in the data, and has a p-value that is not quite significant at conventional levels. The low degrees of freedom are a consequence of the fact that one of the levels of `evaluator independence` has only a few effect sizes in it: |
| 69 |
|
| 70 |
```{r} |
| 71 |
table(dropoutPrevention$eval) |
| 72 |
``` |
| 73 |
|
| 74 |
## metafor model |
| 75 |
|
| 76 |
`clubSandwich` also works with models fit using the `metafor` package. Here we re-fit the same regression specification, but use REML to estimate the variance components (`robumeta` uses a method-of-moments estimator), as well as a somewhat different weighting scheme than that used in `robumeta`. |
| 77 |
|
| 78 |
```{r, message = FALSE} |
| 79 |
library(metafor) |
| 80 |
m3_metafor <- rma.mv(LOR1 ~ study_design + attrition + group_equivalence + adjusted |
| 81 |
+ outcome + eval |
| 82 |
+ male_pct + white_pct + average_age |
| 83 |
+ implement + program_site + duration + service_hrs, |
| 84 |
V = varLOR, random = list(~ 1 | studyID, ~ 1 | studySample), |
| 85 |
data = dropoutPrevention) |
| 86 |
summary(m3_metafor) |
| 87 |
``` |
| 88 |
|
| 89 |
`metafor` produces model-based standard errors, t-tests, and confidence intervals. The `coef_test` function from `clubSandwich` will calculate robust standard errors and robust t-tests for each of the coefficients: |
| 90 |
|
| 91 |
```{r} |
| 92 |
coef_test(m3_metafor, vcov = "CR2") |
| 93 |
``` |
| 94 |
|
| 95 |
Note that `coef_test` assumed that it should cluster based on `studyID`, which is the outer-most random effect in the metafor model. This can be specified explicitly by including the option `cluster = dropoutPrevention$studyID` in the call. |
| 96 |
|
| 97 |
The F-test for degree of evaluator independence uses the same syntax as before: |
| 98 |
```{r} |
| 99 |
Wald_test(m3_metafor, constraints = constrain_zero(10:12), vcov = "CR2") |
| 100 |
``` |
| 101 |
|
| 102 |
Despite some differences in weighting schemes, the p-value is very close to the result obtained using `robumeta`. |
| 103 |
|
| 104 |
## References |
| 105 |
|
| 106 |
Fisher, Z., & Tipton, E. (2015). robumeta: An R-package for robust variance estimation in meta-analysis. [arXiv:1503.02220](https://arxiv.org/abs/1503.02220) |
| 107 |
|
| 108 |
Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. _Psychological Methods, 20_(3), 375-393. https://doi.org/10.1037/met0000011 |
| 109 |
|
| 110 |
Tipton, E., & Pustejovsky, J. E. (2015). Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. _Journal of Educational and Behavioral Statistics, 40_(6), 604-634. https://doi.org/10.3102/1076998615606099 |
| 111 |
|
| 112 |
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. _Journal of Statistical Software, 36_(3), 1-48. URL: https://doi.org/10.18637/jss.v036.i03 |
| 113 |
|
| 114 |
Wilson, S. J., Lipsey, M. W., Tanner-Smith, E., Huang, C. H., & Steinka-Fry, K. T. (2011). Dropout prevention and intervention programs: Effects on school completion and dropout Among school-aged children and youth: A systematic review. _Campbell Systematic Reviews, 7_(1), 1-61. https://doi.org/10.4073/csr.2011.8 |