Debian Codegraph @

vignettes/Wald-tests-in-clubSandwich.Rmd

raw
1 ---
2 title: "Wald tests of multiple-constraint null hypotheses"
3 author: "James E. Pustejovsky"
4 date: "`r Sys.Date()`"
5 output:
6 rmarkdown::html_vignette:
7 number_sections: true
8 toc: true
9 bibliography: bibliography.bib
10 csl: apa.csl
11 vignette: >
12 %\VignetteIndexEntry{Wald tests of multiple-constraint null hypotheses}
13 %\VignetteEngine{knitr::rmarkdown}
14 %\VignetteEncoding{UTF-8}
15 ---
16
17 ```{r, echo = FALSE, results = "asis", message = FALSE, warning = FALSE}
18 library(clubSandwich)
19 AER_available <- requireNamespace("AER", quietly = TRUE)
20
21 knitr::opts_chunk$set(eval = AER_available)
22
23 if (!AER_available) cat("# Building this vignette requires the AER package. Please install it. {-}")
24 ```
25
26 Version 0.5.0 of `clubSandwich` introduced a new syntax for `Wald_test()`, a function for conducting tests of multiple-constraint hypotheses. In previous versions, this function was poorly documented and, consequently, probably little used. This vignette will demonstrate the new syntax.
27
28 For purposes of illustration, I will use the `STAR` data (available in the `AER` package), which is drawn from a randomized trial evaluating the effects of elementary school class size on student achievement. The data consist of individual-level measures for students in each of several dozen schools. For purposes of illustration, I will look at effects on math performance in first grade. Treatment conditions (the variable called `stark`) were assigned at the classroom level, and consisted of either a) a regular-size class, b) a small-size class, or c) a regular-size class but with the addition of a teacher's aide. In all of what follows, I will cluster standard errors by school in order to allow for generalization to a super-population of schools.
29
30 ```{r, message = FALSE, warning = FALSE}
31 library(clubSandwich)
32
33 data(STAR, package = "AER")
34
35 # clean up a few variables
36 levels(STAR$stark)[3] <- "aide"
37 levels(STAR$schoolk)[1] <- "urban"
38 STAR <- subset(STAR,
39 !is.na(schoolidk),
40 select = c(schoolidk, schoolk, stark, gender, ethnicity, math1, lunchk))
41 head(STAR)
42 ```
43
44 # The Wald test function
45
46 The `Wald_test()` function can be used to conduct hypothesis tests that involve multiple constraints on the regression coefficients. Consider a linear model for an outcome $Y_{ij}$ regressed on a $1 \times p$ row vector of predictors $\mathbf{x}_{ij}$ (which might include a constant intercept term):
47 $$
48 Y_{ij} = \mathbf{x}_{ij} \boldsymbol\beta + \epsilon_{ij}
49 $$
50 The regression coefficient vector is $\boldsymbol\beta$. In quite general terms, a set of constraints on the regression coefficient vector can be expressed in terms of a $q \times p$ matrix $\mathbf{C}$, where each row of $\mathbf{C}$ corresponds to one constraint. A joint null hypothesis is then $H_0: \mathbf{C} \boldsymbol\beta = \mathbf{0}$, where $\mathbf{0}$ is a $q \times 1$ vector of zeros.[^more-general]
51
52 [^more-general]: In @pustejovsky2018small we used a more general formulation of multiple-constraint null hypotheses, expressed as $H_0: \mathbf{C} \boldsymbol\beta = \mathbf{d}$ for some fixed $q \times 1$ vector $\mathbf{d}$. In practice, it's often possible to modify the $\mathbf{C}$ matrix so that $\mathbf{d}$ can always be set to $\mathbf{0}$.
53
54 Wald-type test are based on the test statistic
55 $$
56 Q = \left(\mathbf{C}\boldsymbol{\hat\beta}\right)' \left(\mathbf{C} \mathbf{V}^{CR} \mathbf{C}'\right)^{-1} \left(\mathbf{C}\boldsymbol{\hat\beta}\right),
57 $$
58 where $\boldsymbol{\hat\beta}$ is the estimated regression coefficient vector and $\mathbf{V}^{CR}$ is a cluster-robust variance matrix. If the number of clusters is sufficiently large, then the distribution of $Q$ under the null hypothesis is approximately $\chi^2(q)$. @tipton2015small investigated a wide range of other approximations to the null distribution of $Q$, many of which are included as options in `Wald_test()`. Based on a large simulation, they (...er...we...) recommended a method called the "approximate Hotelling's $T^2$-Z" test, or "AHZ." This test approximates the distribution of $Q / q$ by a $T^2$ distribution, which is a multiple of an $F$ distribution, with numerator degrees of freedom $q$ and denominator degrees of freedom based on a generalization of the Satterthwaite approximation.
59
60 The `Wald_test()` function has three main arguments:
61 ```{r}
62 args(Wald_test)
63 ```
64
65 * The `obj` argument is used to specify the estimated regression model on which to perform the test,
66 * the `constraints` argument is a $\mathbf{C}$ matrix expressing the set of constraints to test, and
67 * the `vcov` argument is a cluster-robust variance matrix, which is used to construct the test statistic. (Alternately, `vcov` can be the type of cluster-robust variance matrix to construct, in which case it will be computed internally.)
68
69 By default, `Wald_test()` will use the HTZ small-sample approximation. Other options are available (via the `test` argument) but not recommended for routine use. The optional `tidy` argument will be demonstrated below.
70
71 ## Testing treatment effects
72
73 Returning to the STAR data, let's suppose we want to examine differences in math performance across class sizes. This can be done with a simple linear regression model, while clustering the standard errors by `schoolidk`. The estimating equation is
74 $$
75 \left(\text{Math}\right)_{ij} = \beta_0 + \beta_1 \left(\text{small}\right)_{ij} + \beta_2 \left(\text{aide}\right)_{ij} + e_{ij},
76 $$
77 which can be estimated in R as follows:
78 ```{r type-treat}
79
80 lm_trt <- lm(math1 ~ stark, data = STAR)
81 V_trt <- vcovCR(lm_trt, cluster = STAR$schoolidk, type = "CR2")
82 coef_test(lm_trt, vcov = V_trt)
83
84 ```
85
86 In this estimating equation, the coefficients $\beta_1$ and $\beta_2$ represent treatment effects, or differences in average math scores relative to the reference level of `stark`, which in this case is a regular-size class. The t-statistics and p-values reported by `coef_test` are separate tests of the null hypotheses that each of these coefficients are equal to zero, meaning that there is no difference between the specified treatment condition and the reference level. We might want to instead test the _joint_ null hypothesis that there are no differences among _any_ of the conditions. This null can be expressed by a set of multiple constraints on the parameters: $\beta_1 = 0$ and $\beta_2 = 0$.
87
88 To test the null hypothesis that $\beta_1 = \beta_2 = 0$ based on the treatment effects model specification, we can use:
89 ```{r}
90 C_trt <- matrix(c(0,0,1,0,0,1), 2, 3)
91 C_trt
92 Wald_test(lm_trt, constraints = C_trt, vcov = V_trt)
93 ```
94 The result includes details about the form of `test` computed, the $F$-statistic, the numerator and denominator degrees of freedom used to compute the reference distribution, and the $p$-value corresponding to the specified null hypothesis. In this example, $p = `r if (AER_available) format.pval(Wald_test(lm_trt, constraints = C_trt, vcov = V_trt)$p_val, digits = 3)`$, so we can rule out the null hypothesis that there are no differences in math performance across conditions.
95
96 The representation of null hypotheses as arbitrary constraint matrices is useful for developing theory about how to test such hypotheses, but it is not all that helpful for actually running tests---constructing constraint matrices "by hand" is just too cumbersome of an exercise. Moreover, $\mathbf{C}$ matrices typically follow one of a small number of patterns. Two common use cases are a) constraining a set of $q > 1$ parameters to all be equal to zero and b) constraining a set of $q + 1$ parameters to be equal to a common value. The `clubSandwich` package now includes a set of helper functions to create constraint matrices for these common use cases.
97
98 ## `constrain_zero()`
99
100 To constrain a set of $q$ regression coefficients to all be equal to zero, the simplest form of the $\mathbf{C}$ matrix would consist of a set of $q$ rows, where a single entry in each row would be equal to 1 and the remaining entries would all be zero. For the `lm_trt` model, the C matrix would look like this:
101 $$
102 \mathbf{C} = \left[\begin{array}{ccc} 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right],
103 $$
104 so that
105 $$
106 \mathbf{C}\boldsymbol\beta = \left[\begin{array}{ccc} 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right] \left[\begin{array}{c} \beta_0 \\ \beta_1 \\ \beta_2 \end{array} \right] = \left[\begin{array}{c} \beta_1 \\ \beta_2 \end{array} \right],
107 $$
108 which is set equal to $\left[\begin{array}{c} 0 \\ 0 \end{array} \right]$.
109
110 The `constrain_zero()` function will create matrices like this automatically. The function takes two main arguments:
111 ```{r}
112 args(constrain_zero)
113 ```
114 * The `constraints` argument is used to specify _which_ coefficients in a regression model to set equal to zero.
115 * The `coefs` argument is the set of estimated regression coefficients, for which to calculate the constraints.
116
117 Constraints can be specified by position index, by name, or via a regular expression. To test the joint null hypothesis that average math performance is equal across the three treatment conditions, we need to constrain the second and third coefficients to zero:
118 ```{r}
119 constrain_zero(2:3, coefs = coef(lm_trt))
120 ```
121 Or equivalently:
122 ```{r}
123 constrain_zero(c("starksmall","starkaide"), coefs = coef(lm_trt))
124 ```
125 or
126 ```{r}
127 constrain_zero("^stark", coefs = coef(lm_trt), reg_ex = TRUE)
128 ```
129 Note that if `constraints` is a regular expression, then the `reg_ex` argument needs to be set to `TRUE`.
130
131 The result of `constrain_zero()` can then be fed into the `Wald_test()` function:
132 ```{r}
133 C_trt <- constrain_zero(2:3, coefs = coef(lm_trt))
134 Wald_test(lm_trt, constraints = C_trt, vcov = V_trt)
135 ```
136 To reduce redundancy in the syntax, we can also omit the `coefs` argument to `constrain_zero`, so long as we call it inside of `Wald_test`[^under-the-hood]:
137 ```{r}
138 Wald_test(lm_trt, constraints = constrain_zero(2:3), vcov = V_trt)
139 ```
140
141 [^under-the-hood]: How does this work? If we omit the `coefs` argument, `constrain_zero()` acts as a functional, by returning a function equivalent to `function(coefs) constrain_zero(constraints, coefs = coefs)`. If this function is fed into the `constraints` argument of `Wald_test()`, `Wald_test()` recognizes that it is a function and evaluates the function with `coef(obj)`. It's a kinda-sorta hacky substitute for lazy evaluation. If you have suggestions for how to do this more elegantly, please send them my way.
142
143 ## `constrain_equal()`
144
145 Another common type of constraints involve setting a set of $q + 1$ regression coefficients to be all equal to a common (but unknown) value ($q + 1$ because it takes $q$ constraints to do this). There are many equivalent ways to express such a set of constraints in terms of a $\mathbf{C}$ matrix. One fairly simple form consists of a set of $q$ rows, where the entry corresponding to one of the coefficients of interest is equal to -1 and the entry corresponding to another coefficient of interest is equal to 1.
146
147 To see how this works, let's look at a different way of parameterizing our simple model for the STAR data, by using separate intercepts for each treatment condition. The estimating equation would then be
148 $$
149 \left(\text{Math}\right)_{ij} = \beta_0 \left(\text{regular}\right)_{ij} + \beta_1 \left(\text{small}\right)_{ij} + \beta_2 \left(\text{aide}\right)_{ij} + e_{ij}.
150 $$
151 This model can be estimated in R by dropping the intercept term:
152 ```{r type-sep}
153
154 lm_sep <- lm(math1 ~ 0 + stark, data = STAR)
155 V_sep <- vcovCR(lm_sep, cluster = STAR$schoolidk, type = "CR2")
156 coef_test(lm_sep, vcov = V_sep)
157
158 ```
159 In this parameterization, the coefficients $\beta_0$, $\beta_1$, and $\beta_2$ represent the average math performance levels of students in each of the treatment conditions. The t-tests and p-values now have a very different interpretation because they pertain to the null hypothesis that the average performance level for a given condition is equal to zero. With this separate-intercepts model, the joint null hypothesis that performance levels are equal across conditions amounts to constraining the intercepts to be equal to each other: $\beta_0 = \beta_1$ and $\beta_0 = \beta_2$ (note that we don't need the constraint $\beta_1 = \beta_2$ because it is implied by the first two).
160
161 For the `lm_sep` model, which has separate intercepts $\beta_0$, $\beta_1$, and $\beta_2$, the C matrix would look like this:
162 $$
163 \mathbf{C} = \left[\begin{array}{ccc} -1 & 1 & 0 \\ -1 & 0 & 1 \end{array} \right],
164 $$
165 so that
166 $$
167 \mathbf{C}\boldsymbol\beta = \left[\begin{array}{ccc} -1 & 1 & 0 \\ -1 & 0 & 1 \end{array} \right] \left[\begin{array}{c} \beta_0 \\ \beta_1 \\ \beta_2 \end{array} \right] = \left[\begin{array}{c} \beta_1 - \beta_0 \\ \beta_2 - \beta_0 \end{array} \right],
168 $$
169 which is set equal to $\left[\begin{array}{c} 0 \\ 0 \end{array} \right]$.
170
171 The `constrain_equal()` function will create matrices like this automatically, given a set of coefficients to constrain. The syntax is identical to `constrain_zero()`:
172 ```{r}
173 args(constrain_equal)
174 ```
175 To test the joint null hypothesis that average math performance is equal across the three treatment conditions, we can constrain all three coefficients of `lm_sep` to be equal:
176 ```{r}
177 constrain_equal(1:3, coefs = coef(lm_sep))
178 ```
179 Or equivalently:
180 ```{r}
181 constrain_equal(c("starkregular","starksmall","starkaide"), coefs = coef(lm_sep))
182 ```
183 or
184 ```{r}
185 constrain_equal("^stark", coefs = coef(lm_sep), reg_ex = TRUE)
186 ```
187 Just as with `constrain_zero`, if `constraints` is a regular expression, then the `reg_ex` argument needs to be set to `TRUE`.
188
189 This constraint matrix can then be fed into `Wald_test()`:
190 ```{r}
191 C_sep <- constrain_equal("^stark", coefs = coef(lm_sep), reg_ex = TRUE)
192 Wald_test(lm_sep, constraints = C_sep, vcov = V_sep)
193 ```
194 or equivalently:
195 ```{r}
196 Wald_test(lm_sep, constraints = constrain_equal(1:3), vcov = V_sep)
197 ```
198 Note that these test results are exactly equal to the tests based on `lm_trt` with `constrain_zero()`. They're algebraically equivalent---just different ways of parameterizing the same model and constraints.
199
200 # Testing an interaction
201
202 Let's now consider how these functions can be applied in a more complex model. Suppose that we are interested in understanding whether the effect of being in a small class is consistent across schools in different areas, where areas are categorized as urban, suburban, or rural. To answer this question, we need to test for an interaction between urbanicity and treatment condition. One estimating equation that would let us examine this question is:
203 $$
204 \begin{aligned}
205 \left(\text{Math}\right)_{ij} &= \beta_0 + \beta_1 \left(\text{suburban}\right)_{ij} + \beta_2 \left(\text{rural}\right)_{ij} \\
206 & \quad + \beta_3 \left(\text{small}\right)_{ij} + \beta_4 \left(\text{aide}\right)_{ij} \\
207 & \quad\quad + \beta_5 \left(\text{small}\right)(\text{suburban})_{ij} + \beta_6 \left(\text{aide}\right)(\text{suburban})_{ij} \\
208 & \quad\quad\quad + \beta_{7} \left(\text{small}\right)(\text{rural})_{ij} + \beta_{8} \left(\text{aide}\right)(\text{rural})_{ij} \\
209 & \quad\quad\quad\quad + \mathbf{x}_{ij} \boldsymbol\gamma + e_{ij},
210 \end{aligned}
211 $$
212 where $\mathbf{x}_{ij}$ is a row vector of student characteristics, included just to make the regression look fancier. In this specification, $\beta_3$ and $\beta_4$ represent the effects of being in a small class or aide class, compared to being in a regular class, but _only for the reference level of urbanicity_---in this case, urban schools. The coefficients $\beta_5, \beta_6, \beta_7, \beta_8$ all represent _interactions_ between treatment condition and urbanicity. Here's the model, estimated in R:
213 ```{r}
214 lm_urbanicity <- lm(math1 ~ schoolk * stark + gender + ethnicity + lunchk, data = STAR)
215 V_urbanicity <- vcovCR(lm_urbanicity, cluster = STAR$schoolidk, type = "CR2")
216 coef_test(lm_urbanicity, vcov = V_urbanicity)
217 ```
218 With this specification, there are several different null hypotheses that we might want to test. For one, perhaps we want to see if there is _any_ variation in treatment effects across different levels of urbanicity. This can be expressed in the null hypothesis that all four interaction terms are zero, or $H_{0A}: \beta_5 = \beta_6 = \beta_7 = \beta_8 = 0$. With Wald test:
219 ```{r}
220 Wald_test(lm_urbanicity,
221 constraints = constrain_zero("schoolk.+:stark", reg_ex = TRUE),
222 vcov = V_urbanicity)
223 ```
224
225 Another possibility is that we might want to focus on variation in the effect of being in a small class or regular class, while ignoring whatever is going on in the aide class condition. Here, the null hypothesis would be simply $H_{0B}: \beta_5 = \beta_6 = 0$, tested as:
226 ```{r}
227 Wald_test(lm_urbanicity,
228 constraints = constrain_zero("schoolk.+:starksmall", reg_ex = TRUE),
229 vcov = V_urbanicity)
230 ```
231
232 ## Lists of constraints
233
234 In models like the urbanicity-by-treatment interaction specification, we may need to run multiple tests on the same estimating equation. This can be accomplished with `Wald_test` by providing a _list_ of constraints to the `constraints` argument. For example, we could test the hypotheses described above by creating a list of several constraint matrices and then passing it to `Wald_test`:
235 ```{r}
236 C_list <- list(
237 `Any interaction` = constrain_zero("schoolk.+:stark",
238 coef(lm_urbanicity), reg_ex = TRUE),
239 `Small vs regular` = constrain_zero("schoolk.+:starksmall",
240 coef(lm_urbanicity), reg_ex = TRUE)
241 )
242
243 Wald_test(lm_urbanicity,
244 constraints = C_list,
245 vcov = V_urbanicity)
246
247 ```
248 Setting the option `tidy = TRUE` will arrange the output of all the tests into a single data frame:
249 ```{r}
250
251 Wald_test(lm_urbanicity,
252 constraints = C_list,
253 vcov = V_urbanicity,
254 tidy = TRUE)
255
256 ```
257 The list of constraints can also be created inside `Wald_test`, so that the `coefs` argument can be omitted from `constrain_zero()`:
258 ```{r}
259
260 Wald_test(
261 lm_urbanicity,
262 constraints = list(
263 `Any interaction` = constrain_zero("schoolk.+:stark", reg_ex = TRUE),
264 `Small vs regular` = constrain_zero("schoolk.+:starksmall", reg_ex = TRUE)
265 ),
266 vcov = V_urbanicity,
267 tidy = TRUE
268 )
269
270 ```
271
272 # Pairwise t-tests
273
274 The `clubSandwich` package also provides a further convenience function, `constrain_pairwise()` that can be used in combination with `Wald_test()` to conduct pairwise comparisons among a set of regression coefficients. This function differs from the other two `constrain_*()` functions because it returns a _list_ of constraint matrices, each of which corresponds to a single linear combination of covariates. Specifically, the `constrain_pairwise()` function provides a list of constraints that represent the differences between every possible pair among a specified set of coefficients. The syntax is very similar to the other `constrain_*()` functions.
275
276 To demonstrate, consider the separate-intercepts specification of the simpler regression model:
277 ```{r}
278 coef_test(lm_sep, vcov = V_sep)
279 ```
280 This specification is nice because it lets us simply read off the average outcomes for each group. However, we will naturally also want to know about whether there are differences between the groups, so we'll want to compare the small-class condition to the regular-size class condition, the aide condition to the regular-size class condition, and the small-class condition to the aide condition. Thus, we'll want comparisons among all three coefficients:
281 ```{r}
282 C_pairs <- constrain_pairwise(1:3, coefs = coef(lm_sep))
283 C_pairs
284 ```
285 Feeding these constraints into `Wald_test()` gives us significance tests for each pair:
286 ```{r}
287 Wald_test(lm_sep, constraints = C_pairs, vcov = V_sep, tidy = TRUE)
288 ```
289
290 The first two of these tests are equivalent to the tests of the treatment effect coefficients in the other parameterization of the model. Indeed, the denominator degrees of freedom are identical to the results of `coef_test(lm_trt, vcov = V_trt)`; the `Fstat`s here are equal to the squared t-statistics from the first model:
291 ```{r}
292 t_stats <- coef_test(lm_trt, vcov = V_trt)$tstat[2:3]
293 F_stats <- Wald_test(lm_sep, constraints = C_pairs, vcov = V_sep, tidy = TRUE)$Fstat[1:2]
294 all.equal(t_stats^2, F_stats)
295 ```
296 It is important to note that the p-values from the pairwise comparisons are _not_ corrected for multiplicity.[^multiplicity] For now, please correct-your-own using `p.adjust()` or your preferred method.
297
298 [^multiplicity]: Options to include multiplicity corrections (Bonferroni, Holm, Benjamini-Hochberg, etc.) might be included in a [future release](https://github.com/jepusto/clubSandwich/issues/33). Reach out if this is of interest to you.
299
300 Pairwise comparisons might also be of use in the model with treatment-by-urbanicity interactions. Here's the model results again:
301 ```{r}
302 coef_test(lm_urbanicity, vcov = V_urbanicity)
303 ```
304 Suppose that we are interested in the effect of small versus regular size classes, and in particular whether this effect varies across schools in different areas. The coefficients on `schoolksuburban:starksmall` and `schoolkrural:starksmall` already give us the differences in treatment effects between suburban schools versus urban schools and between rural schools versus urban schools. The difference between these coefficients gives us the difference in treatment effects between suburban schools and rural schools. We can look at all three of these contrasts using `constrain_pairwise()` by setting the option `with_zero = TRUE`:
305 ```{r}
306 Wald_test(lm_urbanicity,
307 constraints = constrain_pairwise(":starksmall", reg_ex = TRUE, with_zero = TRUE),
308 vcov = V_urbanicity,
309 tidy = TRUE)
310 ```
311 Again, the results of the first two tests are identical to the t-tests reported in `coef_test()`.
312
313 # Remark
314
315 All of the preceding examples were based on ordinary linear regression models with clustered standard errors. However, `Wald_test()` and its helper functions all work identically for all of the other models with supporting `clubSandwich` methods, including `nlme::lme()`, `nlme::gls()`, `lme4::lmer()`, `rma.uni()`, `rma.mv()`, and `robu()`, among others.
316
317 # References