By Ryan McNeely
Mathematica Policy Research just released a preliminary report of their large, multi-year study of the effectiveness of KIPP charter schools on increasing educational attainment. Their finding: “For the vast majority of KIPP schools studied, impacts on students’ state assessment scores in mathematics and reading are positive, statistically significant, and educationally substantial.” Further, “estimated impacts are frequently large enough to substantially reduce race- and income-based achievement gaps within three years of entering KIPP.” Right now, only data related to test scores is available, but the results are striking:
The results for reading are equally impressive. But what about the fact that the study only looked at 22 schools? A nice thing about using propensity score matching as part of a quasi-experimental design is that there are a lot of upfront costs associated with crafting the comparison model, but once the investment is made it can be easily scaled up. Rather than do so immediately, however, the evaluators are going to take advantage of the fact that they can conduct a true randomized experiment — something that is normally difficult to do — using an existing KIPP lottery system. Then, they can use the more robust method to confirm that their model is accurate. Or as the report states, “If the nonexperimental impact estimates match the experimental impact estimates, we will have evidence that the nonexperimental design can produce unbiased impact estimates.” An interim report is due in mid-2012 with the final report due in 2014 after four years of follow-up.
A final note: since all program evaluations require making assumptions that may or may not be valid, it’s important to see in which direction the results would be biased if the assumptions did not hold. Mathematica did this correctly, by basically making assumptions that would tend to understate the positive effects of KIPP schooling. As one example, KIPP schools disproportionally hold students back if they’re not ready to advance to the next grade, which complicates comparison between KIPP and non-KIPP schools across grade levels. The evaluators handled this by assuming “that each retained student does neither better nor worse than before retention. If KIPP in fact has a positive impact on the achievement of grade repeaters, this would cause us to underestimate KIPP’s impact.” This is the sign of an honest evaluation.