Pre-K, Charters: Peas in Evidentiary Pod?
Posted: February 20th, 2013 | Author: Michael Goldstein | | 6 Comments »
1. Gina D
My colleague Gina D and I exchanged some emails on pre-K. She’d read about a famous program, called Perry PreSchool Project (pdf). She wondered if I knew about it.
I did. This study gets cited over and over. Why? One reason is that the research design was “random assignment,” which is the gold standard. Since there aren’t a ton of these experiments, those that exists get a lot of play.
Result?
It found evidence of preschool program effects on children’s readiness for school and their subsequent educational success, economic success in early adulthood, and reduced number of criminal arrests throughout their lives.
Thing is, this was a boutique program. 123 kids. And it was done a long time ago. Launched in 1962! Older than me. That is saying something.
2. State of The Union
Pre-K has been a hot topic these past few weeks. President Obama:
Let’s make it a national priority to give every child access to a high-quality early education.
If you’re interested in this topic, you should read Sara Mead. She blogs at Edweek.
3. Pre-K evidence and charter evidence
Let’s ponder the movement to expand access to pre-K, and the movement to expand charter schools.
Interesting, the worlds rarely overlap. In fact, often the folks who push hard for greater pre-K oppose charters. And while it’s not exactly parallel, some of the folks who push hard for more charters are skeptical of pre-K.
I am not saying the evidence base here is identical. But I do think the topography is similar.
a. Some small samples, random assignment, with great outcomes.
b. Some medium sized implementations with good outcomes.
c. Overall, however, the average pre-K program, and the average charter, do not seem to be game-changers for kids. In Boston, yes, game-changer. In Massachusetts, yes, high-poverty charters are game-changers. In USA as a whole across the 6300 charters, I don’t think so, although we can’t say for sure.
* * *
For more on pre-K evidence, see here.
Tomorrow I’ll examine 2 good analysts who have recently examined charter school evidence.
PREVIEW:
The short version is there are 10 gold-standard studies of charters that show good things and zero that show bad things, and those studies are associated with many more actual children than the gold-standard pre-K studies (I know of two, but there may be more). Also there are some randomized studies of pre-K that don’t have good outcomes.
So while I do think the charter evidence is stronger than the pre-K evidence on the whole, the evidence at “large scale” for charters is not compelling, or at least not yet.
Now: with 10-0 score on the gold-standard studies, why wouldn’t we conclude the average charter is indeed a game changer for the kiddos? We’ll dig in manana.
* * *
4. Bonus Thought Experiment About Reasoned Discourse
What would happen if …
a. You grabbed 20+ studies of charters and pre-K.
b. You stripped out any mention of those words.
c. You simply called them “Intervention 1, 2, 3, 4″ etc.
d. You changed the data sets a little so you couldn’t tell which study was which, even if you knew the studies off top of your head.
and….
e. You got gathered all the big name education advocates, with fiercely competing beliefs, to sit in a room and consider the evidence?
Here is what I think would happen.
Polarity and stridency would go way down.
If you’re a big pre-K advocate and hate charters, if someone holds up a piece of evidence and says “X worked in a limited context,” you’re not going to want to offer a strong opinion.
If you say “See, X works?” then you might have just endorsed charters by accident.
If you say “X is a meaningless study” you might have just undermined your own case for pre-K. And vice-versa.
I’ve often wanted to do the same “blind discussions” where video is excerpted from real life classrooms from all types of schools, in such a way you can’t say what type of school it is, you can only see a real teacher and real kids.
Again, I predict polarity and stridency would go way down.
If a reform advocate says “Wow, that room is amazing,” well, it might be teaching from a school labeled “failing” and about to be shut down. If someone says “That class is militaristic and teaching to the test,” and it turns out to be a class led by a prominent union leader in a traditional school, well that might not bode so well for some edu-pundits.
Mike Petrilli, I’m handing you the ball to organize this and get everyone to the table. You’ve got time. Just watch less Caillou.
I find your advocacy of tutoring to be more convincing than your advocacy for charters in general, and tutoring is a much better parallel to pre-school. For tutoring and pre-school, you’re comparing to no treatment (right?) and thus if it works, you can look at the effect and cost, and decide yes or no for applying the treatment.
With charters vs. district schools, it is much less clear what the actual treatment is, what its cost really is, etc. We know that for each of the factors attributed to high performing charters in isolation there aren’t dramatic effects in most traditional schools — except perhaps for high-intensity tutoring.
And of course there are various structural advantages (and disadvantages, perhaps) in enrollment, retention, etc. attributed to charters which by definition can’t be applied to all schools, the effect of which has not been quantified.
As to your bonus thought experiment, I’d add another: What if we discount any evidence that an expert would not apply to their own children? Has a single person with the money and motivation withdrawn their own child from pre-school based on research showing its relative ineffectiveness? What about among the people citing this research?
How many charter advocates would hesitate to send their children to a high quality district school in their neighborhood? Or vice versa?
(And I would note that my hostility to the current reform is largely driven by its systematic *destruction* of high quality public schools in my high-poverty neighborhood and failure to improve the low quality ones. I didn’t have a problem with charters 10 years ago.)
Mike,
I think it’s important to not get too bedazzled by the whole “gold standard” idea. I completely agree that some studies are more rigorous than others, and one should more heavily weigh the results of well designed studies than those of others with inherent limitations. But some of the so-called RCTs in education (such as lottery based charter school studies) aren’t quite equivalent to the RCTs in, say, drug trials. My former Rutgers colleague, Bruce Baker, makes some interesting posts about this here:
http://schoolfinance101.wordpress.com/2012/12/20/thoughts-on-randomized-vs-randomized-charter-school-studies/
I think he still considers “lottery-based” studies to be rigorous, but points out that they are only randomized at one point along a complicated sequence of events. Where I sort of disagree with him is that he worries about conflating peer effects and setting effects with treatment effects, but I think peer effects are often a powerful part of the treatment. Schooling is not just about instruction. I think many charter schools consciously try to leverage peer effects through culture building, etc., to drive things like motivation, effort, and aspirations.
And I know you read Shanker Blog, so you’ve probably seen Matt DiCarlo’s recent post here:
http://shankerblog.org/?p=7687
Also, although the initial design of studies is an important factor in weighing the findings, it’s also important to consider the fidelity of implementation of the treatment, the quality of the research implementation, and participant attrition patterns. These can often weaken the findings and post threats to the inferences drawn.
I agree the Perry study is rather limited in terms of sample size, but what it has above the charter school studies to date is the long term longitudinal data on a host of academic and life outcomes. That’s been the power of that study, and it’s ability to generate a cost/benefit estimate. The downside of course, is it was a program delivered almost 40 plus years ago, in a very different educational and societal context.
Tom, good stuff.
Yes, I think you’re right that tutoring is a better parallel in comparing against no treatment. I suggested charters and pre-K, though, because I don’t know that tutoring is really much part of the policy discussion right now. Wish it were.
Although I suppose no pre-K doesn’t mean an unattended kid, it just means some “other” — grandma is watching the kid, for example.
Yep, I don’t begrudge you the hostility to charters — if I lived in your neighborhood in RI, I might have exactly same reaction. Here in Boston, I think things have played out differently over the past 10 years.
Ed, yep, I was going to put up DiCarlo’s blog tomorrow.
Yep, on Perry notes.
And I’d add Winship’s recent paper on RCT cautions too. It’s possible to be the best in class and still have limitations.
I think your point about peer effects is very sharp and you should write about it if you haven’t already. Guest blog here if you don’t have time for a scholarly paper with the word “endogenous.”
Thanks for the comment. Hope all is well.
http://www.brookings.edu/experts/whitehurstg
4 must read blogs on PK, first starting Jan 16 2013, most recent today…
One has to understand that even a great PK program will fall short if the follow through in K and early elementary is not there…
Vocabulary is more than words; it is also knowledge… it needs to be compounded…
Looking forward to more “peer effect” discussion.
The most nuanced take I’ve read:
“How much of KIPP’s impacts on student achievement might be explained by the fact that KIPP tends to attract higher-achieving replacement students in the upper grades? Our findings confirm suggestions by Kahlenberg and Miron that selective replacement of departing KIPP students produces a gradual improvement over the course of middle school in the pre-KIPP achievement levels of the KIPP student population. But it is difficult to ascertain the magnitude of any resulting peer effects at KIPP schools. One way to extrapolate the possible size of peer effects at KIPP is to combine our findings with other research on how peers’ prior scores affect student achievement. Although we employ this method in the following paragraphs, this should be viewed as only a rough approximation of peer effects at KIPP schools.”
They go on to suggest (and tepidly so) that peer effects at KIPP schools do not account for the large cumulative impacts on student achievement found by other researchers (e.g. that Angrist et al. KIPP Lynn paper).
Full paper:
http://www.mathematica-mpr.com/publications/pdfs/education/KIPP_middle_schools_wp.pdf
Kane et al. approach it differently and find something surprising:
“Contrary to the usual view of high-achieving peers, (our evidence) shows that the score gain from charter middle school attendance varies inversely with peer means. For example, students who apply to charter schools in a risk set with peer means 0.1σ be- low the sample mean are estimated to have an ELA gain that is roughly 0.08σ higher, and a math gain about 0.1σ higher, for each year spent in a charter.”
Full paper:
http://qje.oxfordjournals.org/content/126/2/699.full.pdf