More Houston
Posted: September 7th, 2011 | Author: Michael Goldstein | | 7 Comments »
I got some emails and calls yesterday. The New York Times did a story on the Apollo turnaround schools in Houston. Positive story, and very deservedly so, for HISD and EdLabs.
However, I thought the story was framed imprecisely. Not inaccurately, but imprecisely.
I’ve blogged before about the work we did there in Summer 2010: dispatching a delta force named Patti, Cathryn, Tim, Christie, Erica, and Eli. Their job was to recruit, train, and deploy 200+ full-time tutors in just a few weeks. They killed it.
The Times story ascribes Apollo gains to the “copying the charter school model” broadly. But there was, to my knowledge, only one aspect of the experiment thus far which has really cleaned up.
In the Apollo schools, the really big gains were in 6th and 9th grade math only. Those were the 2,500 or so kids who got daily MATCH-style math tutoring.
So far as I’m aware, there were not really big gains in 7th, 8th, 10th grade math, or 6th, 7th, 8th, 9th, 10th grade English. None of those subjects was tutored. (There were some gains, just not biggies).
At the high-performing charter schools, the gains tend to happen roughly across all subjects in all grades, with math gains typically larger. That didn’t happen here.
Now this was only Year 1 of the intervention. MATCH wasn’t tested in our first year of operation (2001), but if we had been, the results would have been terrible. So we’ll learn more this coming year as the Apollo turnaround effort both includes several additional Houston schools, and launches in Denver.
I felt a little bad for our troopers who put in 16-hour days in Houston. Wish they’d gotten recognition. Don’t worry peeps. It happens. When I wrote a few articles, back in the day, for New York Magazine, sometimes I’d simplify a story to get a “cleaner” narrative. Certainly editors press for it.
My only regret is readers may take the wrong policy prescription. It’s not “copy no excuses charters” per se. To copy the no excuses model, you gotta be ALL IN, 100 one-percent solutions. This is not a recipe that works if you “Sort of” follow it.
The right policy conclusion, I’d submit, is that high dosage, 2-on-1 tutoring by “mere mortal tutors” (i.e., good people where you reject 4 of 5 applicants, but not TFA alums who are Yale grads; working 40 hours a week, not 75) can lead to unusually large gains.
At least in math….
* * *
Part 2: Matt’s Study
I’ve blogged before about this issue: math gains in good charters typically much bigger than English gains.
Commenter Tom pointed out
One problem with our testing regime is that it makes math look like half of the goal of a school. Math as a discipline is constructed and taught differently than all others. If your school is constructed to optimize math instruction, you’re not half way there; you’re more like 1/5th of the way.
I agree. English is typically more important to a kid, whether for college success or the ability to compete in the labor market.
Enter tutoring. Can tutoring help bolster English scores?
Our experience at MATCH high schools was that a high-dosage weekend tutoring program, which we deployed in 2003 and 2004, seemed to help a lot in math, but not in English. That is, our math scores went way up in those years. But not our English scores. Causal? Unclear. An alternative explanation would be our math teachers outperformed our English teachers. But that didn’t seem right, based on observation.
In Fall 2004, we changed our tutor delivery model. Most tutoring was no longer provided by capable work-study students who were undergraduates, helping us on weekend. Instead, we hired full-time tutors called MATCH Corps. Our kids math gains were already quite high. But could these tutors drive English performance, too?
Yes. Or at least it seemed that way. Our English scores went up after MATCH Corps got going.
And now Harvard doctoral student Matt Kraft has essentially done the quantitative equivalent of “reconstructing the scene of the crime.” (Well it wasn’t a crime, it was the opposite of that, but you know what I mean).
Matt went back to 2004 and 2005. Reassembled all the data. Did a lot of fancy calculations. And the result is this paper.
It’s called:
How to Make Additional Time Matter: Estimating the Causal Effect of Extending the School Day with Individualized Tutorials.
Here is the graph.
Matt writes:
Two large shocks to trends in student achievement at MATCH are evident in this graph. First, there was a dramatic improvement in ELA achievement during the first year of ELT tutorials in 2005. Secondly, there was an ever larger gain in mathematics achievement in 2003, two years prior to the implementation of ELT tutorials.
This earlier increase of 0.60 standard deviations in mathematics achievement raised MATCH into the 81st percentile of schools statewide in term of average mathematics achievement. Comparatively, MATCH students were only at the 50th percentile of average ELA achievement in the same year.
The comparative effect sizes of attending a semester at MATCH in 2004, the first year that lottery data are available, confirms the differential impact of attending MATCH on ELA and mathematics scores. Using an instrumental-variables approach where the offer of enrollment at MATCH serves as an instrument for attendance, I estimate that attending a semester of MATCH in 2004 had no effect on students’ ELA achievement (-0.053, p=0.676) but provided almost a quarter of a standard deviation difference (p<.001) on achievement in mathematics per semester. Thus, given the extremely large effect size of attending MATCH on students’ achievement in mathematics before 2005, it may be that there was a far smaller margin for subsequent improvements in mathematics achievement through the addition of ELT tutorials as compared to ELA.
A review of the changes in academic programing at MATCH reveals that the large gains in mathematics achievement evident in 2003 coincided with the first major expansion of instructional time at the school. In 2003, two years prior to the implementation of ELT tutorials, MATCH expanded and formalized its fledgling weekend tutorial program so that students were required to attend a total of 25 weekend tutoring sessions. The school hired college students through work-study programs to tutor sophomores for four hours in mathematics and English on either Friday afternoons or Saturday or Sunday mornings.
However, the effect of these weekend 25 tutoring sessions on MCAS scores cannot be isolated from other important changes that occurred concurrently at the school. MATCH was still in the very early phases of its development, and like all new schools, was undergoing rapid organizational change from year to year. In addition to adding the weekend tutorials, MATCH moved from its original cramped location at a converted synagogue to its current building and underwent several important curricular changes.
Because of these confounding factors and the absence of lottery data, I cannot attribute these gains definitively to additional time for weekend tutoring sessions, and hence decided not to make them the subject of a more detailed analysis.
A second possible explanation for the differential impact by subject of expanded learning time at MATCH is one of quality. It is possible that the rotating group of undergraduate volunteers who tutored MATCH students on the weekends were able to provide valuable assistance in mathematics, but not in English language arts. These minimally trained college volunteers might have been more successful at teaching rules and concepts from Algebra and Geometry during unstructured and isolated weekend sessions than improving reading comprehension or writing skills.
It is also possible that gains in mathematics achievement are simply easier to produce than those in ELA. Recent evaluations of TFA Corps members (Decker, Mayer, & Glazerman, 2004) and no-excuses charter schools find consistently that students of TFA teachers and no-excuses charter schools are making gains in mathematics that are two to three times as large as gains in ELA (Abdulkadiroglu et al., forthcoming; Dobbie & Fryer, 2011; Angrist et al., 2010). Thus, producing measurable gains in ELA might have only been possible with the high-quality individualized tutoring that full-time Tutor Corps members were able to provide through structured, curriculum-based and sequenced tutorial sessions.
Great work by a great young scholar.
You need to convert that .tiff to a different format to work in all (any? my?) browsers.
It is certainly easier to believe intuitively that high-dosage tutoring could have big impacts compared to most of the other snake oil that’s being peddled.
My point is the same as Tom’s. So, why not invest in high-quality tutoring? Why not invest in win win solutions? Why not light candles in challenging schools rather than curse “the status quo” like Grier is doing?
Why experiment with Grier’s other dangerous policies? Turnarounds, like Charles Payne explains, are “burnout” jobs. The only way they have a sustainable future is if they can be team efforts. Why not give neighborhood schools a fighting chance before using turnarounds to escalate our educational civil war?
I’m glad you respect the arguments on why its easier to raise math scores. When “reformers” show they can teach reading for comprehension, so that kids learn to read and then read to learn, then lets talk about scaling up within reason. Until you can show results for the overwhelming majority of kids who won’t (and in many cases can’t) put in all of those extra hours, what have you shown?
Don’t get me wrong. I admire what you do. Your approach wouldn’t be for me. But combine enough niche programs and who knows what good will happen. Calling something a niche market is not a criticism. The idea that “No Excuses” schools can be more than a niche, however, is pretty strange.
I sometimes wonder whether this gap doesn’t actually speak to the possibility of isolating discrete skills in math and testing them, rather than educating kids to understand math as a system or language that they can employ in a variety of contexts. I think the range of questions you can ask a kid on the math MCAS is small enough that kids who really don’t get math can do well. In my experience, those same kids often do not do very well in calculus or on the SAT, which, I think, rewards kids how having the kind of familiarity with how numbers work to deal with questions they may never have seen before using he skills they have.
This is harder to test in English, where it’s easier to produce a 10th grade level test that feels unfamiliar to a student. There’s really not a discrete set of “skill” that one employs in isolation from one another in reading and writing. But in my experience in no excuses charters, kids are trained to look for formulas, just as in public schools. It doesn’t make them effective readers, writers, or thinkers, and those formulas don’t get them as far in ELA as basic math techniques can get you even without an understanding of why you’re doing them.
Amanda, I we need to be cautious about not constantly raising the bar when we examine gains.
For example, I saw a blog the other day where the writer was saying kids who had aced AP Calc still didn’t “really” know any calc.
For a bigger picture, we want to see movement on the ladder. Kids who arrive to high school unable to multiply 8 times 7, and then a few years later are able to solve algebra and geometry problems, that is meaningful, very large progress.
Is that kid an “effective thinker”? He’s much more effective than he was. Basically you’re getting both the absolute gain he’s achieved, plus the wild card of whether he may even progress farther up the ladder (towards whatever your standard is for effective).
Literacy skills can be taught (e.g., the critical reading skills of paraphrasing, inference, vocab in context, summarizing/inferring main idea), but the comprehension process overall relies heavily on the use of prior knowledge to derive meaning. That’s why you can master algebra but never completely master reading comprehension: we’re always reaching toward new knowledge, using what we know with more or less success.
For anyone interested, a more thorough explanation of my theory of comprehension can be found on the “Comprehension 101″ page of The Literacy Cookbook:
http://www.literacycookbook.com/page.php?id=5
Cheers,
ST
I have a theory about the discrepancy between math and reading scores at No Excuses schools. I am assuming here that the achievement gap is at least partly caused by different learning experiences in the homes of kids at different income levels. I believe that pre-literacy skills are taught in upper income homes far more than pre-numeracy skills are, making the achievement gap in literacy wider. When I think about my daily interactions with my 20 month old daughter, I probably have hundreds of language based interactions per day; from reading books, to explaining what a word means, to pointing out cause/effect relationships, to saying a sentence back to her with correct syntax. I can only think of two numeracy based interactions that we have had today. We counted Legos. And we compared the sizes of her cars. So I am going to have a much greater hand in my daughter’s literacy development and her school will be primarily responsible for her numeracy development. Hence why good schooling can change math skills much faster.
Allison makes a good start, but what none of you are discussing and what is absolutely crucial is how different learning language is from learning mathematics. In short, learning language is instinctive (Steven Pinker provides a good introduction to the basic linguistics involved in “The Language Instinct”), whereas there is nothing instinctive about learning algebra; therefore the latter is entirely acquired through direct instruction, the former mainly indirectly acquired through being raised in a more or less optimal environment.