Houston: Liftoff
Posted: May 30th, 2011 | Author: Michael Goldstein | | 6 Comments »
Last September I blogged about an unusual assignment. The Houston district superintendent, Terry Grier, wanted to launch a turnaround effort in 9 of their worst schools, in collaboration with Harvard economist Roland Fryer’s EdLabs.
We were fortunate to have the chance to help out.
MATCH sent a little posse down there. Pictured are Patti, Tim, Cathryn (along with Jeremy, who was the superintendent overseeing all 9 schools). We also sent Christie, Erica, and Eli for shorter stints.
The idea was to help the turnaround effort, called Apollo 20, conceive and launch a program modeled on our MATCH Corps: to recruit (in 6 weeks!) and train 250 full-time math tutors, design a curriculum, etc.
Now, 9 months later, test scores are out. So how’d it go? Here’s what the Houston Press used as a headline on Friday: HISD Lesson for Today: Tutoring Works, Now What Do We Do?
The HISD press release has the details:
Sixth-grade students who were enrolled at Apollo 20 schools for the entire school year posted an 85 percent passing rate on the TAKS math exam, 22 points higher than sixth graders at those schools in 2010. Their “commended” rate is now 30 percent, an 18-point jump from last year.
Apollo 20 ninth graders enrolled the entire school year also made great strides with the help of the math fellows. These freshmen produced a 72 percent passing rate in math, which is 16 points better than last year’s freshman class. Their “commended” rate is now 20 percent, an 8-point increase from 2010.
Roland says kids will do even better next year. I believe him. Moreover, I believe that improvement is critical.
Let’s be real. It’s great that more kids passed. But the “passing bar” is low. Even with all that work by kids and tutors, that accomplishment is not nearly enough to change life trajectory for most kids. Those same students could, with the help of Apollo tutors, reach “advanced” status, which is called “commended” in Texas.
MATCH definitely learned some lessons in our first year of having full-time tutors (2004-05). HISD and Apollo will too, I’m quite sure. Tim (tall guy pictured above) remains in Houston. Best wishes to him, Brandi, Jeremy, Roland, and Terry on helping kids make the Climb To Commended.
I appreciate the perspective in the “Let’s be real” paragraph. Like KIPP’s recent report on college graduation, your write-up on Apollo 20 is refreshing in that you posted truly impressive results, but didn’t lose sight of your longer-term goal, and acknowledged that it hasn’t been reached yet.
Congrats! That’s a major feather in your cap, and no doubt it will only get better.
Do you know what other changes were implemented in the Apollo 20 schools (as I’m assuming tutoring wasn’t the only change)? Any thoughts from your staff if those other changes were helpful, neutral or detrimental to the tutoring efforts?
Thanks James.
Paul: The original Apollo turnaround plan can be read here.
http://houstoninformer.net/hisd-launches-apollo-math-program-p161-1.htm
and
http://www.chron.com/disp/story.mpl/metropolitan/7156131.html
Leader change, teacher change, 1 hour longer day, 5 days more per year.
On test scores, my sense was that it ended up somewhat neutral.
The tutored subjects (math 6/9) went up; the others (English 6 to 11; math 7, 8, 10, 11) were similar to previous year.
When I visited 2 schools, kids said that school culture was much improved. Certainly that helped the tutoring efforts.
I was generally impressed when I visited: tutorials were focused, calm, substantive.
How is this sustainable economically? How are you working with the classroom teachers so this is a team effort as opposed to 250 tutors sweeping in? Were you paid by SIG money? What is the parent involvement piece? What efforts to teach kids math is being made at the schools that feed into these schools? How do you make math relevant to these kids? How do you make it fun? What happens when the tutors go away?
Unfortunately, you use proficiency scores/pass rates to calculate gains in achievement and conclude the effort is a success. As Koretz (2008) plainly points out in his book, you simply cannot use passing rates to calculate gains and get an accurate result. The gains in passing rates are driven by the distribution of scores around the cut point and the percentage already passing. Let’s say that only 10% of the scores are 2 questions below passing in cohort 1 and 40% are 2 questions below passing in cohort 2. Whichcohort is posied to make large gains? The 2nd cohort of course. In fact, it is entirely possible that the first cohort of kids made GREATER gains than the second cohort of kids even though the first cohort made a smaller increase in the percentage passing. This is a FUNDAMENTAL mistake by many and leads to inaccurate conclusions about such efforts. Such simple analyses abound nowdays and are often to promote efforts that don’t work very well, but the PR machine ratchets up to say they do so reform leaders can rake in more money. Now, I’m not saying good tutoring cannot lead to large gains–in fact, as a former tutor, I believe they can. But you also have to understand that tutoring that uses practice items is a serious “teaching to the test” effort and makes the so-called gains useless because of test score pollution (see Koretz again).
My point here is to learn to use data correctly and accurately. If you do not know how to do this, then simply do try to dabble in using data.
Renee,
Lots of questions!
1. When tutors go away, I assume it’s like when “teachers” go away. Kids probably learn less.
2. Parents get called each week by the tutors. They seem to like that.
3. My sense was classroom teachers loved the tutors, particularly math teachers.
Ed,
Glad we have you around to patrol for dabblers! Onward.
You might look at Roland Fryer’s NBER paper on the same topic. He has about 27 different statistical controls in place.
On one hand, he is the youngest economist to get tenure at Harvard. So that’s to his credit. Of course he may be a data dabbler too.