Dear Matt DiCarlo
Posted: March 13th, 2013 | Author: Michael Goldstein | | 5 Comments »
A friendly open letter to Matt DiCarlo at the Albert Shanker Institute.
Dear Matt,
I’m a fan. Keep writing!
I noticed you have a Phd from Cornell. It’s freaking cold there. I assume that led you to spend a ton of time in the library, learning stats n stuff. You learned well and do great work with numbers.
I thought your KIPP post was quite good. Two quibbles I’d like to examine here.
First, the big picture of agreement. You write:
We can and should discuss the possibility of unmeasured factors such as peer effects, but it seems unlikely that these factors would come close to explaining away the estimated impacts. In general, KIPP schools are well-run and they do a good job with their students. At this point, arguing otherwise is unsupportable.
At the same time, it’s worth noting that Mathematica’s analysis found that roughly one in three KIPP middle schools did not produce significantly better results in reading, and one in four failed to do so in math (though, in both subjects, virtually none showed significant negative results). Even for KIPP, some “failure” is part of the game. Running schools is difficult.
Agreed. Especially that last sentence. I watch my colleagues work tirelessly to run our schools.
I’m pretty sure the folks at KIPP would agree, too. Now to my 2 quibbles.
1. Money
You write:
KIPP is very expensive, but they may get a worthwhile return on that additional investment. The existing evidence, though still a bit scarce (due mostly to data availability), suggests that the average KIPP school spends substantially more money than comparable regular public schools.
I don’t think that’s true. I won’t dig in extensively here. The 2012 report you cite contends “average spending per pupil was around $12,000 to $14,000 citywide” in the traditional NYC schools. Then it says NYC charters spend more.
The $12k to $14k figure is pretty far from what others say the traditional NYC schools spend.
a. US Census Bureau says “New York City spent $19,000 per student.”
b. The Citizens Budget Commission says New York City spends $20,276.
c. And here’s back and forth between the IBO and a charter advocacy group.
There are fair questions here. We could dig in on them, of exactly what should count as spending per pupil both for districts, and what should count for KIPP. Each type of school has some sort of central office expenses.
Moreover, KIPP does attract an unusual type of teacher. There is almost surely a limited supply of those folks. And that could go to some reasonable call for “more money,” to expand that labor supply in various ways. When Steve Brill considered this point in his excellent book Class Warfare, he threw up his hands. Essentially, the high-performing charter used equivalent cash to buy more labor, because it was attracting some workaholic teachers who were discounting themselves. So the kids did, in fact, get “more inputs.” Bottom line, I agree with you on the idea of examining “true cost.”
2. You write:
KIPP’s approach only works for some students, but these are among the students who need the most help to catch up. Through 8-9 hour days and summer school, KIPP adds the regular public school equivalent of about 3-4 months to the school calendar. In addition, exceedingly rigid disciplinary policies and parental contracts are used to enforce high-bar academic and behavioral standards.
I respectfully differ that “exceedingly rigid” describes discipline that I’ve seen in various KIPP schools. It’s totally a fair question.
I’m trying to imagine the falsifiable experiment or study that would allow us to empirically analyze discipline in the classroom. Do you have ideas?
And I say “examine the classroom” — not the code of conduct as written (because often there is distance between what is written and what actually happens), but the lived experience of the kids (which is what I assume you care about).
Here’s my idea.
a. Get an agreed-upon “neutral” scholar.
b. Find a group of schools — some traditional but with a dress code, some Catholic, some KIPP, some low-performing charters.
c. Shoot videotape of actual classes at random. Make sure there are no identifying characteristics (like a cross, a KIPP poster, etc).
d. Have folks interested in the KIPP and charter debate, pro and con, watch video with school identifications “blind” — no idea what they’re looking at — and rate what they see.
Is it a productive classroom? Is it joyful? Is it “exceedingly rigid?”
And so forth. Then try to predict which classrooms are in which type of schools.
Do you think this would be a useful experiment?
Here is a second idea. This goes to concern about whether kids are “forced out.”
a. Get an agreed-upon “neutral” scholar.
b. Find a group of schools — some traditional, some KIPP, some other charter schools, like the UFT-run school in New York City.
c. Create a list of all of their departed students.
d. Interview departed students at random. Or their parents. Possibly this could be done in the presence of a former teacher or principal, who might also comment. Provide incentives to be able to conduct these interviews. Roland Fryer, I believe, was able to interview students who lost the admission lottery to Harlem Childrens Zone schools this way.
e. We’d ask: Why did you leave School X? Did the teachers and principal try to entice you to stay? What do you think of your new school, School Y? That sort of thing.
I think this survey work would shed light on the concern that charter students are commonly “forced out.” I also think KIPP would fare very well in this type of study.
The interviewers could and should include some real charter skeptics. For example, Paul Hoss wrote a comment on your blog along these lines. I’ve read many of his other comments on edu-blogs. I think he is a smart educator. He remarks that he’s read the “forced-out” meme about KIPP. That suggests he hasn’t had a chance to decide for himself one way or other by talking to kids who’ve left KIPP and other schools. I’d nominate him as the perfect sort of guy to include in a study like this. I’d predict he’d end up converted, and conclude that the most typical KIPP story is teachers and a principal who desperately try to get a kid to STAY, but who chooses to leave in search of lax academics in a traditional school.
*
Why do I belabor how you and others characterize discipline?
We both want to learn why some charters, like KIPP, tend to out-perform other charters.
And school culture, I believe, is at the heart of why some charters do well and others don’t. It’s certainly not the whole story but it’s an essential component. We need precision here or we won’t learn any lessons.
I think KIPP and similar no excuses schools tend (imperfectly) to create a fairly positive, warm, and disciplined culture. I admire them. I think other charter schools often have exactly the same discipline policies as KIPP and don’t achieve that culture to the degree that KIPP, and by extension, don’t achieve the academic gains.
If we’re going to understand the no excuses charter school, and here is a scholar who has done just that, I’d encourage you to really dig in on this issue.
*
One more thing. A friend — a union leader — says he gets hit by “both sides” whenever he makes what he sees as a reasonable, centrist, evidence-driven statement….ed reformers attack him as half empty, some of his members call him weak and appeasing. It’s safer to stay in a single tribe. You’re to be commended for trying to call em the way you see em.
1) Study on excessive rigidity. Not sure your study would get at all areas where parents/kids struggle with high discipline systems. Sure, some classrooms can be excessively rigid as a whole. Other times, a single kid feels (rightly or wrongly) singled out. This might not come out in your matched classroom study. The other place I can imagine parent/kids having complaints are with the school-wide factors that support the productive classroom environments (that often don’t exist in other schools where teachers fend for themselves, management-wise). Examples might be systems (demerits, detentions, attendance and uniform policies, rules) or people (deans, principals).
2) Why kids leave: I think it is sometimes about chafing against the factors I mention above. Other times, its about the academic piece you mention (kids going to easier schools); I think this is less prevalent in K-8, though not unheard of (happens usually at year end when the student will be retained). Sometimes it’s about geography – kids moving away. Often, its about other great opportunities (exam schools, charters with a high school option, independent or parochial schools). At the two high achieving no excuses charter schools where I’ve worked, it has rarely been about expulsion, threatened expulsion, or long suspensions (I’d say 5 or 6 times in 9 years, all for weapons, alcohol/drugs, or repeated harassment/bullying). I have NEVER heard of a student being asked or encouraged to leave for academic shortcomings. And I agree, most of the time when kids are going to leave, they are highly encouraged to stay by teachers, administrators, etc. One of our 5th graders just got offered 3 full rides to independent schools; she has our blessing to grab one of those and not let go.
3) Finally, there are often accusations that charters “dump” low achieving kids right before state testing. I hope it’s not true. If it is, it should be a story / study all on its own. It gives charters a bad name…
I agree Paul. Hard to come up with a comprehensive experiment that is falsifiable. I do think the rules themselves are fairly easy to scrutinize. In fact, Fryer may have done that in his NYC charter study, and found no difference. Legit No Excuses school often had same Code of Conduct as lower performer.
My big point is: my experience over the years is that visitors whose critique of charters is based on reading accounts of them as “militaristic” are often surprised and sometimes even shocked when they actually visit. I went with a group of leading academics to New Orleans, a project called Futures of Ed Reform, and many thought “Wow, these schools seem NICE.” I was part of a group called Boston Schoolchildrens Consortium, which included supes of BPS and Archdiocese, and their visit to RoxPrep, again, it opened them to reconsider charters. In each case, these aren’t sworn enemies of charters, just skeptics.
I suspect a writer like Matt DiCarlo, who seems open to evidence, would also be surprised to walk through a classroom like yours at Brooke and say “This is No Excuses? Where’s the rigidity?”
That’s why I’m attracted to the video experiment. It’s also something that could be put into a website, so the 166,000 people trained each year to become teachers could actually see different schools, instead of simply reading highly-charged characterizations of them.
Still working on getting EduShyster into my school (she’s being coy about her anonymity), but I hope she really is open to new ideas and we can have a solid dialogue.
Hi Mike,
Thanks for the open letter (my first, by the way). I won’t try to fully address your points, but quick reactions:
First, as you know, the results of that CMO spending report aren’t just raw comparisons of per-pupil spending. They come from models that control for factors such as school size and student characteristics. The aggregate per-pupil figures were used to put the adjusted differences in context (e.g., to express them as a proportion of total spending). But you’re obviously correct that there are decisions to be made in terms of what gets included and what doesn’t.
The report includes *extensive* documentation of these decisions, and Bruce Baker (the lead author) has a lot of experience making them. I won’t get into a discussion about the methodology, because they are painfully boring and I’m not particularly qualified to do so, but I would encourage interested readers to check out the full report here:
http://www.shankerinstitute.org/images/CharterSpending.pdf
If anyone (particularly those who are well-versed in these methods) has a substantive critique of the methodology, I would strongly encourage them to lay it out. Bruce is generally very good about responding, as he did here:
http://schoolfinance101.wordpress.com/2012/05/07/no-excuses-really-another-look-at-our-nepc-charter-spending-figures/
I would add one thing: The findings of this report were really all over the place. In some locations, CMOs, even well-known organizations, spent more, and in some places they spent less. Other than the need for better data (which is crucial – this kind of analysis can’t even be performed in most states), the only consistent finding was KIPP’s spending (and one or two similar studies reached the same conclusion).
Second, with regard to behavioral policies, I did not mean the phrase “exceedingly rigid disciplinary policies” to imply that KIPP schools are boot camps filled with miserable kids. I can see how it might be interpreted that way, so that’s my bad – poor choice of words. In contrast, I have no trouble at all believing that they are positive environments.
And I don’t think most kids who leave are “forced out” against their will by the discipline policies (this is an empirical question that I would think might be addressed directly).
My guess (and that’s all it is) is that most separations are voluntary, and are most often due to students simply not thriving under the model (e.g., the long days – I doubt I would have done well with a nine hour day), rather than direct actions based on their breaking the rules of conduct or parental contracts (both of which I would characterize as the formal expression of norms that are “enforced” via informal means). Maybe I will write about this at some point.
Similarly, I have no doubt that school “culture” is important, and that KIPP schools have good cultures. At the same time, however, “culture” is an outcome, not a policy intervention. How does one foster culture? Isn’t it possible that “culture” is as much endogenous to success as a cause of it? Et cetera, et cetera.
Basically, I’m a heartless, arrogant technocrat. On the other hand, I’m *very* receptive to the idea that school “quality,” however you define or measure it, is not nearly as “susceptible” to direct policy intervention as is often implied. In fact, one of my primary issues with today’s debate is the tendency to assume otherwise, and to make promises that cannot possibly be kept. Policy is important and can make a huge difference, and there’s a role for urgency and expectations. However, among policy makers and researchers, the calibration of expectations is critical, and failure to do so is dangerous.
Finally, and most importantly, thanks for the kind words. Our ongoing communication is a great example of how I learn the most from people with whom I don’t necessarily agree on everything.
MD
P.S. One idea for your blog’s name: The Gold Steindard
Matt, thanks. I’m a technocrat too. You should see my weak social skills in action.
Yes, in the end, schools are not as susceptible to policy interventions as implied. When I went to get a degree in ed policy back in 1996, that’s what I surprised to learn.
My wish: it seems like health outcomes are less susceptible to policy interventions, too. Yet there’s a parallel track of drug discovery that is at least pretty decent, even while we argue over health policy. I don’t see the same level of edu-discovery in our field, even while we argue over ed policy.