College Persistence: Josh checks in
Posted: December 27th, 2011 | Author: Michael Goldstein | | 6 Comments »
MIT economist Josh Angrist wrote me a note. He suggested I read four studies of interventions on college persistence.
I’ve done two randomized evaluations of college retention/grade treatments on community college campuses.
One published (STAR).
One in review (OK).
Both involve a mix of $ aid and services. These look at Toronto campuses w many LEP/immigrants
See also the MDRC randomized evals of various interventions including “Learning Communities”, a widely used post-secondary model and the ongoing “Opening Doors” trial for community college campuses with lots of African-American single moms
And here.
The bottom line so far: some short-term gains (for a term or two). Not much in the way of lasting gains. Pretty close to nothing works for men.
Okay, Josh. I’m digging in on Paper 1, the one you published in the American Economic Journal. Just a sec while I take out my slide rule.
[If you're a No Excuses teacher but don't like wonkish experiments, skip all the way to the bottom of the blog. Read what the college students said.]
This paper reports on the Student Achievement and Retention Project (Project STAR), a randomized evaluation of academic services and incentives at one of the satellite campuses of a large Canadian university. In American terms, this institution can be thought of as a large state school, with tuition heavily subsidized. Most students are from the local area and have a common secondary school background.
For the purposes of the study, all first-year students entering in September 2005, except those with a high school grade point average (GPA) in the upper quartile, were randomly assigned to one of three treatment groups or a control group.
One treatment group was offered an array of support services including access to mentoring by upper-class students and supplemental instruction.
A second group was offered substantial cash awards, up to the equivalent of a full year’s tuition, for meeting a target GPA.
A third treatment group was offered a combination of services and incentives, an intervention that has not been looked at previously.
The control group was eligible for standard university support services but received nothing extra.
Okay. Pause. What do you think happened?
Ah, first you want to know more about the details about the treatments. What type of support? How much money?
I thought so. Here are some details:
Group 1: “An array of support services”
250 kids in the first group got “Facilitated Study Groups (FSGs).” Peer advisors were trained upper-class students in the treated students’ program of study. Advisors were meant to offer academic advice and suggestions for coping with the first year of school. Advisors e-mailed participants regularly and were available to meet at the STAR office. FSGs are class-specific sessions designed to improve students’ study habits without focusing on specific course content. The FSG model is widely used in North American colleges and universities.
I wanted a bit more info, and I found this later in the study.
Peer advisors in the STAR program had exceptional social and academic skills. They participated in a three-day training course and received continuous training and feedback from supervisors. Advisors e-mailed advisees at least biweekly as a reminder of the advisors’ availability and to solicit questions about university assimilation, scheduling, studying, and time management.
The advisors complemented existing student services by reminding advisees of the availability of STAR and non-STAR services, and by encouraging advisees to use these services and to attend tutorials and make use of faculty office hours. Advisors were also trained to identify circumstances that called for more professional help and to make appropriate referrals.
That wasn’t all. They also got Facilitated Study Groups.
FSGs were voluntary, course-focused, weekly sessions open to all treated students in (Group 1 or Group 3). FSG student facilitators were previously successful in the course they were hired to facilitate. They attended the same course with their assigned STAR students and tried to help students develop reasoning skills useful for the subject they were facilitating. FSGs were designed to complement the regular content-based tutorials taught by graduate students. Rather than walking through sample problems, FSGs focus on critical thinking, note-taking, graphic organization, questioning techniques, vocabulary acquisition, and test prediction and preparation.
Group 2: Money if you get good grades in college
My parents gave us $5 for good report cards. Here the stakes were higher.
250 students (got) the opportunity to win merit scholarships for solid but not necessarily top grades in the first year. Award targets were set based on high school grade quartiles.
Participants from the lowest grade quartile were offered $5,000, roughly a year’s tuition, for a B average (a GPA of 3.0) or $1,000 for a C+
(a GPA of 2.3).Participants from the second quartile were offered $5,000 for a B+ average or $1,000 for a B-
Award thresholds were raised to A- and B for those in the third quartile.
Upper-quartile students were not eligible to participate in the study.
Group 3: 150 kids who got everything in Group 1 and Group 2.
Control Group: 1,006 college students got nothing.
Results?
The estimated effects of SSP and SFP are consistently zero across most cuts of the data.
Women benefitted from Group 3 (the combo). An average GPA of 1.7 went up to 2.0.
Men did not benefit from anything. C’mon guys! Shape up.
This is what it looks like graphically, men v women. Remember, the goal is to shift the “control” (light color) curve “to the right.”
First, the male response to these interventions. SSP = Group 1 (support). SFP = Group 2 ($$$). SFSP = Group 3 (both).
Now the female response to these interventions.
Perhaps the clearest pattern coming out of the subgroup analysis is that the SFSP effects are larger for women whose parents did not attend college.
I know that No Excuses charter schools are increasingly trying to support their alumni. It was interesting to read some of the interviews:
Some interviewees from the fellowship groups suggested that the fellowship reminders through e-mails and advisor contacts mattered quite a bit. No student said there were too many fellowship reminders or too many e-mails from advisors.
Those we talked to about the SSP focused almost exclusively on the advisor program (and not on the study skills classes which were available). Many students were pleased with their advisor interactions, or were simply glad to know that they could communicate with their advisor if needed. One male SSP students noted, “University is something different and I also wanted extra help. The peer advisor, I personally did not meet him, but he was really helpful because at every 15 days he used to e-mail me and ask me how it’s going and how I did on the test.”
Another female student said, “I thought that it was nice to know that there was someone there that you could run to and ask for help. At the beginning I just started to talk to my advisor and she did give me some advice but I found that at the time it’s hard to just all of a sudden change all of your schedules just because she said you should have good time management and you should do this and that. So I guess that you slowly start to see what your peer advisor is talking about then you try to change for the best.”
Another student never met or responded to her advisor, but nevertheless felt the advisor’s regular e-mails were helpful, “Like somebody who cared.”
I think this sums up MATCH’s experience to date. Generally, our alums in college are okay with us trying to be in touch; many seem to like it. But we find what Josh finds: gentle, helpful nudges have not seemed to change academic behavior.
Charter schools and others (including a growing number of nonprofits dedicated to college success) making new investments in “alumni coordinators” to provide precisely this sort of “support” should think hard.
Same with charter networks which are fundraising to provide alums with various forms of Pell-like cash donations (see yesterday’s blog).
Are there better interventions to drive college persistence? If so, what?
Here are a couple of studies from edweek on a coaching intervention and psychological confidence exercise. http://blogs.edweek.org/edweek/curriculum/2011/03/one-hour_confidence_exercise_c.html
I think it has to be as nearly compulsory as possible and that actual personal contact would be beneficial. Not in a study skills group sort of way — in that I think most college kids either feel like idiots in those classes or they feel like they’re wasting time they could be studying (even if they wouldn’t be!)
I’d envision something more like what some schools have set up for college football players (and other sports that eat up a huge amount of time). A study hall that provides some sort of incentive — food, tickets to events, prizes, money, something motivational enough to get you away from the movie, video game, nap — with the peer counselors there to sit and look at your schedule, your upcoming week, review your tests, make you send off an email to a professor to meet about a paper, etc.
At the same location, there should be the tutors available — so that a student can see a math tutor and get help on a paper and review errors on a biology test all in the same evening. If necessary, they can set up an appointment with a tutoring center then.
In large part, something like this would have helped kids I know who aren’t expected to struggle in college, but who are also not used to asking for help, not used to having to study to learn something, etc.
It’s very easy to get behind and feel like you’re the only one in college. Having a year’s worth (at least) of “intro to figuring out how to do this yourself” with real live people who can talk about that student’s specific situation seems to me to be the only way to actually see real change.
You’d think my last long comment would have been clear enough, but no, must clarify further!
Rereading even more carefully: all of the study help was *general* — even in the class specific study groups, they didn’t work problems, as it notes, but worked on “bigger picture skills.”
To me, this is the biggest problem. It’s why kids don’t come. Most struggling kids need MORE working of problems — even if the facilitator doesn’t go over the assigned problems, they should come with similar problems prepared and work them with students until those students can look at their assignment and realize they can do it on their own. That’s helpful, that raises test scores.
Imagine spending an hour of your time talking big picture, only to sit down later with your assignment and realize you don’t even know where to begin, let alone how to do the harder problems. The hour you spent discussing now feels like an hour less you have to try and teach yourself what to do!
Jen, agree with you on both counts:
a. Mandatory study halls seem to work for many D1 athletes (though I’m not sure if it’s been studied empirically), why wouldn’t colleges offer more of them?
Is there any way to set up a private sector version of mandatory study hall?
b. Yes, I agree with content. What we see if kids who need more simply wrestling with the dry text, whether it’s a psych or poli sci or crim justice class….
However, it’s possible to imagine “specific” versions of study skills….ie, not how to plan your time generically, but how to plan YOUR time as a college student with Classes A, B, C, D which meet this semester at the following times, and account for your part-time job and the time you like to go to the gym et al.
I agree on the specific general skills — before a semester starts, that would be a good time to have a “study hall/mentor meeting”:
do you have all your books, or know where to go to find copies available without purchase?
do you have a plan for eating/sleeping around your schedule so that you don’t end up tired and unable to pay attention?
And yes, a preliminary schedule/calendar listing classes AND acknowledging and planning for the study time needed for each class. That is, on the schedule, mark off the hours each day needed for studying and for which class.
Both of those last two should be updated several times during a semester to reflect the actual demands the student encounters.
I don’t know how to really attack/police the sleeping and eating one. Many kids get into a stay up late every night and completely screw up their sleep schedule in college and it’s hard to stop on one’s own. Some kids going off to college haven’t had a good or adequate sleep supply in years and don’t really realize how much it’s hindering them — not just the lack of sleep, but the effects on mood, personality and motivation.
I thought the student responses to financial incentives were also really interesting. It seems like for students who are successful, the incentives can be a really strong motivator to continue to push themselves. But for students who are not immediately successful, the economic incentive gets lost because their focus is switched to saving their failing grades rather than earning higher ones. So effectively, the incentive only works if students are in a stable enough position academically to be thinking about moving towards a higher GPA at all. It makes sense.
I wonder if there is a way to redesign this program of incentives to reach students at all levels. Perhaps an incentive for raising your GPA by one point, as opposed to an incentive for every A?