Survey Hijinks

Posted: May 15th, 2012 | Author: Michael Goldstein | | 3 Comments »

Do you think wording matters in public opinion polls? That is, if a question is phrased slightly differently, can the pollster get very different results?

The correct answer is: yes.

If you want an NY Times summary of how wording matters in polls, it’s here.

One of my favorite articles on the topic was written in 1941, by George Gallup. Yep, that Gallup.

Anyway, now let’s move on to a specific poll.

There were two versions of the question. Version 1 was asked in certain years. Version 2 was asked in certain years.

Much has been made of this opinion survey.

(You can click on the image to make it clearer).

So what do you think?

Specifically,

1. Do you trust the upward line? Is this policy getting more popular?

2. Do you think the 2011 drop is caused by something? Or is it simply an outlier? Or are you not sure? It’s just one data point.

Now let’s re-examine the graph, but with the years listed on the X-axis.

Now what do you think? Sputtering around until 1995, then a surge. Then a down point in 2011.

So if you want to connect the public opinion here to federal policy, you’d connect it to Clinton/Bush, and then maybe blame Obama.

Okay, now I’ll show you the whole image.

Now what do you think?

Well here’s what happened when this survey came out.

1. Some reporters wrote: “Teacher satisfaction has dropped to a 20-year-low.” Example.

2. Some commentators argued: The decline is the fault of…President Obama….Michelle Rhee….teacher bashing.

Let me briefly address the commentary. If you want to opine on Race To The Top (federal policy) “causing” the 2011 low point, would you also ascribe the high scores to President Bush and No Child Left Behind? My guess: no. Then how do you determine causality?

To the reporters, however, I’d question the “20-year-low” meme.

Because….drum roll….this survey has used TWO different questions.

And we agreed from the jump that wording matters in public opinion polls.

Let’s create 2 new graphs from the data, shall we?

This graph is for the question:

All in all, how satisfied would you say you are with teaching as a career?

*

This next graph is for the question:

All in all, how satisfied would you say you are with your job as a teacher in the public schools?

The first poll question uses the word “career.” We can probably agree that evokes certain emotions.

The other poll question omits “career” and injects “public schools.” Also evocative.

So how might a reporter more precisely write up this poll? Well, I know it would kill the theme (teachers are unhappy!), but these would be more precise:

Teachers were asked a question about job satisfaction 5 times in the last 26 years. Three times (2011, 1987, 1985) roughly 40% said they were “very satisfied.” Once that number hit 52% (2001) and once it hit 33% (1986).

That would be a neutral version.

Or:

Teachers were asked a question about job satisfaction 5 times in the last 26 years. The second highest level of satisfaction was in 2011, with 44% saying “highly satisfied.”

I wouldn’t choose to write it that way, even though it’s factual. The 44% is probably within margin of error to the 40% numbers.

Or:

Teachers were asked a question about satisfaction 8 times between 1985 and 2009. The proportion of “very satisfied” teachers rose steadily.

Now what causes teacher opinion to change? I have no idea. It seems to me that the results could lend themselves to all sorts of plausible guesses.

But I do fault the survey, and the reporters, for not separating the results of the 2 questions. It’s likely that the wording skews the results in some way.

“A 20-year-low” would suggest to the many readers that you’d asked a question every year or so, which did not happen. A “20-year-low” would not convey what really happened: you asked 3 times in 20 years.

Put another way:

Let’s say this 2012 survey reverts to the “public schools” wording of the question. The one which has shown steady increase over the years.

I’d bet a beer that MORE than 44% are very satisfied — “despite” Obama, Gates, Rhee, the Red Sox, Mayan calendar, and other proposed “causes” of teacher dissatisfaction.

Anyone want that bet?