Sunday, November 23, 2008

How not to conduct a survey

John Ziegler's recent survey of Obama voters will now be a case study of how not to word a questionnaire, and also how not to defend one's methods in the face of criticism.

Here's a description of Ziegler's survey:
The conservative website reports that it has commissioned Zogby International to conduct a poll of 512 Barack Obama voters as part of what can best be described as a viral marketing effort to discredit the intelligence of Obama supporters.

The website, created by former radio talk show host John Ziegler to promote a forthcoming documentary, features a YouTube clip of interviews with 12 Obama voters who "were chosen for their apparent intelligence/verbal abilities and willingness to express their opinions to a large audience". The clip portrays the Obama supporters as giving "incorrect" answers to political questions such as "which candidate said his policies would likely bankrupt the coal industry and make energy rates skyrocket". Of the 12 Obama supporters interviewed for the clip, 7 (58%) are black; nationwide, about 23% of Obama supporters were black according to the national exit poll.

In connection with the YouTube clip, Ziegler describes that he "also commissioned a Zogby telephone poll which asked the very same questions (as well as a few others) with similarly amazing results." Partial results of the survey from among 512 Obama voters are reported on the website. It is not clear if voters for non-Obama candidates were screened out by the survey, or Ziegler has chosen not to report their results.

Most of the questions on the survey take the form of a multiple choice political knowledge test, stating a "fact" to the respondent and asking them which of the four major candidates (Obama, McCain, Biden, Palin) the statement applies to. Questions include the following:

"Which of the four [candidates] said his policies would likely bankrupt the coal industry and make energy rates skyrocket?"

"Which of the four [candidates] started his political career at the home of two former members of the Weather Underground?"

"Which of the four [candidates] quit a previous campaign because of plagiarism?"

"Which of the four [candidates] won his first election by getting opponents kicked off the ballot?"

As should be obvious, the veracity of several of these claims is -- at best -- debatable, yet they are apparently represented as factual to the respondent. It is not clear whether the respondent is informed of the "correct" response after having had the question posed to him.

Not all of the items in the poll are intended to apply to Obama or Biden. Several apply to Sarah Palin, although the items about Palin, while probably unflattering ("which of the four [candidates] has a pregnant teenage daughter?") are nevertheless apparently true. The exception is a "twist" question about Palin in which the respondent is asked "which candidate said that they can see Russia from their house?". Ziegler claims in the video that none of the four answers is correct because the statement was made by Tina Fay rather than Sarah Palin. (In her interview with Charlie Gibson, Palin said that "you can actually see Russia from land here in Alaska", not that she can see Russia from her house.)

To my mind, this survey meets the definition of a "push poll", which the Random House Dictionary defines as "a seemingly unbiased telephone survey that is actually conducted by supporters of a particular candidate and disseminates negative information about an opponent." That (i) several of the items on the survey contain information which, in addition to being negative, is arguably also untrue; (ii) Ziegler brags that the survey includes a trick question to which no correct answer can be provided, and that (iii) apparently only Obama voters were targeted by the survey (although this is not 100 percent clear), also inform my opinion that the survey can fairly be described as a "push poll".

In an item on his personal website dated today, 11/18, Ziegler claims that Zogby will officially release the results of the survey tomorrow. Ziegler also appeared on Fox's Hannity & Colmes news program yesterday (11/17) to promote his documentary, on which clips from the YouTube video were shown.

Why Zogby International has decided to accept this client and conduct a survey in this fashion is not clear. I would hope, however, that any and all clients that need legitimate polling work conducted would take their business elsewhere. These clients include C-SPAN and Reuters, two organizations with longstanding and well-deserved reputations for accuracy and neutrality; contact information for C-SPAN and Reuters can be found at their respective webpages.
One of the cardinal rules of survey construction is that one should never ask leading questions. We'll also say that surveys should use neutral wording, and that "trick questions" are generally ill advised. There's also the question of sampling, and the extent to which it was truly representative. If Ziegler was merely interested in confirming his own presuppositions, he succeeded - but in terms of being accurate, Ziegler's methodology failed miserably. What sort of long-term effect that the incident has on Zogby's (the polling company commissioned to conduct the survey) reputation is unclear, but suffice it to say this is not one of Zogby's better moments. Credibility in the survey business - to the extent that the biz has any basis on scientific methodology - goes only so far as a pollster's accuracy, which presupposes the pollster carefully screens questions before foisting them on the public.

Both Zogby and Ziegler have taken some heat for the survey (whether it really is a push poll is subject to debate, but the criticisms of poor question wording and sampling certainly do have merit). Zogby's organization has handled criticisms reasonably professionally if not always in a straight-forward manner. Ziegler for his part has responded to criticism with paranoid ranting and obscenities (Ziegler's background in talk radio may be a factor in explaining his rather abrasive style when challenged). There is no such thing as the perfect survey (or the perfect experiment, etc.), and any reported finding will be subject to questions and criticisms from reasonable skeptics. Responses to such skeptics should be kept civil - Ziegler's responses of "go fuck yourself" to at least one skeptic didn't exactly score him any points except, perhaps, with those who would be Ziegler's groupies and other like-minded individuals.

I suppose we'll just let Nate Silver have the last word for now:
The only reason that Mr. Ziegler's original survey got stupid answers from Obama's supporters is because he asked stupid questions.

No comments:

Post a Comment