Poll Crazy   

America's news organizations poll the public on a staggering variety of subjects, from Iraq to the sniper to whether Elvis is still alive. Does all of this surveying increase understanding, or does it simply amount to more random noise?

Freelancer Dana D. Kelley wrote a column in the October 18 Arkansas Democrat-Gazette about drug laws. In it, the writer opined: "They don't take polls on such things, but my guess is that 95 percent of Americans would support a 'shoot on sight' policy toward the sniper."

As far as a Lexis-Nexis search can tell, they haven't taken a poll on such a thing. But it's a wonder.

Americans--or those who don't have caller ID and haven't mistaken the pollster for yet another telemarketer--have been asked everything from the standard "Whom will you vote for?" to "Do you think Elvis is alive?" A mere day after the September 11 attacks, the poll machine was up and running, featuring questions ranging from whether people would change their travel plans to how long it would take the United States to find those responsible. NBC News asked if the terrorist attacks were worse than Pearl Harbor.

In June, a Fox News Channel poll asked, "While it is a highly unlikely situation, if you had the opportunity, would you personally kill Osama bin Laden?" TV Guide, in what it called a "fun" survey, inquired whether respondents would rather have Barbara Walters or Diane Sawyer interview bin Laden. (He's popular in poll questions.) Other polls have asked Americans what they think about things they can't possibly know, such as when the war in Afghanistan will end or if Iraq has nuclear weapons. A Time/CNN poll queried, "Just your best guess, do you think Osama bin Laden is alive or dead?"

And then there are the Internet polls--sorry, "surveys," "questions" and "live votes," because everyone (we think) knows that these are not real polls and the results mean nothing. There's more license in what you can ask when the results mean nothing. Hence, a nearly endless range of topics. From CNN.com (and read and touted on air): "Do you think authorities will ever catch the sniper?" and the incredibly balanced question, "Should the feds learn to leave California's medical marijuana growers alone?" MSNBC TV's Question of the Day (which can be e-mailed to your inbox every day) was "Do you think this latest terror arrest will lead us to Osama?" after news broke of the capture of the al Qaeda operative who allegedly planned the USS Cole attack. The day before was more pop-culture--"Should Michael Jackson's kids be taken away from him?"

"We've become poll crazy as a society," says Ken Dautrich, director of the Center for Survey Research and Analysis at the University of Connecticut. "When you look at a few decades ago, there were a handful of polls that had a code of ethics and kind of a code of science and a methodology that was applied to what they do. Given that polls are so popular, there are hundreds of organizations...that have gotten into the polling business." And, says Dautrich, there are polls "on any topic you can imagine."

No one is counting the number of polls out there, so it's hard to say when a dramatic increase occurred. Kathleen A. Frankovic, CBS News' director of surveys, puts the brunt of the upsurge in the '90s. "I think we do see more polls, but I think the number is not hugely larger than it was, say, seven or eight years ago.... I think the number of surveys increased in the 1990s--the number of real polls--and then secondly, that's when you started to see all these Internet quick polls out there."

So, with all this picking of people's brains, we have a better idea of what Americans think, right?

Maybe not. Steve Farkas, senior vice president and director of research at Public Agenda, a public opinion research organization, offers the following as a possible lead for this story: "We have more surveys than ever, yet our understanding of the public's mind is impoverished, and why is this so?" Surveys that are meaningless, he says, answering his own question, "add to the random noise, so it's difficult to distinguish quality work from just plain silliness."

Which brings us to the AJR Question of the Month: Has polling gotten a bit out of control?

A silly question is clearly in the eye of the beholder. Take the would-you-kill-Osama poll. The majority of those interviewed for this story either groaned or laughed when they heard that one, though one person--a media critic--said the issue of whether he would kill bin Laden had crossed his mind.

After hemming and hawing for a few seconds, National Journal's William Powers says yes, he'd kill him. (Wait--have you been skimming ahead to find out how many Americans would kill Osama?... Yeah, yeah, guilty pleasure. See the box on page 44.)

With the large number of polling organizations, and opinions, out there, questions that are silly, ridiculous or just plain bad are bound to pop up. And to be fair, most of these inquiries are part of lengthy polls that include about 20, often disparate, questions. "There are some things that people will ask and other people won't, and I think everybody draws that line quite differently," Frankovic says, adding that she has seen questions that she wouldn't have asked, and she's sure she has asked things that other people wouldn't. Frankovic readily takes credit for that Elvis inquiry. (Don't worry, only about 7 percent of those polled thought The King was still breathing.)

Other pollsters are more perturbed by the current poll-o-rama, especially when they consider online surveys. "I think one of the things that people in my profession...are quite wary of is that polling has become entertainment, and to us...polling is a very serious business. It's a science requiring very great skills," says Mark A. Schulman, president of the American Association for Public Opinion Research. "And we every day see examples of how our profession, our science is being misused...or being used frivolously."

Regarding that Pearl Harbor question, Farkas says: "People say [these are] public opinion polls. Public opinion polls for what? What is the purpose?" To inform policymakers of the concerns of the American public? he asks. "Some of these questions become a little bit of a stretch.... Some of this stuff is a little cheesy."

Iain Murray, director of research at the Statistical Assessment Service, an organization that helps journalists understand scientific and social surveys, says hypothetical questions, in particular, are pretty meaningless. "If the purpose of the poll is to find out information, then obviously a meaningless question doesn't help you in any way."

Frankovic agrees, though she says that doesn't mean she would never ask a hypothetical. She adds questions that could harm someone or frighten people to the list of polling don'ts. Also, "you don't ask things where the respondent can't make a judgment," she says. "I don't like asking the American people to be experts...but other pollsters have no problem with that."

Farkas says he remembers seeing questions that asked the public how long the war in Afghanistan would last. "Asking people questions that they have no clue about and yet they're willing to give you an answer...I think that's very unfortunate." Some of these, he says, "make me chuckle, and I feel guilty about chuckling." Pollsters shouldn't be so high and mighty about what can be asked, Farkas adds, but at the same time, frivolity contributes to an anything-goes attitude. "There were some questions about Clinton and Lewinsky...that I'm happy I don't remember anymore." (Perhaps he means questions that asked the public to predict the outcome of the Clintons' marital crisis and speculate as to why Hillary was putting up with Bill's shenanigans. Or, "How would you rate President Clinton as a husband?")

"Having fun with polls is OK," says Schulman. But, he stresses, using polls as entertainment instead of as a bridge between government or corporate decision makers and the public--the original purpose of polls--"really undermines the credibility of a lot of what we do."

Because so many surveys are commissioned by news organizations, this is not just a question of polling etiquette. It's one of media ethics as well. "There's always the serious question of what is your journalistic purpose," says Al Tompkins, who teaches broadcasting and online journalism at the Poynter Institute. "You don't ever want to do anything that would end up compromising your journalistic integrity."

Of course, where you draw the integrity line is quite subjective.

Some think the media need to stop using polls as much as they do, silly questions or no silly questions. Mike Littwin, a Rocky Mountain News columnist, says journalists should rely less on surveys. "It's religion," he says of the media's infatuation with public opinion polls. "We have come to believe in them even when we don't."

When poll results are ambiguous, journalists often find some meaning in them anyway. Tompkins says when a news organization has forked over the money for a survey, it wants an answer. "And it's not much news if a poll turns up inconclusive," he says. "I think news organizations feel some pressure" to get a poll to say something that it might not.

For instance, a CNN/Time magazine poll released September 13, 2001, asked Americans whether Congress should declare war. Sixty-two percent said yes. When they were then asked against whom war should be declared, 61 percent said they didn't know. To some, this would be comical, or at least suggest that it was inappropriate to be asking such questions already. But journalists found some significance in the results. CNN's Garrick Utley surmised, "That uncertainty gives President Bush flexibility and time to determine what kind of a war he intends to wage." (Two percent, by the way, chose "no one/just declare war.")

Littwin wrote a column questioning the validity of another nationwide poll that asked, "Thinking about the recent serial killings by a sniper in the Washington, D.C., area, how worried are you that someone in your family might become a victim of this kind of violence?" Forty-seven percent of the respondents said they were either "very" or "somewhat" worried.

Littwin had a hard time swallowing this one. "I don't think that half the people in America are afraid that a sniper is going to shoot them," he says. "Some of these [polls] defy reason and that was one of them."

The question, asked in a poll commissioned by Newsweek, also took a beating in a scathing column by Powers in National Journal, in which he chastised the media for instilling fear in the public. "They may have asked the question at a time when media outlets like Newsweek had flogged the story" so much that the poll results were true, he says in an interview. "It's like they were fulfilling their own self-created prophecy."

Powers says he also wondered if the question was designed to get the answer it got. "If you say, no, you're not concerned, it seems like you're irresponsible.... Sometimes people are not as stupid as their aggregate poll answers suggest."

The results are irrational. Newsweek's Marcus Mabry, chief of correspondents and senior editor, says he was surprised, initially, by the response. The magazine included the topic because it had been asking a similar question about fear of a terrorist attack, he says. Once Mabry started thinking about the sniper results, he says, he realized that unlike the terrorist attacks, which to many people still seemed remote, the sniper shootings were occurring at gas stations and a school, places every American knows. "All of those things were part of everyday Americans' intimate experience," he says, "places where they feel safe."

"How logical is it for someone in California to be fearful of [a sniper attack]?" says Mabry, who orders up polls for the magazine. "But it's not about logic.... It's greater than that."

A list of percentages may not provide enough information about what Americans are thinking and why. Such is the limitation of a poll. Farkas suggests journalists use common sense in reporting survey results. "If it doesn't ring true and it doesn't make sense," he says, "then you are probably on to something and you should probably not take any poll finding at face value."

MSNBC.com takes pains to make viewers realize that its live votes and questions of the day are not scientific polls--a fact that should be obvious. The public isn't really in a position to answer meaningfully whether they think Saddam Hussein has "come clean" with his weapons list or which strategy could be the most fruitful in the search for extraterrestrial life.

The results are not meant to be taken seriously. Says MSNBC.com Managing Producer Reed Price, "We offer them because it's an opportunity for interactivity. It's fun.... It's an opportunity for the viewer or the reader to interact with the Web site...but it isn't meant to be anything more than that."

MSNBC.com and CNN.com both use the term "vote" or "question" instead of poll and include disclaimers saying that the results are not scientifically accurate and only represent the views of the people who choose to respond. MSNBC.com also provides with every vote a link to a page that explains in detail the difference between online tallies and real polls.

CBS News' Frankovic says news organizations do a good job of making a distinction between these pseudo polls and scientific polls, and she likens the online phenomenon to call-in surveys that were very popular pre-Web. "Internet companies or Web site designers believe that putting a poll on your site keeps people there longer" and that people love to answer these questions, she says. (This, in fact, is the reason a few of those interviewed for this article give for the high number of polls out there in general.)

Since the results are not representative of anything, what information is someone supposed to take away from online surveys? The obvious answer: absolutely none. But it's hard to look at a bar graph with percentages (an informational graphic) and not think something.

"Our fear," says AAPOR's Schulman, "is that in the public's mind a poll is a poll is a poll." Many of the Internet "polls" are done by the news media, which, Schulman says, lends them "the aura of this news organization's credibility." As for the disclaimers, he says, "How many people read the small print?... It sort of looks like a poll and sounds like a poll to the uninitiated."

Pollsters like Schulman wish these proliferating online surveys would just go away.

"I think especially with the invention of Internet polls, [the state of polling seems] to be worse now," says Iain Murray of STATS. "People seem to want to find out their opinion instantly...with the Internet." But "they're actually not able to do that."

Schulman calls these votes and surveys "gimmicks" used to drive traffic to a Web site.

Newsweek's Mabry says he does get letters from online readers asking why the results have been different between online polls and regular Newsweek polls. Frankovic says she gets fewer such inquiries now than she did five or six years ago. She heard from a lot of people during the Clinton-Lewinsky scandal, she says, when the findings of Web polls were much different than real polls and the feelings about the results were intense. "So I don't know if we've sort of fallen into a less intense period," she says, "or if people are more likely to recognize the difference now."

It's unclear what the public thinks of these Internet surveys (I couldn't find a poll on this), but in a January/February 2001 AJR article (see "Polled Enough For Ya?"), Gallup Poll's Frank Newport said the average American doesn't understand the mechanics behind scientific polling. People do not think that a poll of 1,500 to 2,000 people can represent how all Americans feel, he said. "They find that hard to believe."

So the responsibility, as usual, rests with journalists. They have to ferret out the statistically sound from the sloppily compiled in their reporting. And news organizations need to ponder what they're polling about and why.

The real question that needs to be asked, says Poynter's Tompkins, is, "What's the news value here? What's the real value of this?"

News outlets seem to be less inclined to report the results of someone else's poll than they are to broadcast their own surveys. (Most of the polls mentioned in this story got little play outside stories by the sponsoring media organizations.) Which suggests that sometimes polls are more about promotion than information.

Ask yourself, "If you're Channel 9 and Channel 7 has a poll, would you report it?" Tompkins says. "Sometimes we overplay them--overhype them--because they're ours."