Evaluating The Toolbox   

The public journalism movement has had a major impact on the way some news organizations approach election coverage. Which techniques have proven successful?

When researchers try to evaluate public journalism, we are much like the blind men and the elephant. Each chooses a part that interests him or her, sizes it up, and then tries to generalize about the whole.

Eventually, we might accumulate enough descriptions of the trunk, the ear and other body parts to build a coherent picture. Until we do, public journalism is nothing but a kit of tools for trying to help media and democracy work better. All we can do to help media managers now is evaluate the tools, one by one. Declaring public journalism a success or a failure based on any one tool makes no sense.

Given that limitation, and because another national election is coming up, I've been looking for one of the foundations that push public journalism to assemble a summary of what works and doesn't work in that toolbox. None has. Both news organizations and their nonprofit patrons are wary of publicizing their failures.

That's unfortunate, because failure is an excellent teacher. Thomas J. Watson Sr. of IBM was one who recognized this. He was a strong advocate of exploiting the learning opportunities created by screwing up. "Double your rate of failure," he once advised a discouraged young writer.

In that spirit, here is an oh-hell-I-might-as-well-do-it-myself listing of lessons learned on public journalism's front lines in the past four years.

Where public journalism has gone wrong

*Ignoring the horse race is a mistake. When voters are deprived of information on which candidate is ahead, their interest in the election declines. So does their knowledge about issues.

This goes against the intuition of some editors, especially when they are looking at their budgets for campaign coverage. Polling is expensive. Public journalism can be used as a rationale for moving that money to reporter travel and more one-on-one coverage of candidates.

But in their public defense of curtailed poll coverage, managers don't talk about the budget consideration. Instead, they propose a model of a citizen whose brain capacity is too limited to consider poll and issue information at the same time. Dumping the polls, so this argument goes, will free up readers' brain cells to hold issue information.

Recent research tells us otherwise. Associate Professor Xinshu Zhao of the University of North Carolina at Chapel Hill and student Glen L. Bleske (now teaching at California State University, Chico) tested the effect of horse-race polls on what readers learn about issues in three projects: a laboratory experiment; a poll; and a panel survey, in which the same individuals were interviewed at different points to get a clearer sense of who changes and why. They got the same answer in each project.

In the lab, student subjects were divided randomly into two groups and given news stories about a fictitious campaign. One set of stories focused on issues and had no mention of polls. The second set included poll and issue coverage. Both groups were tested on their knowledge of the issues, and the group that received the information within the horse-race context did significantly better. Like spectators at a basketball game, their knowledge of the score helped them understand and appreciate the action.

The Zhao-Bleske surveys confirmed this advantage for poll followers. So did a more massive project, a 20-market survey funded by the Poynter Institute as part of a larger evaluation by Deborah Potter and me. Between August and November 1996, the poll followers learned more about the issues--even after we adjusted for their greater interest and education.

*Cram courses on issues are disappointing. Michael Schudson, in his excellent book "The Good Citizen: A History of American Civic Life," suggests an answer to all those academic studies that show how little our citizens know about public policy issues. An informed public would be good, he acknowledges, but a more realistic goal in today's information-overloaded environment is a "monitorial" public--one that scans rather than reads a large variety of news. A good newspaper--making smart use of the much-scorned pyramid style of writing--should be able to meet those needs. The reader can easily go deep or shallow as needed.

Efforts to explain everything in-depth to everybody are doomed to fail. The Record of Bergen County, New Jersey, found that out in its 1996 public journalism experiment. From Labor Day to the general election, it dedicated an open page in the first section to issue stories, teasing them from the front if they weren't running there. And then it surveyed its market in the hope of finding that Record readers knew more about the issues than others. They didn't.

This failure was no surprise to communication scholars, who have known since the late 1940s that media effects are not quick and dramatic. Instead of a magic bullet, they provide a slow drip, drip that can eventually have a measurable effect when the matter is really important. For example, it took the public nearly a generation to react meaningfully to the information that smoking causes lung cancer. To extend Schudson's analogy, equipping the citizens with a kit of information to take to the polls--as though they were backpackers headed out for an overnight hike--isn't workable if they don't really want to take that hike.

Issue coverage is needed. But if it is lengthy and divorced from the campaign action--the strategy, maneuvering and horse-race polls--readers will pass it up.

*Don't rely on citizens to set the agenda. A staple of public journalism election coverage is the early poll that asks citizens what issues they are concerned about. The foolish way it's used squeezes the creative juices out of a campaign.

In the 1996 election for a U.S. Senate seat in North Carolina, black Democrat Harvey Gantt wanted to challenge white Republican Sen. Jesse Helms on racial issues. But polls of the public journalism consortium showed that voters didn't want to hear about race. Reporters ignored Gantt's message. It never got out, and he lost.

In January 1998, an expanded consortium of newspapers and broadcasters in North Carolina took another poll and found that the percent of respondents concerned about our nation's military preparedness was in the low single digits. So they scratched that one from their priority campaign coverage. The timing of that decision was terrible. At that moment in history, trouble was brewing in Iraq and in Yugoslavia that could easily have led to more demands on our military than it was equipped to meet. The poll showed that the people were asleep on that issue. Public journalism in this case meant taking care not to wake them.

The candidates weren't too interested, either. Republican Lauch Faircloth, who lost his 1998 bid for a second Senate term, told me later that he had no desire to campaign on the issue of military preparedness. "I tried it in a couple of speeches, and it was a turnoff," he said.

Where is the motivation to discuss painful issues to come from? If the candidates don't want to do it, and if the people are dozing, who's going to nudge them awake? This is why we have editors. It's OK to use polls to find out where the public is asleep, but journalists should not be afraid to wake it up when they think the issue is important. To fail to do so is a spineless abdication of responsibility.

Some tools that work

*Issue grids are good. The best thing about a grid that lists issues on one dimension and candidates on the other is that the voids become obvious. No candidate wants to see a blank space by his or her name and picture because of a failure to take a position. So the grids, by their very existence, help spur conversation among the candidates and the electorate. It is an ideal tool for Schudson's monitorial citizen, because it gives just enough information to tell a reader whether he or she can relax or has a need to learn more. Where will the "more" come from? This is where your newspaper's investment in a Web site pays off. Use it to expand on what appears in print.

*Information campaigns can work in single-issue elections. The best public journalism success stories have come about in referendum elections. Daniel Yankelovich, psychologist, author and founder of the market research firm bearing his name, is the intellectual father of public journalism. In his 1991 book "Coming to Public Judgment: Making Democracy Work in a Complex World," he said the public can't think about more than two-and-a-half issues at any given time. That makes a single-issue referendum a special opportunity, because the voter has only one decision to think about.

One much-publicized early study that seemed to show a connection between public journalism and increased voter turnout proved to be flawed. Mecklenburg County, home to the Charlotte Observer, had a dramatic increase in turnout in the 1992 election--but so did other major cities in North Carolina where there was no public journalism. In fact, Charlotte's increase was less than in some of the nonparticipating cities. The higher turnout stemmed partly from the close national race between President Bush and Democratic challenger Bill Clinton, spiced by the third-party antics of Ross Perot. It was also a Senate election year in North Carolina, which had not been the case in the 1988 election to which it was compared.

But when the Rochester Democrat and Chronicle applied public journalism techniques to a 1997 statewide referendum on whether to hold a constitutional convention, the results were palpable. In the six-county area covered by the newspaper and its broadcast partner, PBS' WXXI, 82 percent of those who voted in the general election also voted on the convention referendum, according to James R. Bowers of St. John Fisher College. This compares with 71 percent in the remainder of Upstate New York and only 36 percent in New York City.

Could New York City always have been that much lower? No. The 1997 turnout was a definite departure from historic voting patterns. Bowers found that the three areas had been significantly more similar to each other in turnout in referendum elections since 1990.

*Public meetings help. A newspaper should be the "convener of the community," said Rich Oppel in 1992, when he was editor of the Charlotte Observer. Getting citizens to talk to one another is good, because when they meet, they start to modify their ideas to accommodate to one another. That was the main function of that democratic archetype, the New England town meeting. A citizen attended not just to express his or her views, but to hear and adjust to opposing views. Experiments in "deliberative polls," in which a representative sample of the electorate is assembled to meet candidates and discuss issues, have made that clear. By attaching real people to the views they express, the meetings arouse interest generally.

*Just making an effort is helpful. In the Meyer-Potter study of the 1996 presidential election, there was evidence of a mysterious "X" factor at work in the markets where public journalism was practiced. Potter and I measured the intent of 20 newspapers and TV stations to practice public journalism, and then we measured their content to see if their deeds matched their intentions. They did. The self-identified public journalists had more issue content and less horse-race information in their respective media.

But we did not find any connection between this content and the presumed effects of public journalism. Strangely, the intent of newspapers to practice public journalism was a stronger predictor of voter trust and knowledge than was their actual performance. This discrepancy suggests a missing link somewhere. One possible explanation for the missing link is that there was already something wholesome going on in the relationship between those newspapers and their communities. Perhaps it was the soundness of the community ties that inspired the editors to do public journalism in the first place. Kris McGrath's 1985 study for the American Society of Newspaper Editors found evidence of a connection between community ties and newspaper credibility. As the rock group America sang in the 1970s, maybe Oz never gave the Tin Man what he didn't already have.

Or it might have been the Hawthorne effect. Named for a famous experiment in worker efficiency in the 1920s, it means that visibly caring enough to pay attention does more than any specific expression of that care. Show that you care, and people will respond. It can confound our attempts to analyze and isolate the specific mechanisms of public journalism, but it's still an effect worth having. And that's the best news of all. Even bad tools, if they are visible enough, can have a good effect.