Tilt   

Online Exclusive » Two University of Chicago economists’ findings about the political slant of American newspapers are based on a linguistic version of “fuzzy math.”

Interested in joining the liberal media elite?

Just publish a newspaper that writes about "family values," the "Republican Party" and "Vice President Cheney." It doesn't matter what you write on those subjects--mere mention is enough.

That's the gist of a recent study by two University of Chicago economists. Lauded in a New York Times column, it's also been cited on National Public Radio and is picking up steam in the blogosphere. All of the attention is coming even before the study has been published in an economics journal.

But scratch beneath the provocative uses of the term "death tax," and there's less there than meets the eye.

One of the researchers' primary conclusions is that newspapers tend to match the ideology of their markets. A liberal newspaper ain't gonna sell in Cheyenne, the thinking goes, so a newspaper there will write the stories that make it more conservative.

But how exactly do the researchers determine a newspaper's ideology?

Starting with the Congressional Record, authors Matthew Gentzkow and Jesse Shapiro determined phrases that Republicans use and phrases that Democrats use. Next, they measured how often newspapers printed those phrases in their news stories.

They checked for 1,000 "politically loaded" phrases in 417 newspapers. After analyzing the number of times an article used the phrase and the phrase's level of partisanship, they derived the "slant" for each newspaper and ranked it as if it were a member of Congress.

Newspapers that used more Democratic phrases, such as "tax cuts for the wealthy," were categorized as leaning Democratic, while those using Republican phrases, such as "repeal of the death tax," were classified as leaning Republican.

Beyond the conclusion that newspapers respond to the ideology of their markets, Shapiro and Gentzkow also found that newspaper chains don't impose an ideology on their papers nationwide.

But the conclusion I think will cause a stir among the media-bias crowd is this: "[W]e find that the average newspaper's language is similar to that of a left-of-center member of Congress." They add their results are "consistent" with a study from 2005 that claimed the media had a strong liberal bias.

Overall, the new study finds the average newspaper has a "slant" of .47, or slightly left of center. It says three major newspapers–the New York Times, the Los Angeles Times, and the Washington Post–have a slant of .43 and are similar to "a fairly liberal congressperson" such as Democratic Sens. Barbara Boxer of California or Charles Schumer of New York.

The study also ranks members of Congress based on the language they use. Oddly, the senator ranking closest to those papers is Republican Johnny Isakson of Georgia; Republican Sen. Tom Coburn of Oklahoma ranks to the left of all three. Both score a perfect 100 percent in the American Conservative Union's congressional ratings.

Sure, every big study has individual oddities. They shouldn't negate the overall conclusions–if the underlying methodology is sound.

Is it?

Only if you believe that writing about "American people" makes a newspaper liberal.

The researchers crunched millions of words and millions of stories. But their exercise has structural flaws, as well as problems with the quality of their data:

• The researchers excluded copy from wire services, such as the Associated Press, because those stories aren't always archived in news databases. But for the vast majority of small- and mid-size newspapers, wire services supply all national and international news. One of an editor's greatest sources of ideological power is deciding whether to run a national political story at all. So it's impossible to know how ideological most papers are if a huge amount of the news they run isn't measured. The lack of such wire service stories is partly why some papers were analyzed based on 3,000 or 4,000 phrase mentions while others were analyzed based on 50,000 or more.

• Some partisan phrases have multiple meanings. Consider the very first example they highlight in the paper: The Washington Post used Democratic "estate tax" 10 times more often than Republican "death tax." But that calculation includes every time the Post wrote "real estate tax" in the metro, business and homes sections (which accounts for a majority of all mentions, my review in Nexis shows).

• The researchers include 417 newspapers–a big number, but just a third of newspapers nationwide. The authors said their study used all papers available in their databases, which meant they got almost all the larger papers in the country but a much smaller share of small papers. By wide a margin the larger papers were on the liberal side of the authors' rankings.

• Of the 1,000 phrases, 617 are Democratic and 383 are Republican. The reason for that disparity is unclear, and the authors' calculations don't require that the phrase count break exactly even. But it makes me wonder if they have truly figured out how to gauge the real differences in partisan language.

• Given limitations in news databases, it's difficult to reliably exclude wire service copy. Indeed, the study only cites two specific stories, and in an interview, Shapiro mentions them as well. But one of those stories shouldn't even be in the study, since it's a wire service story about the "death tax" from a Salt Lake City paper. Yes, yes: That's one story out of millions analyzed. But it's one they chose to highlight.

Or take opinion pieces, which they excluded "whenever possible." While I'm not using the same news database they did, it is possible to come close to matching their data runs. Their data, for example, have the New York Times printing the Democratic phrase "bring our troops home" 20 times last year; my review of Nexis has 21 mentions of that phrase, but nine of those are in letters, columns or editorials (the rest are news stories that quote or paraphrase politicians or war protesters). Their data have "tax cuts for the rich" in the Times 22 times; my review has it 20 times–and 18 of those are editorials, columns or letters to the editor. In other words, content that should have been excluded--and the kind of opinion-page matter that could have skewed a newspapers' measure of ideological slant.

• Then there are strange gaps in their data. Is it true the Washington Post didn't use the phrase "political party" at all last year? Or mention the "National Security Agency"? (Somebody really should alert the national intelligence reporter over there.) Or "senior citizens" or "natural gas"? The data say the Post mentioned a "nursing home" eight times. LexisNexis shows at least 500 mentions each for "political party," "senior citizens" and "nursing home," more than more than 400 for "natural gas" and more than 200 for "National Security Agency."

• Most perplexing is the substantial mismatch between what the authors write and what their data show.

The text of the paper gives the impression that politically loaded phrases such as "death tax," "tax cuts for the wealthy" and "war on terror" drive the results.

But in the New York Times, for example, "death tax" and "estate tax" were used a combined 149 times. "Credit card" (1,238 times) and "Justice Department" (1,128 times) were the "partisan phrases" used the most. Both phrases, according to the study, have a Democratic slant.

Assume the New York Times never used the phrase "tax cuts for the wealthy." What would have happened to its score?

Virtually nothing. Indeed, eliminating all mentions of "Memorial Day" would have had a bigger impact. Killing references to "Rhode Island" or "credit card" also would have changed the Times' score more.

Looking at data for 12 large papers provided by the authors for this story, "African American" had more impact than any other phrase. If those newspapers used "black" instead of "African American," the score would have moved a full four percentage points to the right--a huge change. The Republican phrase with the most impact was "natural gas" (which also happens to be one of those phrases subject to strange data gaps).

Examining all 1,000 phrases reveals a huge number that have nothing to do with ideology--phrases such as "pass the bill," "urge support," "assistant secretary" and "witnesses may testify," all deemed to have a Republican slant.

Among the most liberal newspapers in the study: the Times-Picayune of New Orleans. Among the most Democratic phrases: "Hurricane Katrina."

In interviews and e-mails, Gentzkow and Shapiro say their study's power lies in sheer numbers. There may be some irregularities--the "real estate tax" vs. "estate tax" issue--but Shapiro says that "when you do it on this scale, lots of those errors will average out."

The inclusion of opinions and editorials should also not be a concern, he said. While they know the search procedures can't "guarantee exclusion of all editorial content," they did perform other tests on the data to see if it would be a problem. They have "confidence that the inclusion of some editorial material is not a major driver of our results," Shapiro says.

Gentzkow was perplexed by the missing data – phrases like "political party" that did not appear in the Washington Post. "That's not something I had noticed," he said in an interview a week ago. By that night, he had identified a glitch in the programming for 40 of the papers, including a big share of the larger ones. "We are fairly confident that correcting these errors will not change the paper's results," he said, but added, "there is no excuse for letting an error of this kind remain in results we were distributing publicly." He said they will re-run the faulty data.

Both said that emphasizing the "left-of-center" finding is a distraction from their main point: Newspapers tend to mirror their markets. The actual position of newspapers on the slant scale is less important than the relative position–i.e., the Wyoming Tribune-Eagle in Cheyenne is to the right of the San Francisco Chronicle. In fact, Gentzkow wrote in an e-mail: "We are also happy to concede that the left-of-the-median-congressperson result (besides being not especially interesting) is likely to be sensitive to changes in methodology in a way that our central findings are not."