The Trouble with the Texas Tribune’s Poll

The following is the first of many guests posts we hope to publish, which will bring additional expertise and insight to the Don’t Grow Texas community. R.G. Ratcliffe is a senior political writer and analyst with decades of experience in newspaper journalism.

When I was a child in Dallas, I possessed a rock of salt the size of a bowling ball that came from the Morton Salt mine in Grand Saline. It looked like a rock, but was salty to the taste when you put your tongue on it. I think of that rock every time I see a University of Texas/Texas Tribune political survey. Like that salt rock, it looks like something other than what it actually is.

During the course of my journalism career, I wrote about dozens – if not hundreds – of political surveys. The poll is to a political reporter what the tout sheet is to a horse-race junkie. From the perspective of having watched the sausage made, I can tell you all political polls have about them an element of voodoo.

R.G. Ratcliffe

But the opt-in Internet survey methodology used by the U.T. pollsters and the Texas Tribune may be one of the most black magic of all the polling methods. It essentially uses people who have volunteered to be surveyed and then uses statistical weighting to make the results match the expected voter turnout. (Click here to see About These Polls). It’s a survey methodology so suspect that news organizations such as The New York Times, The Washington Post and Roll Call magazine have refused to use it.

Now, when the most recent survey came out, showing Republican Greg Abbott was leading Democrat Wendy Davis in the governor’s race by 47 percent to 36 percent, my liberal friends groaned in agony that the survey was completed before Abbott received a lot of bad publicity for campaigning with has-been rocker Ted Nugent.

But even if the survey had been done during the bad publicity, that’s like saying we’re going to rely on unforeseen events to win the election. We saw this effect in 1994 when the Houston Chronicle and The Dallas Morning News combined on a survey in the governor’s race. It was a three day survey. On day one, George W. Bush led by about five percentage points. On day two, Ann Richards received the endorsement of Ross Perot and the survey found the race tied. On day three, Bush was back to having a five point lead. It was obvious that days one and three were the accurate days, but we had to report the entire survey, so it showed Richards narrowly training Bush. He ended up winning the following week by seven percentage points.

The biggest problem with the U.T./Tribune poll was not when it was done but how it was done. The opt-in survey is fast and cheap and may only be more reliable than one of those television station click surveys because a trained professional political scientist is weighting the results.

(Click here to see the Sampling and Weighting Methodology for the February 2014 Texas Statewide Study. Keep in mind, they say there is a YouGov panel of 20,000 Texans registered, and 1,327 opted to take the survey and then they winnowed that down to 1,200 to create the final dataset. Here’s some key numbers to keep in mind, the Republican primary results were drawn from a panel of 543 voters while the Democratic primary numbers were drawn from a panel of 381.

Those numbers right there would appear to give the survey – and possibly rightly so – a Republican bias. Let’s face it, all of you who see Red because you’re so Blue, the top Democratic vote-getter of the past decade scored just 44.8 percent of the statewide ballots. Maybe it’s the state, maybe it’s the U.T./Trib survey.

“Buyer Beware: There are polls, and Then There Are Polls,” headlined a report by national political observer Stu Rothberg in a column for Roll Call, the prestigious magazine of Capitol Hill. “Wisely, in my view, Roll Call does report opt-in online surveys,” Rothberg wrote. (Click here to read Buyer Beware: There Are Polls, and Then There Are Polls)

Rothenberg this past November took a quick look at the polling, comparing how a Public Policy Polling survey gave Abbott a 15 point lead over Davis, while a U.T./Tribune survey done just days earlier using the opt-in method found it to be a six point race. The UT/Tribune poll had Abbott’s support at 40 percent, while the PPP survey had his support at 50 percent. (Interesting to note, the new Tribune survey is more in keeping with the PPP survey of last fall.)

These opt-in surveys have been subject to a great deal of criticism in the polling industry in the past. The American Association of Public Opinion Research warned against using them. A Stanford University study in 2009 found problems with this polling methodology, and an ABC News report stated that ABC, the Associated Press, The New York Times and the Washington Post declined to run such polls because of questions of their accuracy. (Click here to read about survey opt-in troubles).

A research paper by Langer Research Associates in 2012 warned against the use of such polling because “there is no theoretical basis on which to conclude that they produce valid and reliable estimates of broader public attitudes or behavior.” Their paper concludes with numerous links to studies on opt-in surveys. (Click here for more survey opt-in studies) 

No doubt some improvements have been made in the opt-in survey methodology in recent years. And the Associated Press joined the Internet polling age last year by partnering with a German firm. But the AP noted that its surveys would be based on a panel developed both on the Internet and by telephone. Those who chose to participate who did not have Internet access would be given a laptop and a connection. (Click here to read Moving the AP-GFK polling online). And AP noted that many of the older opt-in methods remained unreliable.

Although opt-in polling is fast and cheap, some national news organizations still refuse to use them.

The New York Times, which some may know as the Texas Tribune’s content partner, has a stated policy about why it will not use polling such as that run by the Tribune: “Self-selected or ‘opt-in’ samples – including Internet, email, fax, call-in street intercept, and non-probability mail-in samples – do not meet The Times standards, regardless of the number of people who participate.” (Click here to see the New York Times Polling Standards).

The Washington Post on multiple occasions has warned against relying on opt-in surveys. “But there are polls — and there are polls. Regular readers know that we have urged caution against relying on opt-in Internet surveys that appear to make broad claims about estimating population values.” (Click here to read about “a misleading Obamacare poll”).

In mid-February, the Post noted that right wing web sites went nuts with a YouGov survey that found 71 percent of President Obama’s voters regretted their vote, but when the Post took a hard look at the survey it found that was based on just 35 respondents. (Click here to see the survey). 

All of this is not to say the U.T./Tribune poll is a rigged game or a wild roll of the dice. But it does say the average reader needs to know a lot more about the panel. How does it stack up in men versus women? Are men more likely to fill out an online survey than a woman? The Davis campaign and Battleground Texas are counting on turning out Hispanics who do not normally vote, how is this accounted for in the survey?

Essentially, at the moment, I could tell you from a quarter century of covering Texas politics that the race is 45 percent Abbott and 35 percent Davis. Why? Because that is the base vote of both parties at the present, and the voters who will decide the election will not start paying attention until September.

At the moment, you can either believe Abbott’s gain in the U.T./Tribune survey was a real increase in support or that the October survey was an anomaly of a survey methodology that several major news organizations still reject.

But apparently, the Texas Tribune knows better than ABC, The New York Times and The Washington Post. And the rest of the state’s news media, hungry for a pulse of the race, dutifully reports the results as if handed down to Moses on Mount Sinai.

Personally, I prefer to remember that rock of salt that rested for years in the front flowerbed of an orange-bricked house on Preston Crest Lane.