Nate Silver at the NYTimes has been widely respected for his ability as a statistician.
His reputation, though, seems to stem largely from his facility at what amount to rhetorical parlor tricks (he once earned a bit of a living counting cards at poker, and he made a name for himself with baseball stats), and his calling of the vast majority of the 2008 election slate correctly (with the help of an epochal wave election and lots of access to Obama campaign internal polling), leading to his hiring at the NYTimes in time for the 2010 race.
Silver’s method at the NYTimes involves…:
- Taking regional polls – from polling services as well as media polls – and…
- “weighting” them according to some special sauce known only to Nate Silver, Registered Statistical Genius.
Now, I wrote about Silver’s method two years ago, when he spent much of the race predicting Mark Dayton would win by six points (with an eight-point margin of error). As I pointed out, Silver’s “methodology” involved giving a fairly absurd amount of weight to polls like the long-discredited Star Tribune “Minnesota” Poll and the since-discontinued Hubert H. Humphrey Institute poll (for whose demise I sincerely hope I deserve some credit, having spent a good part of the fall of 2010 showing what a piece of pro-DFL propaganda it has always been). During the middle of th 2010 race, Silver gave the absurdly inaccurate-in-the-DFL’s-favor (especially in close elections) HHH and Minnesota Polls immense weight, while undervaluing the generally-accurate Rasmussen polls and, to a lesser extent, Survey USA.
I said Silver’s methodology was “garbage in, garbage out” – he uses bad data, and gets bad results. I was being charitable, of course; his methodology, untransparent and proprietary as it is, processes bad data into worse conclusions.
That was in 2010.
Today? NRO’s Josh Jordan reaches the same conclusion:
While many in the media (and Silver himself) openly mock the idea of Republicans’ “unskewing polls” (and I am not a fan of unskewedpolls.com by any means), Silver’s weighting method is just a more subtle way of doing just that. I outlined yesterday why Ohio is closer than the polls seem to indicate by looking at the full results of the polls as opposed to only the topline head-to-head numbers. Romney is up by well over eight points among independents in an average of current Ohio polls, the overall sample of those same polls is more Democratic than the 2008 electorate was, and Obama’s two best recent polls are among the oldest.
But look at some of the weights applied to the individual polls in Silver’s model. The most current Public Policy Polling survey, released Saturday, has Obama up only one point, 49–48. That poll is given a weighting under Silver’s model of .95201. The PPP poll taken last weekend had Obama up five, 51–46. This poll is a week older but has a weighting of 1.15569.
So it wasn’t just Minnesota!
And remember – PPP polls, while leaning a little left, are not generally flagrantly inaccurate in the sense that the Strib is and the HHH was.
And it’s not a fluke…:
The NBC/Marist Ohio poll conducted twelve days ago has a higher weighting attached to it (1.31395) than eight of the nine polls taken since. The poll from twelve days ago also, coincidentally enough, is Obama’s best recent poll in Ohio, because of a Democratic party-identification advantage of eleven points. By contrast, the Rasmussen poll from eight days later, which has a larger sample size, more recent field dates, but has an even party-identification split between Democrats and Republicans, has a weighting of .88826, lower than any other poll taken in the last nine days.
Jordan reaches a conclusion that even I didn’t:
This is the type of analysis that walks a very thin line between forecasting and cheerleading. When you weight a poll based on what you think of the pollster and the results and not based on what is actually inside the poll (party sampling, changes in favorability, job approval, etc), it can make for forecasts that mirror what you hope will happen rather than what’s most likely to happen.
Well, you can – if your goal isn’t so much to measure the nation’s zeitgeist (and report on it) but affect the election.
Which has, of course, been my contention all along.