By their fruits you shall know them

I started off yesterday with a quick jaunt through some of the major predictions for how the electoral college would pan out yesterday during my live-blogging of the night’s returns. What I think is absolutely necessary to do now is the review those and look at exactly how things turned out. A few more honest predictors have already admitted it, but whether they recognize it or not, a shorter version of this is already well-known and getting passed around the internet today:

Keep calm and trust Nate Silver
(I first saw it here, but who knows who put this together originally.)

That conclusion having already been drawn, I think it’s worth analyzing it further. How much is trusting in Nate Silver actually worth (in comparison to other sources)? There’s a simple measurement to be done in terms of how accurate his calls were in comparison to the overall outcomes (hint: pretty much spot on). But there’s a variety of different problems with other predictions having flaws other than inaccuracy, most typically unwarranted uncertainty. It might be safe for pundits to only predict states that you can be absolutely certain about, but if you can’t provide an estimation for trickier situations, why should viewers rely on you for information? I want to be provided a service and I want it to be excellent. I’ll allow for some uncertainty, as it’s inevitable, but beyond a certain point, it’s an excuse for not doing your job, if your job involves predictions.

With those two potential problems in mind, I think it’s worth looking at both accuracy and overall information provided. To freshen in your minds what the various predictions were and to make comparisons easier to understand, here are the major predictions that were made as of Monday night, in map-form:

This slideshow requires JavaScript.

And here’s how those translate into the usefulness of the prediction:

Prediction Correctly Predicted Polities (#) Predicted Polities (#) Correctly Predicted Polities of Predicted Polities (%) Correctly Predicted Polities of All Polities (%)
Nate Silver (1) 51 51 100% 100%
Nate Silver (2) 50 50 100% 98.0%
RCP 40 40 100% 78.4%
Sam Wang 50 50 100% 98.0%
Michael Barone (1) 43 51 84.3% 84.3%
Michael Barone (2) 43 49 87.8% 84.3%

*I’m counting with 51 electoral blocs, as neither Nebraska or Maine appear to have split, and the District of Columbia gets 3 electoral votes as well.

You’ll notice, it’s relatively easy to get a high score in predictions, if you don’t make any for the tighter states. On the other hand, it’s a bit trickier to translate your predictions into a high degree of accuracy for the entire map unless you actually know what you’re doing (like, say, a professional statistician). In fact, counting only the errors in polity predictions understates the problem in the flawed analyses. I count 25 polities with fewer than 10 electoral votes each which all other these predictions agreed on. Allowing them to be factored in gives everyone a greater than 50 percent prediction rate while only working with 127 electoral votes. To be fair, there’s only 115 electoral votes from 9 states that the various models included here disagreed on. For the terminally curious, those are Nevada, Colorado, Iowa, Wisconsin, Ohio, Pennsylvania, New Hampshire, Virginia, and Florida.

Looking just within those selected states creates a very different perception:

Prediction Correctly Predicted Select (#) Predicted Select (#) Correctly Predicted Select of Predicted Select (%) Correctly Predicted Select of Total Select (%)
Nate Silver (1) 9 9 100% 100%
Nate Silver (2) 8 8 100% 88.9%
RCP 0 0 N/A 0%
Sam Wang 8 8 100% 88.9%
Michael Barone (1) 1 9 11.1% 11.1%
Michael Barone (2) 1 7 14.3% 11.1%

So there you have it. Sam Wang and Nate Silver are both highly accurate and highly useful. Real Clear Politics was accurate, but only because they literally said nothing about where it really mattered. As for Michael Barone, whose hilarious predictions can be read here in full (although they might disappear shortly), there’s a reason he could only get published at Rasmussen. Take note, statistics just defeated an actually baseless “sense” in yet another round. I’ll hope that Real Clear Politics’ cowardice and Michael Barone’s wrongness don’t go unnoticed, but I won’t hold my breath.

Advertisements
Tagged , , , , , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: