Brooks vs. Silver: The Limits of Forecasting Elections

Updated below.

In Tuesday’s Times, David Brooks had a pop at political forecasters, including his colleague, Nate Silver, whose blog, FiveThirtyEight, is a popular feature on the Times Web site. Of course, Brooks was too polite to personalize his argument, but given Silver’s popularity and profile there can be little doubt whom Brooks was referring to when he wrote “I know … how I should treat polling data. First, I should treat polls as a fuzzy snapshot of a moment in time. I should not read them, and think I understand the future. If there’s one thing we know, it’s that even experts with fancy computer models are terrible at predicting human behavior.”

Silver, of course, spends much of his time analyzing the latest polls and converting them into election projections. On a much more casual basis, so do I, and so does Brooks, who entitled his column: “Confessions of a Poll Addict.” The difference is that Silver has built a mathematical forecasting model, and people take it very seriously: he is widely regarded as a political astrologer who gets things right. In the 2008 Presidential election, he correctly picked the winner of forty-nine states. Political junkies of all stripes, but particularly Democrats, follow his site religiously. Over the past few weeks, as the opinion polls in this year’s Presidential election have tightened, I’ve lost count of the number of people who have said to me something along the lines of, “Yes, but Nate Silver still says Obama’s got a seventy-five per cent chance of winning.”

As it happens, he doesn’t. On Wednesday afternoon, FiveThirtyEight said that the probability of an Obama victory was 68.1 per cent, and the probability of Mitt Romney winning was 31.9 per cent. That’s roughly in line with the online bookmakers, which now have Obama as the 1/2 favorite, or thereabouts, which implies that his probability of winning is 66.7 per cent. (At Intrade, the political prediction site, the implied probability of an Obama victory is now quite a bit lower. On Wednesday afternoon, it was 57.9 per cent.)

According to Brooks, all of these figures are basically meaningless. “If it’s hard to predict stocks or the economy, politics is a field perfectly designed to foil precise projections,” he wrote. And he continued:

Stuff happens. Obama turns in a bad debate performance. Romney makes offensive comments at a fund-raiser. These unquantifiable events change the trajectories of tight campaigns. You can’t tell what’s about to happen. You certainly can’t tell how 100 million people are going to process what’s about to happen. You can’t calculate odds that capture unknown reactions to unknown events.

As op-ed columnists often do, Brooks overstates his case. Up to a point, political outcomes are predictable. Sitting here today, I can forecast with certainty that neither candidate will win the popular vote by more than ten percentage points. History and common sense tells us this. Indeed, based on the opinion polls, I can predict with a good deal of confidence that neither candidate will win by more than five points. Voting patterns aren’t completely random. They reflect history, demography, sociology, economics—fundamental relationships exist that statisticians can, in principle, attempt to capture and turn into mathematical equations.

Unfortunately for forecasters, people aren’t interested in rough predictions, such as “it’s going to be close.” Especially in a tight contest, such as this year’s, they want to know who is going to win. And when it comes to answering this question, voting models based on fundamental factors don’t work very well. Years ago, I got interested in the work of Ray Fair, a Yale economist who pioneered the development of forecasting elections based upon a few simple economic statistics, such as G.D.P. growth, inflation, and unemployment. Fair’s record is mixed. He’s called most elections correctly, but he’s also got some wrong, notably 1992, when he predicted a Republican victory, and 2000, when he said that the Democrats would win. (In 2000, to give Fair his due, he correctly predicted that the Democrats would win the popular vote, which they did. But Al Gore lost the election in the electoral college.)

According to one expert who has looked closely at the record, “The ‘fundamentals’ models, in fact, have had almost no predictive power at all. Over this 16-year period, there has been no relationship between the vote they forecast for the incumbent candidate and how well he actually did—even though some of them claimed to explain as much as 90 percent of voting results.” The expert who wrote these words was Silver. If you want to learn about why most political predictions turn out to be wrong, I thoroughly recommend some of the posts he has written on the subject. And if you’ve got time, you should also consider reading his new book—“The Signal and the Noise: Why So Many Predictions Fail But Some Don’t,” which is largely devoted to the limits of forecasting in areas such as sports, finance, and business—as well as politics.

What I do question—and here I sympathize with Brooks—is the gravity with which some people treat the figures generated by the FiveThirtyEight voting model. Somehow, we are asked to believe that Silver has managed to avoid the pitfalls that affect other forecasters—to the extent that he can generate accurate voting projections down to one decimal point. That is silly. Election forecasts are great fun, but nobody should take the individual predictions they generate too seriously, wherever they come from, or however they are constructed.

FiveThirtyEight’s forecasting model, like any other forecasting model, is subject to at least two sources of error: sampling error and model error. Let’s first take sampling error, which reflects the unavoidable fact that the opinions elicited by pollsters might not be truly representative of the electorate as a whole. When FiveThirtyEight projects, as it does today, that Obama will get 50.0 per cent of the vote and Romney will get 48.9 per cent, what it is really saying—assuming a margin of error of 2.5 per cent (for some reason, Silver doesn’t publish the actual figure)—is that Obama’s vote will fall somewhere between 47.5 per cent and 52.5 per cent, and Romney’s will fall somewhere between 46.4 per cent and 51.4 per cent. In short, it’s too close to call.

Then there’s model error, which, roughly speaking, means that nobody is God. Since statistical modellers don’t know precisely how the world works—if they did, they wouldn’t need a model—they have to rely on something that is necessarily inaccurate—the question is how inaccurate. Silver, having dismissed the fundamental factors in explaining voting patterns, doesn’t put forward any alternative theory of voting behavior. Instead, he relies heavily on opinion polls and attempts to extrapolate from them. (His model also incorporates some economic factors, but, as the election nears, the polling data is what overwhelmingly drives it.) In fact, one way to think about the FiveThirtyEight model is as a proprietary poll of polls, supplemented by some complicated but relatively routine statistical simulations. If, in any given campaign, the polls remain stable through election day, Silver’s early projections are likely to be pretty accurate. If the polls swing dramatically, as they have this year, his early predictions could turn out to be way off. (Even then, though, his projections from nearer to the day of the election, which reflect the latest poll figures, could be considerably closer to the mark.)

I don’t mean to dismiss Silver’s mastery and manipulation of the data, which is impressive. At the risk of simplifying things quite a bit, he takes all the new polling numbers that are published every day, weights them according to how reliable he considers the survey, adjusts them to reflect some other factors—such as how likely each party’s supporters are to vote—and converts them into projected-vote shares at the state and national level. To convert these projections into electoral-college votes, his model then simulates a large number of elections, using a random-number generator to determine vote tallies and the electoral-college votes associated with them. If Obama comes out on top in three hundred thousand of five hundred thousand simulations the probability of him winning is sixty per cent.

Enabled by the development of powerful desktop computers, this statistical methodology, which is known as Monte Carlo simulation, is now widely used in physics, finance, and engineering. As a method of predicting how a stable system with fixed parameters will behave over an extended period of time, it is very useful. Alternatively, it can be used productively to study the evolution of a more chaotic system with lots of data, over short periods of time. Applying it to systems where abrupt shifts are possible—such as bond markets or the outcome of Presidential elections—is a more perilous exercise. During the housing boom, big banks used Monte Carlo simulations to figure out how much they stood to lose by holding onto things like subprime-mortgage securities. Once house prices started falling all across the country—something that hadn’t happened since the nineteen-thirties—these simulations weren’t worth the paper they were printed out on.

Silver, to be clear, isn’t comparing Obama’s prospects to a AAA bond. Such securities are supposed to have a miniscule chance of defaulting. According to FiveThirtyEight’s current projections, Obama’s chances of losing are close to one in three, which sounds reasonable to me. But that merely reflects the model catching up with reality. This time three weeks ago, it was saying that Obama was a virtual certainty to win, which is what almost all the pundits, myself included, were saying. You didn’t need a Monte Carlo simulator to read the polls.

On October 3rd, the day of the first Presidential debate, the Real Clear Politics poll of polls had Obama ahead by three points. The FiveThirtyEight model said the probability of him winning the electoral vote was 86.1 per cent, and it was projecting a margin of victory of more than four points: 51.4 per cent to 47.5 per cent. In a typically data-heavy and informative post, Silver suggested that Romney, like most challengers, would probably get something of a bounce, but it wouldn’t be big one. Since 1976, he pointed out, the biggest post-debate bounce any candidate had enjoyed was three per cent. (Ford in ’76 and Kerry in ’04.) Six days later, Romney took the lead in the R.C.P. poll of polls, and, apart from a couple of days last week, he’s held onto it ever since.

As Brooks said, “stuff happens”—not all the time, but more often than you might think on the basis of a sample of nine elections, going back to 1976, or, indeed, in a sample of nineteen elections, going back to 1935, when George Gallup invented his eponymous poll. Evidently, the FiveThirtyEight forecasting model, like virtually all such models, isn’t very good at predicting turning points. In 2008, there were no surprises. Obama led in the polls throughout, John McCain picked a disastrous running mate, the debates went off without any major incidents. Two weeks before the election, the Gallup poll had Obama up by seven points, which turned out to be his winning margin. Silver had a good year, too.

This year, we’ll see what happens. No candidate in recent times has won from as far back as Romney was in late September, and, until this year, there was no historical tendency for the trailing candidate to make up ground in October—two of the many valuable facts that I have learned from Silver. Now that Romney is surging, and has been surging for several weeks, how likely is he to go on and win? The FiveThirtyEight model, lacking useful precedents or any theory of what determines how people vote, can’t really help us answer that question. It essentially says that if Obama continues to do relatively well in the battleground states, he will probably scrape through. That’s a useful thing to know—in fact, I’ve been saying much the same thing myself—but does it mean that Nate Silver, or anybody else, really knows who’s going to win the election? No.

Thursday morning update: Clearly some people don’t like this post, which is fair enough. One of the roles of the columnist is to provoke, and, occasionally, enrage. Rather than address the criticisms in the comments section individually, I’ll say a few things and leave it at that.

Firstly, it was not my intention to make a political point, or to give succor to the Republicans. As I indicated, I agree with Silver that Obama is still ahead where he needs to be ahead, and that he is likely to win. What I query is the apparent precision of the FiveThirtyEight forecasts, and whether its model adds very much to the opinion polls.

Secondly, in reaching for a strong conclusion, I was overemphatic. Clearly the model helps us to think about the likelihood of a Romney victory: the question is how much. In encapsulating the latest polling data in a timely manner, its forecasts provide a valuable benchmark, to which I and many others regularly refer. But that doesn’t mean it is entirely reliable.

Finally, I will be perfectly happy to be proved wrong. After spending many years watching economists and financial analysts struggling to predict the future, I perhaps have a jaundiced view of the usefulness of statistical forecasts in the social sciences. If a super-smart fellow like Silver has indeed demonstrated a reliable way to predict political outcomes, it represents real progress. The proof of the pudding is in the eating. If Silver calls the popular vote and the electoral college correctly again this year, and if his model outperforms the Real Clear Politics poll of polls in the battleground states, I will gladly eat humble pie and send round a bottle of champagne to the FiveThirtyEight offices.

See our full coverage of the campaign season at The Political Scene.

Photograph by Damon Winter/The New York Times.