In 1982, the decision analysts Marc Alpert and Howard Raiffa asked students in their classes at Harvard to estimate figures like the number of eggs produced daily in the United States. They were each asked to give a range so wide that they believed there was a ninety-eight-per-cent chance it would include the right answer. Instead, the correct figures fell inside the students’ ranges less than a third of the time.
Innumerable decisions depend on forecasting what will happen in the future. Indeed, economic forecasts are so important that the U.S. government generates forty-five thousand of them each year. And yet, as Nate Silver notes in “The Signal and the Noise,” these forecasts often aren’t very good. Silver’s political-forecasting work has shown that good statistical analysis of quantitative data can provide much better forecasts than can human intuition. But what happens when forecasts must be made without the benefit of, say, the mountains of polling data that inform Silver’s statistics? That is, what happens when you have to rely, in large part, on educated guesses from humans?
In 2005, the political psychologist Philip Tetlock found that the pundits who give the most interesting interviews tend to be rewarded with more coverage in the press. But this “interestingness,” he wrote in “Expert Political Judgment,” is weakly correlated with accuracy. Pat Buchanan’s forecasts are bold and, often, totally wrong: consider his July, 2012, prediction that Bashar al-Assad would “be gone” by Labor Day that year. Buchanan’s low accuracy earns him a grade of D from pundittracker.com. But that hasn’t stopped Fox News and “The McLaughlin Group” from featuring his views.
In the courtroom, too, juries find confident witnesses more persuasive, according to research by Gary Wells, a psychologist who studies confidence among eyewitnesses—this despite the fact that witnesses’ confidence, like that of the pundits, is largely uncorrelated with accuracy. In one study, Wells staged a theft and then tested eyewitnesses’ ability to identify the culprit, as well as the persuasiveness of their testimony. It turned out that jurors were more persuaded by witnesses who appeared more confident. Yet witnesses’ confidence was nearly worthless in predicting their accuracy.
In the Harvard forecasting study, the students acted as if they knew the right answers, making predictions that were far too precise. The enormity of the errors impelled Alpert and Raiffa to write to their students in exasperation: “Be honest with yourselves! Admit what you don’t know!”
Even seasoned professionals, working in their domains of expertise, fall victim to overconfidence in forecasting. Itzhak Ben-David, John Graham, and Campbell Harvey recently asked chief financial officers of American corporations to predict the returns of investing in the S. & P. 500. The researchers asked for ranges that should have included the right answer about eighty per cent of the time. In fact, returns fell inside the range less than one-third of the time.
And yet accurate forecasting depends in large part on admitting what we don’t know—that is, having a well-calibrated lack of confidence. Even the wisest minds and most sophisticated computer models can’t predict with certainty what will happen in the future.
For the past three years, I have been working on the Good Judgment Project with Philip Tetlock and Barbara Mellers. The project is part of a first-of-its-kind forecasting tournament. Together with thousands of volunteer forecasters, we have been predicting geopolitical events for the Intelligence Advanced Research Projects Activity, the research arm of the U.S. intelligence agencies. We have forecast, to pick just a few examples, the probability that Angela Merkel would win reëlection in Germany, the probability that Bashar al-Assad would fall in Syria, and the probability that the Afghan government would resume peace talks with the Taliban.
It works like this: IARPA asks us questions about events relevant to U.S. foreign policy, and we open those questions to our forecasters—a collection of smart, curious, and well-informed laypeople who represent a broad cross-section of educated professionals. They are, on average, about thirty-five years old. More than sixty per cent of them have post-graduate degrees. The forecasters rely on many public sources of information to make their forecasts: newspapers, blogs, press releases, Twitter feeds. They also try to get closer to the source, consulting others and even soliciting help from strangers on Web sites like Reddit. They attempt to gather and digest the available information objectively, without over-reacting to hype. Then they submit their predictions on the Good Judgment Project Web site. After we learn the outcome of a given event, we can go back and score people’s forecasts. We reward good forecasts with better scores and post the names of the best forecasters on an online leaderboard. Our forecasters have an impressive record, but that’s not because they are world experts on the topics they are forecasting or because they have access to a trove of complicated data. Instead, they benefit from a little bit of online training—in which we discuss, among other things, acknowledging uncertainty and avoiding overconfidence.
We asked our forecasters whether Dmitry Medvedev would be inaugurated as President of Russia after the 2012 election. When the question opened in September, 2011, our forecasters gave Medvedev a decent chance—about thirty per cent, on average. But after a couple of events at which Medvedev signalled deference to Putin, the consensus forecast dropped close to zero and stayed there. The forecasting began with a great deal of uncertainty, but updated swiftly in the face of useful information.
Something else happened when we asked who would succeed Benedict XVI as Pope. There were four leading candidates, but our forecasters put their collective probability of becoming Pope below fifty per cent, on average, right up until the question closed in March of 2013. With little useful information leaking from the papal conclave in Rome, our forecasters faced a great deal of uncertainty, making it difficult to have much confidence in any forecast. The forecasters—unlike many pundits, courtroom witnesses, or corporate executives—admitted it. And that is what made their forecast so good.
Photograph by Daniel Roland/AFP/Getty.
