How to be a better forecaster

25 June 2016, 20:35
Sherif Hasan

I've posted before on the work of Philip Tetlock, but I can't get enough of him!

(bolding below is mine)
Tetlock assembled a group of more than 2,000 non-expert volunteers under the banner of the Good Judgement Project
  • Working in teams and individually the volunteers were asked to forecast the likelihood of various events of the sort intelligence analysts try to predict every day ("Will Saudi Arabia agree to OPEC production cuts by the end of 2014" was one example)
  • The Good Judgement Project's predictions were 60% more accurate in the first year one than those of a control group of intelligence analysts
Tetlock found that the most accurate forecasts were made by 2% of his volunteers ... it was the way these people made decisions and learned, rather than deep subject knowledge, which gave them the edge over specialist intelligence analysts
In their thought process the superforecasters were almost exclusively what the philosopher Isaiah Berlin described in a famous 1953 essay as "foxes"
  • Foxes have a wide and shallow perspective, an approach which contrasts with what Berlin described as "hedgehogs" who have depth and expertise in a narrow field
  • To Tetlock hedgehogs are confident in their deep knowledge and are often guided by one or two theories (Keynesianism, post-liberalism, communism and so on).
  • Foxes are sceptical about such theories, open-minded, cautious in their forecasts and quick to adjust their ideas as events change.
  • Rather than rely on one or two simplifying ideas to explain events, Tetlock's superforecaster foxes embraced complexity and were comfortable with a sense of doubt.
Another difference is that the best forecasters continuously seek ways to make better predictions
  • They display a mix of determination, self-reflection and willingness to learn from one's mistakes
  • To improve the quality of predictions it is vital to keep a honest tally of successes and failures
  • Bad forecasters ignore their mistakes; good ones acknowledge and learn from them
Tetlock improved accuracy by having the forecasters work in teams
  • results showed that teams were on average 23 percent more accurate than individuals
  • Sharing and debating ideas can significantly improve forecast scores
  • But some teams did poorly; free riding by individual members was one problem; another was a tendency to groupthink which weakened the very scepticism and openness that Tetlock finds essential to good forecasting

Tetlock found that people can be trained to be better forecasters
  • For individuals, the training focused on thinking in terms of probabilities and removing thinking biases - for instance, focussing on the limitations of one's own knowledge and being open to alternative views
  • For groups the training aimed to strike a balance between conflict and harmony. Too much conflict destroys the cooperation that is essential to teamwork. Too much consensus leads to groupthink
Share it with friends: