How to Evaluate a Forecast: A Field Guide for Intelligence and Strategy Teams

Every week, someone puts a forecast in front of you. A consultant says a market will triple by 2030. An industry association says a technology will be standard within five years. A cable-news guest says a war will end by spring. Some of these forecasts come from people who have studied the question for decades, and some come from people who needed something quotable by deadline. They all look the same on the page. Evaluating them is one of the core skills of competitive intelligence and strategic foresight, and it can be learned.

First, make sure it is a forecast

In Future, Inc. (AMACOM, 2007), I gave a working definition: “A forecast is a statement predicting what will happen in the future by a certain date.” Both halves matter. A statement with no date (“AI will transform health care”) cannot be checked, so it cannot be wrong, so it tells you very little. A date with no specific claim (“2030 will be a turning point”) is the same problem in reverse. A real forecast names an outcome and a time, so that someday you can hold it up against what happened.

The book was blunt about the alternative: “Forecasting a date that is not based on knowledge is called ‘guessing.’”

A test: which of these would you trust?

In the book I offered readers three forecasts about Europe’s energy system:

1. By 2015, you will be able to buy either gasoline or hydrogen at most car-refueling stations in Europe.
2. Wind power will compose 10 percent of Europe’s energy consumption by 2020.
3. The cost of biofuels will fall sharply around 2007 and will then account for 20 percent of European fuel needs.

Two came from a 2003 futures report by Denmark’s Risø National Laboratory, whose researchers had spent months studying Europe’s energy infrastructure. One I made up because it sounded plausible. The hydrogen forecast was mine. The point of the exercise was that you cannot tell expertise from guesswork by reading the sentence. You have to look at the source.

Two decades later, the exercise has a second lesson. The hydrogen guess was wrong. But so was the expert biofuel forecast: in 2020, renewable energy of every kind, biofuels included, supplied only 10.2 percent of the energy used in EU transport, according to Eurostat (Eurostat). Expertise improves your odds. It does not make a forecast true. That is why the rest of this guide matters.

How to evaluate a forecast: seven questions to ask

1. Who made it, and what do they know? The best forecaster on a question is someone who has spent a career on it: a petroleum geologist on oil reserves, a chip designer on processors. Be most suspicious of experts speaking outside their field. In the kitchen-of-the-future study I describe in the book, home builders were the right source on the size of tomorrow’s kitchens and the wrong source on tomorrow’s kitchen electronics, so we went to the semiconductor and consumer electronics people for that.
2. Are you reading the original or a summary? Most forecasts reach you through a journalist, a newsletter or a slide in someone else’s deck. Go back to the report itself. You will often find a range where the headline gave a number, and conditions where the headline gave a certainty. The first of the twenty-five worst practices in my satirical second book, How to Predict the Future…and WIN! (2011), is “Listen to major media exclusively.”
3. Is it specific enough to be wrong? In the book I contrasted a vague expert statement that genomics would someday matter to brewing with the kind of forecast that would get my attention, one with a named outcome, a date and a price. Faced with that, I wrote, “my ears would perk up.” The vague version is safe for the forecaster and useless for you.
4. Does it say how likely it is? Good forecasters attach odds, or at least words with agreed meanings. Sherman Kent, who led the CIA’s Board of National Estimates, learned this the hard way when a 1951 National Intelligence Estimate called a Soviet attack on Yugoslavia a “serious possibility.” When he asked the members of his own board what odds they had meant, their answers ran from about 20 percent to 80 percent (Kent, “Words of Estimative Probability”). U.S. intelligence analysts are now required to express uncertainty with defined probability terms (Intelligence Community Directive 203). Hold commercial forecasters to the same standard.
5. Does it line up with the trends? The checklist at the end of the forecasting chapter in Future, Inc. says it plainly: “Check up on the experts!” A forecast that contradicts the trend data is not automatically wrong, but it needs an explanation of what will change. If there is none, be skeptical.
6. What does the forecaster gain if you believe it? Vendors forecast growth in the markets they sell into. Associations forecast that their members’ products will be indispensable. Neither makes the forecast wrong, but both are reasons to look for a second source without the same incentive.
7. Is the confidence earned, or performed? Tip 8 of my satirical book is “Value the probability of forecasts by the charisma of the person delivering them.” The serious note underneath it warns against “whichever charisma crutch supports the credibility of your opinions.” Polish is not evidence. Ask for the track record instead.

One forecast is good; several are better

No single expert sees the whole future of anything complicated. In the book I described a study for a home-appliance client on the kitchen of the future. We combined forecasts from the National Association of Home Builders, semiconductor makers, internet architects and the Consumer Electronics Association, because the future kitchen depended on all of them. As I wrote then, “These forecasts were not available from any one expert. We had to combine the forecasts of many experts to derive a more complete view of the future.”

Research on forecasting accuracy backs this up. In a forecasting tournament run by the U.S. intelligence community’s research agency, IARPA, starting in 2011, the Good Judgment Project led by Philip Tetlock and Barbara Mellers combined the judgments of many carefully selected and trained forecasters. According to Good Judgment, its forecasts “even outperformed intelligence analysts with access to classified data” (Good Judgment).

Keep score

The last step is the one almost everyone skips: write down the forecasts you relied on, and check them when their dates arrive. You will learn which sources to trust, which kinds of questions your team handles well, and where your blind spots are. I did this for my own 2011 book, and the pattern was humbling: my forecasts about people and institutions held up, and my forecasts about resources and prices did not. You can read the results in the fifteen-year scorecard.

A one-page checklist

• Does it name an outcome and a date?
• Is the source a real expert in this specific question?
• Have you read the original, not the summary?
• Is it specific enough to be proved wrong?
• Does it state its odds, in numbers or defined words?
• Does it fit the trend data, or explain why not?
• Who benefits if you believe it?
• Have you found at least one independent forecast to compare it with?
• Have you written it down so you can check it later?

Frequently asked questions

Related reading

• I Wrote a Book About Predicting the Future in 2011. Here’s My Scorecard.
• Strategic Foresight: A Practitioner’s Guide to Seeing What’s Coming
• Cognitive Bias: A Field Guide for Leaders

Competitive Futures evaluates forecasts and builds scenarios for companies and governments; details are on the services page.