Policy Sciences 33: 101^106, 2000.

2000 Kluwer Academic Publishers. Printed in the Netherlands. 101

Landmarks of Policy Sciences

This article is the second in our series of Landmarks Studies in the Policy Sciences

Ascher, William, Forecasting: an Appraisal for Policymakers and Planners.

Baltimore: Johns Hopkins University Press, 1978.

Harold Lasswell was the rst to identify William Ascher's 1978 appraisal of
forecasting techniques as a landmark. In the introduction to the volume,
Lasswell said, `Professor Ascher's volume will become a classical contribution
to the eld of forecasting' (p. xi). It has taken the rest of us a bit longer, but
better late than never.
Ascher's volume is a classic primarily because of two ideas that went against
the tide of that era. He thought it useful to apply the Lasswellian concept of
appraisal not just to the outcomes of implemented policy but also to the inputs
of the policy-making process. And, above all, in doing so, he demonstrated that
the key to a successful forecast is to know what you are talking about ^ and to
have refreshed your knowledge quite recently. In Lasswell's more formal words:
`Perhaps the most striking demonstration is that core assumptions are more
important than technique' (p. xi).
For those of us who have spent the interim quarter century using forecasts
for such purposes as guring out where troops should be committed or how
hundreds of millions of dollars should be allocated, it can be a bit of a stretch to
recall why such thoughts were path-breaking. In fact, at the time they were
revolutionary, and were perceived as counter-revolutionary. The academic era
in which Ascher wrote was obsessed with methodology, with technique, with
tools that would make political science and sociology and economics and
policy science into true science. Although nobody ever stated it so explicitly,
the message to those entering the discipline was very clear: the way to
advancement, the way to contribute to the discipline, was to concentrate on
methodology. Ideally, of course, one wanted to be perfect master of both sub-
stance and method, but in an imperfect world where this was unachievable it
was okay to put most of one's eort into method but obsolete to put most of
one's eort into substance. Since this tendency recurs periodically, we still need
Ascher's forceful demonstration of its unfortunate consequences.

The sociology of forecasting

While most of the analysis in Forecasting is devoted to appraising forecast

accuracy, another of Ascher's important contributions, in this book and more
elaborately in a sequel, was to emphasize that forecasting serves a variety of
purposes and that for some of these purposes accuracy is not the most impor-
tant value. (Indeed, in a later article he demonstrated that the decline in the
102

publication of unconditional forecasts has made the criterion of accuracy less

and less applicable.) Forecasts may seek to help policy makers optimize some
particular goal, in which case accuracy is the greatest need. But forecasts may
also be used primarily for attracting attention to particular issues, for setting
a mood, for establishing priorities, for achieving a particular result, or for
enhancing the prestige of the forecaster. An exaggerated forecast might well
capture attention better than a strictly accurate one, and an esoteric method
might well contribute more to the prestige of the forecaster than a simple but
more accurate extrapolation.
Given these diverse potential goals, one might also appraise a forecast in
terms of how much attention the forecast received, how much revenue it
generated for the forecaster's business, how many votes the forecast shifted
from one candidate to another, how explicitly the forecast was used by decision
makers, how well it dramatized alternative outcomes and their causes, how
decisive the forecast was in aecting policy, or whether the forecaster got
tenure.
Ascher makes a considerable contribution to what one might call the
sociology of forecasting as his dry but incisive prose reviews why various
forecasting programs have been less successful or more. Forecasts are often
better utilized in the private sector, where there is a consensus on the goal of
maximizing prot, than in the government, where consensus on goals is rare.
NRPB eorts at forecasting were rejected because opponents associated fore-
casting with planning and planning with totalitarianism (p. 27). The diculties
of the Planning Programming Budgeting System (PPBS) of the Lyndon Johnson
era show that, `The value of technical expertise is often ignored when weightier
political issues preoccupy the participants of the policy-making process' (p. 27).
He demonstrates that natural resource forecasts are more likely to be heeded if
they help policymakers to nd sensible policy responses to scarcities than if
they tie the policymakers' hand by specifying precisely how much money
should be spent to alleviate the scarcities. Conversely, referring to the successes
of the Census Bureau, Ascher concludes that, `The fact that population projec-
tions are usually far removed from the point of decision means that they are
less threatening to policy-makers guarding their own preferences' (p. 30). As if
to perversely highlight Ascher's point, today the U.S. Congress is objecting to
proposed Census sampling techniques precisely because in the meantime the
population projections have moved close to the point of decision; today, unlike
twenty years ago, better sampling will automatically increase the political
weight of minorities and allocate more budget to them. He also highlights how
the deeply awed forecasts of the book Limits to Growth gained inuence and
fame by exploiting an aura of mysterious technique: `[M]odeling can enhance
the promotion of the forecast by giving it the appearance of technical sophisti-
cation' (p. 39).
 103

Forecast accuracy: methods vs. assumptions

But the core of Ascher's analysis focuses on the straightforward technical

issue: what kinds of forecasts are most accurate? He approaches this empiri-
cally, by collecting and appraising all the relevant and available forecasts
since the 1930s in the areas of population, economics, energy, natural resour-
ces, transportation and technology, examining specic forecasts of such trands
as GNP, electricity usage, nuclear power, motor vehicles, aviation, and com-
puter capacity. By casting the net so broadly, Ascher was able to explore, really
for the rst time, the logic and limitations of forecasting across dierent elds.
This enabled him to discover, for example, that all of the trends examined in
the book, including technological and natural-resource trends, are heavily
dependent upon socio-economic factors.
Ascher developed one important methodological innovation for evaluating
the accuracy of forecasts for which the outcome still lay in the future. He shows
that dispersion of forecasts reects uncertainty and minimum error.
In one area after another he nds that choice of sophisticated methods
(or `forecasting technique') makes relatively little dierence and that paying
inadequate attention to core assumptions makes a huge dierence. For in-
stance, in forecasting population, `The inaccuracy of population projections
... contradicts the widespread impression that newer methods are necessarily
more successful methods. Instead, the projections of the early curve-tters ...
were generally more accurate than were the forecasts derived by the more
sophisticated component and cohort methods used later...' (pp. 49^50).
Population forecasts went very wrong for many because they embodied erro-
neous assumptions that population growth must follow a logistic curve and
that fecundity and morbidity would necessarily be constant over time for each
cohort (pp. 41^42 and 45).
Building on this example, Ascher calls attention to the fatal importance of
assumption drag , the persistence of erroneous assumptions after their validity
has been contradicted by the data. Assumption drag occurs for several reasons.
One is the herd tendency of socially validated opinion to form a persistent
consensus on erroneous assumptions ^ for instance that population growth
must be topping out in the 1930s or that automobile and aviation growth just
could not conceivably sustain the torrid growth pace seen three decades later.
Such faddish consensus makes it easier to dismiss contradictory evidence as
quirks or blips in the data. Second, forecasts are interconnected, and in their
focus on their own core area of expertise, forecasters frequently rely on obsolete
assumptions derived from related disciplines. Third, the complexity of sophis-
ticated methods may mean that the results they obtain are obsolete by the time
all the data have been collected and processed, and the high cost of sophisti-
cated methods may mean that forecasts are repeated too infrequently to keep
up with the change in underlying assumptions and structures. (Another crucial
conclusion from Ascher's data is that the accuracy of a forecast depends
heavily on how recently the forecast was done.)
104

When he appraises decades of eorts to forecast GNP, Ascher demonstrates

that short-term forecasts which rely primarily on expert judgment have achieved
the most accurate results, with econometric models a close second and leading
indicators a distant third. Earlier econometric models did better than judg-
ment; later, more sophisticated ones have done worse. More elaborate recent
econometric models did not forecast better than older, simpler models, either
for quarterly or for annual GNP. Judgment-free models did worse than those
combined with judgment. Similarly, viewing longer-range forecasts, despite the
development of ever more sophisticated methods, `The forecasts show no ten-
dency toward greater accuracy over time.'
Ascher's ndings regarding leading indicators could, if they were more
generally known, save contemporary bankers tens of millions of dollars: `The
lack of improvement in leading indicators forecasts supports the position that
the method's inaccuracies are not due to a failure to develop the method's
potential, but rather stem from inherent limitations in the assumptions and
conditions required for the validity of the approach. If the crucial assumption
of the regularity and uniformity of economic sequences is correct, it is surpris-
ing that more than fty years of searching for those sequences has not produced
results on a par with unstructured judgment' (p. 84).
Because of such ndings, Ascher's book has been a mainstay in policy
scientists' arguments that bounded rationality and uncertainty will always have
to be dealt with, since it showed that presumed improvements in forecast
accuracy due to methodological `improvements' were not actually occurring.
Having established the general superiority of judgmental techniques for
forecasting GNP, the study nonetheless qualies that conclusion in an intrigu-
ing way: `Almost invariably, the long-range judgmental projections of GNP
reect a growth rate that exceeds the historical rate. One reason for this is
forecasters' focus on the potential of the economy' (p. 92). This is a specic
demonstration of a general principle: policy analysts who can see straightfor-
ward rational ways to improve a policy outcome usually assume there will be a
tendency to move toward such policies and improve the result. But frequently
politics or bureaucratic suboptimization moves the outcome away from the
optimum.
Ascher's ndings in other areas of forecasting are similar. Despite the evolu-
tion of more sophisticated forecasting methods, `There is no evidence that ve-,
ten-, or fteen-year electricity forecasts have been improving over time' (p. 102).
`Methodology is of secondary importance for the accuracy of energy forecasts'
(p. 119). Older, simpler methods of general aviation forecasting were just as
accurate as sophisticated modern techniques (pp. 152^153). The key to im-
provement of aircraft transport forecasting was not method so much as the
substantive realization that personal aircraft would play a rather small role
for the foreseeable future (p. 152). For commercial aviation, simple trend
extrapolation would have given accurate forecasts, but forecasters could not
bring themselves to believe in the combination of much greater total travel and
a shift to a higher proportion of airline travel' (p. 156). Similarly, motor vehicle
 105

growth forecasts were consistently too conservative, regardless of method,

because `forecasters were unwilling to believe the real data that indicated
tremendous expansion in consumption and economic growth, and were ex-
pecting the bubble to burst' (pp. 159^160).
Likewise, for nuclear power, regional component forecasts were not more
accurate than less rened techniques because all techniques were swamped by
substantive assumptions about safety, cost and political opposition. And for
computer capacity, `The common characteristic of the highly inaccurate fore-
casts is not methodology as such, but rather the fact that they were not
formulated by computer experts' (p. 184). One variation of `technique' did
improve forecasts, however: forecasts that tapped the judgment of multiple
experts proved more accurate than those using only one.

Forecast accuracy: institutional bias

Perhaps the most heartening conclusion of Ascher's study was that, although
the institutional base of the forecaster may sometimes create systematic bias,
the empirical appraisal of forecast results reveals far less of this than conven-
tional wisdom assumes. Government forecasts of energy use and aviation
growth showed none of the expected tendency to underestimate growth, and
private sector forecasts showed none of the expected systematic bias for over-
estimating growth (pp. 111^112, 117, 154). Sociologists' forecasts of automobile
use demonstrated none of the expected tendency to exaggerate the growth and
impact of automobile use (p. 160). The institutional base of the forecaster made
little dierence in the accuracy of nuclear energy forecasts, although one in-
stitution, the Atomic Energy Commission, seemed to have a better record. (As
some of us in business and politics have discovered, it can be helpful to forecast
accuracy if one is in charge of managing the outcome being predicted.) Across
institutions, scientists were more frequently over-optimistic than non-scientists
(p. 176).
Instead, the worst sources of bias turned out to be the herd instincts of the
day: shared false assumptions that cut across all institutional bases and biased
everyone's forecasts in the same way, for instance leading to the almost universal
underestimation of population growth before World War II and of automobile
and aviation growth after World War II.

The uses of complex models

Having successfully indicted complex models for wasting time and money
without delivering better forecasts, in part because they are disproportionately
vulnerable to assumption drag, Ascher highlights their value in providing con-
sistency checks, in analyzing the dierent implications of alternative policy
options, and in working backward from desired outcomes. He might have
106

added that the very process of building models frequently sensitizes an expert
to the scale of interconnections and tradeos, thereby facilitating more rened
expert judgments.
Ascher's ndings have been largely vindicated by subsequent forecasting
practice. Today most of the world's largest and most sophisticated businesses
do not rely on huge and hugely sophisticated forecasting models. The forecasts
of greatest importance to decisions are made by small groups with great expe-
rience and expertise who have recently and directly reviewed the prevailing
trends and assumptions. Many do subscribe to big models, which they use for
forecasts about areas outside their expertise, for validating some of their own
preconceptions, for consistency checks, for sensitivity analyses, for alternative
views, and the like. But, parallel to Ascher's nding that ever-more-complex
techniques have failed to generate more accurate forecasts, there has been no
tendency for executives to defer to ever-more-complex and ever-more-compu-
terized forecasting processes. Computerization has proved enormously helpful
when used to accumulate vastly more data, to organize it, to communicate it,
and to process it for use by experts. But when Citibank or Goldman Sachs or
General Motors wants to make a decision on whether to invest in Thailand,
they do not roll out the gigantic forecasting model of the Thai economy. They
commission one or more trusted, experienced experts to produce a forecast,
and those experts usually rely on simple models with a few variables, together
with very complex qualitative thoughts about the reasons for presenting the
variables in the ways they do.
On the other hand, there is a vast global industry, much larger than when
Ascher wrote, which spends hundreds of millions of dollars on leading indica-
tors that pretty consistently fail to predict the next turn in the interest rate cycle
^ or predict a dozen turns for every one that occurs. Each major bank employs
a sta to forecast these turns, and it is no longer unusual for the economist
leading the group to earn one million dollars or more annually. In fact, the
expenditure of a single major institution today can be the same order of
magnitude as Ascher's rough estimates for the total national expenditure on
such forecasts a generation ago. One look at the nancial catastrophes of 1994,
from the bankruptcy of Orange County, California, to the derivatives crisis of
that year, to the Mexican peso collapse will both validate Ascher's skepticism
about leading indicator forecasts and demonstrate that much work remains to
be done in his footsteps.

Reviewed by William H. Overholt.