yneasurement of

output of research
and experimental
development :
a review paper.

by Christopher Treeman

Director, Science Policy Research Unit

University of Sussex

United Kingdom

1970
unesco
 I ’

Printed in 1969 in the Workshops of the

United Nations Educational, Scientific and Cultural Organization,
Place de Fontenoy, Paris 7e, France

s-r/s/ 16
COM.69/XVI- 16 A
Printed in France

0 Unesco 1970

.
PREVIOUS TITLES IN THIS SERIES

No. 1 - Film and cinema statistics.

(with resumk in French and Spanish)
0 ut of print.
No. 2 - Book production 1937-1954 and translations 1950-1954.
(Paris, 1958)
Production de livres 1937-1954 et traductions 1950-1954.
(Paris, 1958)
No. 3 - Statistics on libraries. (Paris, 1959)
Statistiques SW les biblioth&ques. (Paris, 1959)
No. 4 - Statistics of newspapers and other periodicals. (Paris, 1959)
Statistiques sut les journaux et autres gtiodiques.
(Paris, 1959)
No. 5 - Statistics on special education. (Paris, 1960)
Rapport statistique sur I’enseignement spicial. (Paris, 1960)
No. 6 - Requirements and resources of scientific and technical
personnel in ten Asian countries- (Paris. 1960)
Besoins et ressources de dix pays d’Asie en personnel
scientifique et technique. (Paris, 1961)
No. 7 - Pre-school education. (Paris, 1963)
L’iducation prescolaire. (Paris, 1963)
No. 8 -Statistics on radio and television 1950-1960. (Paris, 1963)
Statistiques de la radiodiffusion et de la tClivision 1950-
1960. (Paris, 1963)
No. 9 - Methods of estimating the demand for specialists and of
planning specialized training within the USSR. (Paris, 1964)
M.Gthodes d’Pvaluation des besoins en sp&zialistes et de
planification de la formation spicialisCe en URSS.
(Paris, 1964)
No. 10 - Estimating future school enrolment in developing countries,
a manual if methodology. (Paris, 1966)
Estimation des effectifs scolaires futurs dans les pays en
voie de diveloppement - manuel de mCthodologie.
(Paris, 1967)
No. 1 - Methods of analysing educational outlay. (Paris. 1966)
Methodes d’analyse des depenses d’enseignement.
(Paris, 1967)
No. 2 - Methods of long-term projection of requirements for and
supply of qualified manpower. (Paris, 1967)
MCthodes de projection B long terme de l’offre et de la
demande de maiwd’ceuvre qualifiCe. (Paris, 1968)
No. 1 3 - Statistics of educ‘ation in developing countries. An intro-
duction to their collection and presentation. (Paris, 1968)
Les statistiques de I’iducation dans les pays en voie de
d.+veloDDement - Comment les rassembler et les prisenter.
(Paris,.i968)
No. 14 - International developments of education expenditure
. ^_^
1Y5U-
1965. (Paris, 1969)
L’Cvolution internationale des dPpenses d’iducation entre
1950-1965. (Paris, 1969)
No. 15 - The measurement of scientific and technological activities.
(Paris, 1969)
La mesure des activitCs scientifiques et techniques.
(Paris, 1969)

In the press
No. 17 - World Summary of Statistics on Science and Technology/
Statistiques de la science et de la technologie : Apetgu
mondial. (Bilingual publication/ Cdition bilingue)
(Paris, 1969)
 PREFACE

This study on the Measurement of output of research ,of the innovation process anil its impact on the
and experimental development was originally pre- economy and to allow for amorertitional allocation
pared as a working document for the first meeting of limited resources to compet;ingRsnBD activities.
of the Unesco/Economic Commission for Europe However, many theoretic& and :practical problems
Working Group on Statistics of Science and Tech- involved in the compilation of such statistics remain
nology which met in Geneva in June 1969. The to be solved and it cannot be expected that solutions
favourable reception given to this paper, prepared will be reached in the near future in view of the
by Mr. C. Freeman, Director of the Science Policy many difficulties still existing in the far more ad-
Research Unit of the University of Sussex, United vanced area of statistics of R and D inputs. Although
Kingdom, has led Unesco to make it available to a no immediate solution may be found, a review of
wider public by publishing it in the series Statistical the results achieved so far in measuring the output
Reports and Studies. of R and D can nevertheless contribute to a better
In the same series Unesco previously presented understanding of the problem and stimulate further
a document on The measurement of scientific and discussion.
technological activities (Unesco, Statistical Reports The ideas expressed in the present paper are
and Studies, ST/S/15, Paris 1969)bythe same author, those of the author and do not necessarily represent
designed to add to the literature on the classifica- the views of Unesco.
tion and measurement of the inputs of human and The designations employed and the presentation
financial resources into scientific and technological of the material in this publication do not imply the
activities. The present study, a complement to the expressionof any opinion whatsoever on the part of
first, reviews the far more complicated subject of the Unesco Secreiariat concerning the legal status
the measurement of the output of research and ex- of any country or territory, or of its authorities, or
perimental development (R and D) . c.oncerning the delimitations of the frontiers of any
Statistical measures of the output of R and D country or territory.
are needed in order to reach a fullerunderstanding
 CONTENTS

Page

Chapter I INTRODUCTION . . . . . . . . . . . . . . 7
The growth of research activities . . . . . - . . . 7
Professionalization and specialization of scientific research 7
The measurement of inputs into research and experimental
development activities . . . . . . . . . . . . 8
Input measurement and output measurement . . . . . 8

Chapter II SOME PROBLEMS OF OUTPUT MEASUREMENT . . . . 10

Theoretical objections to any scheme of output measurement 10
R and D output as a flow of information and innovation . . 10
“Accidental” factors affecting R and D output . _ . . . 11
Use of input measures as a surrogate for output . . . . 12

Chapter III THE USE OF SCIENTIFIC PAPERS IN MEASUREMENT

OF RESEARCH OUTPUT . . . . . . . . . . . 14
Numbers of papers and other output measures . . . . _ 14
The use of output measures in the sociology of science . . 15
The need to relate “output” to “input” measures . . . . 16
The need to relate output measures to institutional factors
and social changes . . . . . . . . . . . . . 16

Chapter IV THE USE OF PATENT STATISTICS FOR MEASUREmNT . 20

Patents, inventive activity and “applied” research . . . 20
Limitations of patent statistics . . . . . . . . . 21
Applications of patent statistics . . . . . . . . . 22
“Science” and “technology” in relation to papers and patents 23

Chapter V INNOVATIONS AND THE R AND D SYSTEM . . . . . 25

Invention and innovation . . * . - .- . - . - . 25
C&t-benefit analysis of innovkions . ‘.; . . . . . . 25
Indirect measures of the output of innovations . . . _ . 27
Output budgeting . . . . . . . . . . . . . . 28

Chapter VI SUMMARY AND CONCLUSIONS . . . . . . . . . 31

APPENDIX

Tables
1. The flow of ideas through the stages of research, invention
and development to application . . . . . . . . . . . . . . . . . 34

2. Schematic table showing numbers of authors of various degrees

of productivity (in papers per lifetime) and numbers of papers
so produced . . . . . . . . . . . . . . . . . . . . 35
 Page

Tables
3. Origin of science abstracts . . . . . . . . . . . . . . . . 36

4. Percentage of women on the professional staffs of higher

educational institutions in 1947 compared with percentage
of articles in professional journals contributed by women
in1940 and 1960 . . . ~ . . . . . . . . . . . . . . . 37

5. Distribution of research publications and their origin . . . . . . . - 38

6. Institutions from which more than 100 papers were published

during the year under analysis . . . . . . . . . . . . . . . 39

7. Number of discoveries in the medical sciences by nations,

1800-1926 . . . . . . . . . . . . . . . . . . . . . 39

a. Patents granted to foreigners as a percentage of total patents,

195’7-1961 . . . . . . . . . . . . . , . . . . . . . 40

9. Approved enterprise-to-enterprise agreements for the

transfer of technology in Japan . . . . . . . . . . . . . . 41

10. Outline of a proposed output budget covering Home Office

responsibilities . . . . . . . . , . . . . . . . . . . 42

-. - -- _--- ___... -_~.._--..

 Chapter I

INTRODUCTION

THE GROWTH OF RESEARCH ACTIVITIES PROFESSIONALIZATION AND SPECIALIZATION

OF SCIENTIFIC RESEARCH
One of the outstanding features of the Twentieth
century has been the rapid growth of scientific and The larger scale of scientific research was asso-
technological research and a range of associated ciated with its increased professionalization. A
activities. The growth of these activities and their high proportion of scientific and inventive work in
impact on economic, military and social policy has the Eighteenth and Nineteenth centuries was con-
been variously described as the “Research revolu- ducted on a part-time or amateur basis. This
tion”, the “Scientific revolution” or the “Techno- proportion is, of course, still significant today,
logical revolution”. One result of this “revolution”, particularly with respect to inventive activity, but
however it may be defined or described, has been more characteristic of the second half of the Twen-
a demand for statistical measurement of the re- tieth century is the full-time professional research
sources devoted to the generation of new science scientist or engineer and the specialized research
and technology and of the efficiency with which they institute. It is sometimes forgotten that, even in
are being used. the Nineteenth century, the elite scientific organi-
It may well be true, as Derek J. de Solla Price zations, such as the Royal Society, found it diffi-
has suggested, (1) that scientific activities were cult to accept the idea of professional status for
growing very rapidly already in the Eighteenth and scientists. Ben-David has pointed out that: “Aca-
Nineteenth centuries. In this perspective, the demic appointments . . . were regarded as honours
Twentieth century growth may be regarded simply rather than careers and turning science into an
as the continuation of a long-term trend. But the occupation would have seemed something like a
absolute scale of the resources committed before sacrilege. “(3) The very word “scientist” only
1900 was so small, even in the United States, that came into general use in the middle of the century.
it amounted to much less than 0.1% of GNP. Today Even thenthe main awards to scientists were seldom
almost all European countries devote resources to on the professional basis of full costs, includin
research and experimental development which are instrumentation, overheads and supporting staff. ( 47
the equivalent of between 1% and 3% of GNP, (3) It was only towards the end of the century that a
while many developing countries are already spend- few industrial firms began to set up small profes-
ing the equivalent of 0. 1% or more of their GNP. sional research laboratories on a permanent footing.
In these circumstances it was inevitable that Thus, Whitehead was justified in describing the
there should be increasing concern with the effici- greatest invention of the Nineteenth century as the
ency of the research-innovation system. In some “method of invention itself”, (5) in the sense of a
sense, this implies measurement of inputs and out- network of full-time research organizations.
puts of the process. As long as governments or Although the professionalization of science
enterprises were spending only very small sums on proceeded rapidly in the industrialized countries
scientific research, they could afford to regard this in the early part of the Twentieth century, it was
outlay in a very similar way to patronage of the not until the 1930’s that the number of patents taken
arts, using “prestige” criteria rather than attempt- out by corporations in the United States exceeded
ing to assess “efficiency”. But it is one thing to those taken out by individuals, (6) and it was not
endow an occasional eminent scientist; it is quite until 1953 that the first official government survey
another to maintain laboratories regularly employ- was made of the total resources employed in pro-
ing thousands of scientists and technicians on a fessional research and experimental development
continuous basis. The increased scale of scientific in the country.
activities led inexorably to an increased concern It would indeed have been difficult to survey
with their effectiveness. and measure scientific research activities before

7
they reached this fairly advanced level of speciali- systematic comparison - the first international
zation and professionalization. Once it had been statistical year for research and development. (13)
reached, however, it became possible toundertake Within the United Nations, Unesco has taken the
reasonably accurate surveys of the numbers of pro- lead in encouraging and systematizing the measure-
fessional research and experimental development ment of R and D inputs. (14) It has recently initiated
scientists and engineers in industry and in govern- attempts to reconcile definitions and concepts as
ment and of the expenditures necessary to finance between East and West European countries. (15)
their activities. But, even now, one of the biggest As with other similar statistical series, institu-
difficulties in surveys of research and experimental tional differences between socialist and capitalist
development inputs is the problem of part-time re- economies make this especially difficult and pro-
search workers, and no country has satisfactorily gress so far has been small. Unesco has also
resolved this question with respect to university ’ stimulated measurement of scientific and techno-
research. logical services in many of the developing countries,
but here too there are major problems of scope and
comparability of national statistics.
THE MEASUREMENT OF INPUTS INTO Nevertheless, it is not unreasonable to suppose
RESEARCH AND EXPERIMENTAL DEVELOP- that as an increasing number of countries gain ex-
MENT ACTIVITIES perience of regular statistics of Rand D manpower
and expenditure, and as international organizations
The first official government statistics were those become more familiar with the problems of compari-
published by the Soviet Union since the early son, it will be possible to use a fairly wide range
1930’s. (7) These related, however, to a range of of moderately accurate and comparable statistics
scientific and technological activities somewhat of R and D inputs. Persistent efforts will be neces-
wider than those now commonly defined as “research sary to improve their accuracy and range.
and experimental development”. @) Most other
socialist countries now also publish annual statistics
INPUT MEASUREMENT AND
of scientific services, although without compara-
OUTPUT MEASUREMENT
bility in their coverage and definitions.
The first experimental attempt in a market The position with regard to measurement of R and
economy to measure R and D inputs in all sectors D outputs is completely different. There is no
(industry, government and universities) was made nationally agreed system of output measurement,
by J. D. Bernal in 1938-1939. (g) But Bernalhad to still less any international system. Nor does it
make use of very poor data for industry and an un- seem likely that there will be any such system for
satisfactory breakdown of government expenditure. some time to come. At the most, it maybe hoped
Industrial organizations had begun to publish figures that more systematic statistics might become
of expenditure on industrial R and D in the United possible in a decade or two.
States and in the United Kingdom in the 1930’s, (16) This paper, therefore, is not concerned to
but these were incomplete in their coverage and in- make proposals for international systems of R and
consistent in their definitions. It was not until the D output measurement. It has the much more
1950’s that the National Science Foundation (NSF) limited objective of reviewing briefly some of the
in the United States resolved these problems by sys- experimental attempts at output measurement which
tematic comprehensive surveys in industrial and have been made, of selecting those which appear to
government organizations on consistent definitions. offer the hest future prospects, and of indicating
Since the NSF began their regular annual sur- ways in which international organizations might
veys of R and D expenditures hnd manpower in 1953, stimulate further experimental developments to
many other OECD countries have followed suit. the point where regular national and international
Unfortunately, they often did so on the basis of series become feasible.
varying national definitions and concepts, so that First it will be necessary, in Chapter II of this
international comparability was difficult to attain. paper, to deal with some theoretical objections to
The Directorate for Scientific Affairs of OECD took the whole idea of output measurement and with the
the initiative in attempting to standardize definitions present treatment of R and D inputs in systems of
and systems of measurement. The first Frascati national accounts. The next three sections review
Conference in 1963 agreed on a standard system of various attempts to develop systematic output
measurement, (11) and as a result of this OECD measures for part of the R and D spectrum of
was able to undertake an experimental international activities, and in the final section suggestions are
comparison of a few countries(l3) and later a more made for further experimental work.

8
 NOTES

(1) Derek J. de Solla Price, Little (9) J. D. Bernal, The Social Function of Science,
Science, Columbia University Press, 1963 Routledge, 1939
(2) OECD, International Statistical Year for Re- (10) N. E. Terleckj, Research and Development:
search and Development, Vols. 1 and 2, Paris its Growth and Composition, National Industrial
1967-1968
Conference Board, (NICB), 1963
(3) J. Ben-David, “Scientific Productivity and Federation of British Industries, Surveys of In-
Academic Organization in Nineteenth Century dustrial Research
Medicine”, American Sociological Review, Decem-
ill) OECD, Proposed Standard Practice for Sur-
ber 1960, page 836
veys of Research and Development: The Measure-
(4) R. MacLeod, The Institutionalization of Basic ment of Scientific and Technical Activities,
Research: the Government Grant Committee ofthe DASIPDI62.47
Royal Society, 1850-1914, Science Policy Research
Unit, University of Sussex, to be published. (12) C. Freeman and A. Young, The Research and
(5) A. N. Whitehead. Science and the Modern Development Effort in Western Europe, North
World, Pelican, 1937, page 120 America and the Soviet Union, OECD, Paris, 1965

(6) Jacob Schmookler, Invention and Economic (13) OECD, op. cit. (2)
Growth, Harvard University Press, 1966, page 26 (14) Unesco, Provisional Guide to the Collection
(7) J-M Collette. “Recherche-developpement en of Science Statistics, COM/MD/3, Paris, 1968
URSS”, Cahiers de lIZSEA, Institute de Science Unesco, op. cit. (8)
Bconomique appliquge, August 196 2
(15) Unesco, op. cit. (8)
(8) Unesco, The Measurement of Scientific and idem, Questionnaire on statistics of research and
Technological Activities, 1969 experimental development effort, 1967 ( Unesco,
STS/Q/SSl), Paris, July 1968
 Chapter II

SOME PROBLEMS OF OUTPUT MEASUREMENT

THEORETICAL OBJECTIONS TO ANY impossible to define and measure inventive output

SCHEME OF OUTPUT MEASUREMENT raramente (or scientific output). It may be conceded that this
is extremely difficult, but it cannot be denied that
The need for output measurement is seldom disputed there is an output of some kind from all types of
by those actively engaged in the management of re- research and experimental development activity.
search and experimental development, whether in An attempted representation of these outputs and
government, industry or universities. But, how- the corresponding inputs for various stages of re-
ever desirable such measurement may appear to search and experimental development work is shown
policy-makers, it is sometimes maintained that schematically in Table 1. (The differences between
output measurements are either unattainable or this scheme and the usual “input” classification of
useless. R and D are discussed in Chapter IV.) From this
It may be that the satisfactory measurement of it can be seen that the output of all stages of R and
part or all of R and D output will prove unattainable D activity is a flow of information and the final out-
on purely practical grounds. This can only be estab- put of the whole system is “innovations” - new pro-
lished by attempting the measurements with skill, ducts, processes and systems. This information
determination and ample resources over a consider- is conveyed in various forms and through various
able period of time. The measurement of Gross media, with varying degrees of secrecy or freedom.
National Product or of R and D inputs at one time Some of it is “intermediate” or “feedback” output.
also appeared extremely difficult on purely practical But there can be no doubt that such a flow of infor-
grounds. However, there would be no point in making mation exists and that it is valuable.
even the attempt to measure R and D output if it
could be clearly demonstrated that the objective was
in principle unattainable or unnecessary. R AND D OUTPUT AS A FLOW OF
Two such arguments are therefore briefly con- INFORMATION AND INNOVATION
sidered here, as summarized by Machlup:(lG)
The problem is therefore one of defining and measur-
“One might take the position and defend it on
ing the flow of certain types of information and the
good grounds that it is impossible even to define
efficiency with which this flow is utilized by various
invention, let alone to identify, count and weight
organizations to make innovations. The main cri-
inventions, and if it is meaningless to quantify
terion to distinguish this flow from all otherinfor-
the output it must be meaningless to assert or
mation flows is the criterion of novelty. It maybe
posit the existence of a quantitative relationship
readily admitted that this is not an easy criterion,
between input and output ... ”
either in definitions or in practice. But it is not an
impossible one. It is a criterion which, although
“Or one might take a less negative position and
difficult to apply, is constantly used. It has for
grant the possibility of quantifying input and out-
centuries been the foundation of patent law in many
put at least roughly or for the purpose of con-
countries and it is a criterion commonly applied
structional reasoning but at the same time hold
for scientific publication. Whilst it is true that
that the incidence of accidents in making inven-
individual cases may be disputed, it would be hard
tions is too great to legitimize even provision-
to deny that there is an essential difference between
ally the assumption of a production function. ”
repeating information which is already known and
Although “invention” is discussed here, essentially imparting something new, and that there are new
similar arguments may be applied to other types of products and processes. Therefore, the argument
R and D activity. Take first the argument that it is that the whole output of research andexperimental

10

----_--___ -._- --
development is in principle not definable is unaccept- “ACCIDENTAL” FACTORS AFFECTING
able. The problem is reduced to the practical one of RANDDOUTPUT
senaratine: the R and D information flow from other
types of information flow, of trying to measure all We may now briefly c0nside.r the second main line
or part of this flow, and of assessing efficiency in us- of theoretical criticism which, if valid, might be
ing new information generated by R and D activities. sufficient reason to discontinue attempts at output
If we cannot measure all of it because of a measurement. This is the argument that the input/
variety of practical difficulties, this does not mean output relationship is too arbitrary and uncertain
that it may not be useful to measure part of it. The in research and experimental development activity
GNP does not measure the whole of the production to ,justify any attempts to improve efficiency or
activity of any country, largely because of the effectiveness. It rests largely on the view that un-
practical difficulties of measuring certain types of predictable accidents are so characteristic of the
work. The measurements of R and D inputs omit process that rationality in management is impossible
important areas of research and inventive activity. to attain. This argument need not detain us long.
But this does not mean that GNP or R and D input It is evident immediately that the argument
measures are useless. itself assumes some knowledge of both inputs and
Some parts of the information flow are captured outputs in at least part of the range of R and D
and embodied in well-established, accessible forms. activities. Otherwise the la&of relationship could
The best-known examples are published scientific not be presumed. The view is usually largely based
papers and patents. It can scarcely be denied that on some well-publicized anecdotes of supposedly
these do represent a part of the output of research accidental factors in scientific work, such as
and experimental development activity, although it Fleming’s penicillin mould. (17)
may be (as Machlup maintains) (16) that they do not The straightforward answer to this type of
represent the most important part of the output of argument was given by Cottrell when he said:(18)
fundamental research or of inventive work, or that
“If . . . you accept an invitation by a pharma-
they are not representative of the whole. Neverthe-
ceutical firm to investigate the medical effects
less, it must be conceded that if we are able to
of chemicals, you are distinctly more likely,
measure that part of the information flow which is
to put it mildly, to turn up a new drug than a
embodied in scientific papers and in patents, then
new alloy or a new radio-star. ”
we would in principle be able to measure at least
a part of the output of R and D activity. The logical fallacy lies in assuming that, because
It may still be argued that there is as yet no accidental features are present inindividualcases,
satisfactory way of reducing scientific papers or it is therefore impossible to make useful statistical
patents to comparable standards as yardsticks of generalizations about a class of phenomena, whether
measurement. This is a question on which there natural or social. Those concerned with am indi-
has been a certain amount of empirical research vidual street accident are always impressed by the
and on which there are some important findings. peculiar features of the occurrence - if X had not
The third and fourth sections of this paper are, postponed his journey by 15 minutes, it would never
therefore, largely devoted to the use of scientific have happened; if Y had not been worried by his
papers and patents, as a possible means of measur- wife’s illness, if the street sign had been lOyards
ing part of the information flow of research and ex- further down etc., etc. All these factors are un-
perimental development. It is argued that the re- doubtedly extremely important in determining the
sults of empirical work already justify the use of specific form of each accident, which individuals
scientific papers and patents for some output are involved in it, the nature of their injuries and
measurement purposes, despite the severe diffi- so forth. But they in no way prevent the statisti-
culties and limitations involved. cian from forecasting with a high degree of accuracy
The measurement of that part of the output of the number of street accidents which will occur in
research and experimental development work which a given month to a given country, andfrom classi-
is embodied neither in published papers nor in fying many features of the “accidents”. Similarly
patents, presents greater difficulties and evenless those involved in any individual scientific discovery
empirical work has been done. Nevertheless, there or invention are always impressed by the number
are some possibilities of measurement, although of apparently accidental features, and often they
largely indirect. These are discussed in Chapter V may be right to think that but for these accidents
of this paper. this particular discovery or invention would not
For many policy purposes the information flow, have been made. But this need not prevent the
which is generated during the R and D process, is social scientist from making useful generalizations
only a means to an end. Theultimate aimisusually about a class of discoveries or inventions.
a flow of innovations, which may be considered as This is not to deny the presence of accidental
the final output of the system, while the information factors in research, as in many other human activi-
flow is an intermediate output. Chapter V therefore ties. Nor is it to deny the existence of very wide
considers the measurement of the “final output” in variations in the relationship between input and
this sense. output. All industrial production functions involve

11
 a statistical distribution with fairly wide deviations is financed by a private individual donor or a non-
from the norm. (lg) In agriculture, for example, profit institute, it may be treated in national ac-,
when the farmer uses certain inputs such as seed, counts as private consumption expenditure and
fertiliser, land and labour, he knows verywell that included in GNP. Despite this variety of treatment
the actual output per acre, or per hour of labour, by the national income statisticians, it is quite
may vary enormously from season to season. These reasonable for the economist to treat all R and D
variations in output due to “accidental” factors, expenditures as a form of social investment in both
such as weather, or to factors over which he has capitalist and socialist economies.
limited control, such as pests, do not mean that he The treatment of the public sector in many
cannot take rational decisions about the use of in- social accounts systems involves frequent use of
puts, or that he is rendered incapable of improving cost of input measures in lieu of output measures.
average yields. As with R and D, efforts at output measurement
By analogy, many R and D managers or scien- are still at a very primitive stage in areas such as
tists act “as if” they were farmers. They know that education, health services and so forth. Some
there are unpredictable and accidental factors pre- direct indicators have been used, for example,
sent in their work. But they also lmow that, if they numbers of patients in hospital or children in school,
apply their labour with ingenuity and appropriate ratios such as patients per doctor or pupils per
equipment over a sufficiently long period, they will teachers, or indirect indicators such as mortality
probably achieve some useful results. This attitude rates. But it is generally agreed that none of these
has been justified in practice by the whole growth measures yet provides any satisfactory general
of science and technology over the past hundred scheme of “output” or quality measurement for
years. The existence of commercial contract re- these services.
search institutes and the steady increase of company- It may be that a general system of output
financed R and D operations are evidence of the measurement suitable for incorporation in national
economic viability of a large range of R and D ac- accounts will never be attained and that for this
tivities, which can be managed with some degree of purpose we shall have to continue to use input
rationality, despite the unpredictability ofparticular measures. In the socialist countries measurements
experiments. for some service activities, whether of “inputs”or
“Outputs” are in any case often excluded from the
national accounts system. But this need not prevent
USE OF INPUT NIEASURES AS the development of output and efficiency measures,
A SURROGATE FOR OUTPUT which are specific to each activity and which can be
used to compare the performance of organizations,
The converse of the argument on the role of acci- of individuals and of countries in that activity, and
dental factors in research and experimental develop- with the financial outlays for each activity.
ment is the view that variations in output are so One of the greatest difficulties in representing
slight that they can be disregarded or that they research output in a form suitable for a national
average out over alarge enough sample. Some such accounts scheme is that so much of it is:
assumption is in fact involved in the use of input (a) Feedback output to other parts of the system.
measures as a surrogate for output measures. We (b) Output which is used only after long and
substituto unpredictable time lags.
already know enough about output variations to know
that for many purposes input measures are not enough (c) Output which can be “consumed” an infinite
although they are better than nothing. We cannot number of times.
compare relative efficiency unless we have some This applies above all to basic research, whose func-
direct or indirect measure of output as well as input. tion is to generate and maintain a “multi-purpose
In market economies the use of R and D input knowledge base”. (21) It is clear that the results of
measures in national accounts may nevertheless be basic research are, by definition, not intended to
justified in the absence of any output measures, as serve any specific practical aim, but to provide a
in the case of many other service activities. But it flow of general scientific information which maybe
must be remembered that the actual treatment of R used in a great variety of applications. This “output”
and D in nationalincome statistics today is complex, cannot be assessed in relation to the policy goal of any
depending upon the type of economy, the sector of particular government department or industrial en-
performance and method of finance. (20) In many terprise, but only in a much wider context. Even “ap-
capitalist economies, when a piece of research is plied research” which does have specific practical
both performed and financed by government, it will objectives may find very wide applications far outside
normally be treated as part of GNP - final output - the context of the original research. For this reason
and measured by its input cost. This would also be it is both more practical and more logical to attempt
true if the work were performed extra-murally in measurement of --research output initially by the
industry but paid for by government. But if the flow of published information, rather than indirectlv
work is “company-financed” it will normally be through the ultimate applications. This becomes
treated by the firm as a cost of production and will progressively less true as we move across the spec-
not be measured as a final product. If the research trum to experimental development.

12

.__-_-.__ ~ .------- - -..

 NOTES

(16) F. Machlup, paper contributed to The Rate (19) F. Machlup, op. cit. (16)
and Direction of Inventive Activity, Princeton C. Freeman, paper in Problems of Science Policy,
University Press, 196 2 OECD, Paris 1968
(17) R. Taton, Reason and Chance in Scientific (20) F. Machlup, The Production and Distribution
Discovery, Trans. A. J. Pomerans, New York, of Knowledge, Princeton, 1962, pages 184-7
1957
J. P. Lamouche, Recherche Scientifique et Comp-
(18) A. H. Cottrell, Science and Economic Growth tabilite Nationale, EEC, Brussels, 1968
in the United Kingdom, British Association, 1966
(21) J. Schmookler, op. cit. (6)

13
 Chapter III

THE USE OF SCIENTIFIC PAPERS IN MEASUREMENT OF RESEARCH OUTPUT

NUMBERS OF PAPERS AND bitolas the small group. For example, Pareto’s law is
OTHER OUTPUT MEASURES generally valid for a country but not necessarily
for the individual firm or for a village. It may be
Three main yardsticks have been used for the legitimate to use quantitative measures as a sub-
measurement of output of basic research: scien- stitute for a qualitative assessment or a combined
tific publications (usually “papers”), “discoveries” quantity-weighted-by-quality index, if it can be
or other major contributions to the advance of shown for any field of investigation that at the se-
knowledge, and colleague evaluation or peer judge- lected level of aggregation the quantity of scientific
ments. The last two methods often depend upon papers does not vary greatly from the combined
some qualitative evaluation of the first. Moreover, quantity/quality index. Thus, if the national origin
a count of papers, whether weighted or otherwise, of a list of major medical discoveries in the twen-
is the only method which lends itself readily to tieth century conformed almost exactly to the na-
large-scale statistical application. Whilst allthese tional origin of the “key papers” as assessed by
methods can easily be used on a smallscale simul- experts, and the pattern of both in turn conformed
taneously or combined for ranking purposes in a closely to the national origin of the gross number
field which is well-known to the investigators, it of medical research papers appearing in a selected
is difficult to extend such combined indices of per- range of journals, it might be legitimate touse the
formance across a wide range of disciplines and third measure for some purposes as a proxy for the
countries or over an extended time period. other two.
The promotions board, or appointments com- In practice the difficulty is that no one has yet
mittee for research or university teaching posts established the range of applications, or the limits
will normally try to take into account all three within which such surrogate quantitative measures
methods of evaluating the output of candidates. This may confidently be used. Some empirical workhas
is quite feasible, since they are usually familiar shown a degree of correlation between the three
with the individuals concerned, as well as the subject. types of measurement in a few areas of application.
But despite the very widespread practical applica- But the errors and difficulties associated with each
tion of such rough and ready output assessment, type of measurement, as well as the restricted area
little success has been attained in generalizing this of validation, do not yet give sufficient grounds for
experience across a wider frame of reference. confidence in widespread application. Reasonable
Most historians of science and sociologists have also caution dictates that, wherever possible, several
tended to work mainly on a “micro” scale, using methods of assessment should continue to be used
the second and third techniques of output measure- simultaneously as with the interviewing boards.
ment. As in the case of the appointments board, Experimental work should be continued, as some
this is quite reasonable procedure at the micro level. of the results thrown up by straightforward quanti-
To everyone who is familiar with research, it is tative analysis are of great interest for science
obvious that it may be dangerous to rely on a simple policy.
count of numbers of papers in assessing the output For example, Rangarao has estimated that the
of any particular individual or small group. But it average output of papers by Indian researchers is
does not necessarily follow that such quantitative approximately e uivalent to one paper every lo-12
techniques cannot be applied to much larger aggre- scientist-years. 7 22) This may be compared with
gates. A great deal of statistical analysis is based Price’s rough estimate for world science of an out-
upon the knowledge that in a sufficiently large popu- put of one paper every two scientist-years. (23)
lation many individual variations can be ignored, Careful examination of both estimates would be
even though they cannot be ignored at the level of necessary to ascertain the degree of comparability

14
 Qualidade vs Quantidade dos papers

in definition of input (numbers of full-time equivalent of the findings of Wayne Dennis (28) and others on
research scientists) and output (range and proce- the output of the most eminent men of science.
dures of abstracting services) and dates of measure- These show that the most outstanding scientists
ment (which differ slightly). At first sight the dif- have usually been prolific in the volume of their
ference in output is very great and at variance with output. Price is, of course, well aware that one
Price’s own hypothesis of a roughly similar input/ paper by Einstein cannot be compar,ed with one or
output ratio for world basic research in general. even 100 papers by “John Doe”; (2g) and that there is
But if valid it provides important supporting evidence “no guarantee that the small producer is a nonen-
for the views of Dedijer(24) and others on the tity and the big producer a distinguished scientist’!
research environment in Indian research institutes He argues, nevertheless, that in spite of obvious
and universities. Kapitza has also estimated very exceptions and variations, “on the whole there is,
roughly that the output of Russian research scien- whether we like it or not, a reasonably good corre-
tists in terms of papers is only half that of their lation between the eminence of a scientist and his
United States colleagues. (25) He emphasizes the productivity of papers. It takes persistence and
severe statistical difficulties in making such esti- perseverence to be a. good scientist and these are
mates and all these comparisons need very consid- frequently reflected in a sustained production of
erable care in their interpretation. It would be scholarly writing”. (30)
unwise to jump to policy conclusions without serious Price has used the output of scientific papers
critical analysis of the data and methods. It seems and the number of scientific journals to generalize
likely that the Indian definition is much wider than about the long-term growth rate of the scientific
Price’s, including scientists who donot publish at all. community, (31) in the United States and elsewhere.
In broad terms he has deduced an input measure
from an output measure (the opposite procedure
THE USE CF OUTPUT MEASURES from some national income statistics), arguing
IN THE SOCIOLOGY OF SCIENCE that the number of scientists, the number of papers
and the number of journals have allbeenincreasing
Several important contributions to the sociology of at an exponential rate of 5% to 7% per annum (i. e.
science have already been made by studies based on doubling every 10 to 15 years).
the use of scientific papers as a method of measure- These observations provide extremelyinterest-
ment. The broad scope of these contributions has ing hypotheses on the long-term trends in science
covered such questions as the following: the age at and likely future trends, but it should be notedthat
which scientists are most productive in various they are not entirely consistent with Price’smodi-
disciplines; the relative contribution to science of fied version of “Lotka’s Law” and his generaliza-
industrial, government and university scientists; tions about the much slower rate of increase in
the long-term rate of growth of the output of various numbers of “good scientists”. He argues that the
scientific disciplines and sub-disciplines; the rela- “total number of scientists goes up as the square,
tive contribution of various countries to world science more or less, of the number of good ones”. (32)
in particular disciplines; the relative contribution But if “good” scientists are much more prolific
of male and female scientists to research output; than lesser ones, then not only would there be
the growth of multiple authorship of scientific papers diminishing returns, in terms of average quality
and its implications; the relative contribution of of output per scientist, as Price postulates, but
outstanding and lesser scientists to research; the there would also be a continuous slowing of the rate
institutional environment most conducive to high of increase in quantity of papers, by comparison
research productivity. Whilst most of these results with number of scientists. The point is by no means
must be regarded as in need of further validation academic, since the period under consideration is
and testing, they already constitute an important three centuries and “Lotka’s Law” suggests that
body of knowledge. Here it is only possible to indi- the “good” scientists account for a high proportion
cate very briefly some of the most important findings of total output. ( 33) The tendency might, of course,
and some of the hypotheses which these have gener- be offset by other long-run changes, such asvary-
ated or tested. ing pressures to publish, the changing pattern of sci-
Alfred J. Lotka, in a pioneering article in 1926 entific careers, “professionalization” of research,
on “The frequency distribution of scientific produc- the growth of post-graduate scientific research de-
tivity ” , (26) demonstrated for some branches of grees and so forth. The variations which apparently
natural science that for every 100 authors whopro- exist between Indian, Soviet and United States’ out-
duce only one paper in a particular period, the put of papers per “scientist” provide grounds for
number of people.producing “n” papers is approxi- considerable caution in making generalizations
mately “l/n2 ” (Table 2) . Derek Price has provided about “world science” over long periods. Not only
additional evidence supporting Lotka’s observations, are publication practices different but the amount
has reformulated the “law” governing distribution of secondary material in journals varies significantly.
of productivity and has pointed to some of its impli-
cations for the long-term growth of the scientific
community, (27) Price emphasizes the importance

15
 THE NEED TO RELATE “OUTPUT” “input” data on R and D expenditures also suggest that
TO “INPUT” MEASURES there are major variations between countries in the
proportion of GNP devoted to basic and appliedre-
From this it is evident that Price’s daring generali- search, as with R and D as a whole. (37) It would
zations and first approximations will benefit from be surprising if these were not reflected in varia-
detailed national surveys, relating institutional and tions in the proportion of GNP devoted to that part
other sociological factors to the broad long-term of science concerned with the output of papers (this
international trends which he has discerned. One is not necessarily the same as basic research).
of the most valuable lines of research would be to Price accepts, of course (and it is evident
relate his “output” series to an equivalent long- from his table) that a large number of countries in the
term “input” series in each country. Unfortunately, under-developed world do not spend 0.7% of GNP
this is very difficult because satisfactory manpower on basic research. But he argues that 0. 7% is the
“input” statistics are available onlyforveryrecent minimum for a “genuine scientific effort”. It could
periods. The proportion of the total stock of scien- be argued that this figure should be an absolute
tists engaged in research and experimental develop- rather than a ratio if it is a threshold. But Price
ment has been changing fairly rapidly, so that figures states that “with few exceptions, countries either
for total numbers of scientists are insufficient for participate in basic science to this extent or they
this purpose. Moreover, the distribution of research stay out of the race completely”. (33) But anumber
scientists between industry, universities, and other of developing countries are trying to enter the
laboratories will affect the output as measured in race and some have only joined it recently, so that
terms of papers. Even for the last few decades it would seem a more reasonable hypothesis that
where we have figures for the total numbers of R countries will tend to spend a slowly rising propor-
and D scientists and engineers for a few countries, tion of GNP on R and D in general and on basic re-
there are major methodological problems to be re- search in particular. This would also be consistent
solved. The “input” manpower statistics for Rand with the observed phenomenon of a rising proportion
D do not distinguish between those who produce of expenditure on higher education, which closely
papers and those who do not, nor by the type of papers. affects the output of scientific papers. It would also
Price draws a very sharp distinctionbetween “sci- be more consistent with Price’s own long-term
ence”, which leads to an output of “papers” and historical observations on the growth of scientific
“technology” which does not, in terms of his defini- activities.
tions. Many engineers and scientists and particu- These criticisms are made not in order tobe-
larly social scientists would not accept this rigid little the major contribution which Price has made
distinction and would argue that an important part to social studies of science but in order to bring
of the “output” of “technology” or “applied research” out the need for studies relating “input” and “output”
is also embodied in the form of research papers. measures and for refinement of his first herioic
Price concedes this for electronics, industrial generalizations on the output of scientific papers.
chemistry and computer science. (34) This question
is discussed further in later chapters.
THE NEED TO RELATE OUTPUT MEASURES
Perhaps even more controversial are Price’s
TO INSTITUTIONAL FACTORS AND SOCIAL
first generalizations on the distribution of scientific
CHANGES
papers by country of publication (Table 3). He is,
of course, well aware that country of publication
The general justification for Price’s original at-
is not the same as country of origin (whether in the
tempts to quantify aspects of scientific activity
sense of where the work was carried out or of citi-
was wellputby Dobrov andMackay in “Nature”:( 3g)
zenship of the author(s), and suggests some adjust-
ments to allow for well-imown cases of international “Science, in its material aspects as a complex
journals, such as “Nuclear physics” (Netherlands) developing system involving people, informa-
and “Nuovo cimente” (Italy). But even if it is as- tion, materials and money, invites study in
sumed for purposes of argument that country of pub- terms of statistics, cybernetics, systems analy-
lication is equivalent to country of origin, few econo- sis and other appropriate modern techniques.,
mists would accept the conclusions which he draws The aim of the science of science is the con-
from his analysis. He argues that the “cost” of pro- struction of theoretical foundations for the or-
ducing a research paper is equivalent to $20,000 per ganization, planning, management and progno-
paper. It is not clear how this figure is derived but he sis of science. These foundations must be
deduces from it that all countries contributing to built from a solid corpus of observations and
world science are spending the equivalent of 0.7% of experiment which will satisfy the criteria of
GNPonbasic research. (35) These estimates disre- science. . . The interesting preliminary corre-
gard the substantial data on international variations lations found by Price indicate that as in the
in research costs, which show, for example, that the case of lung cancer there is something worth
input cost per research scientist is much lower in detailed study . . . Similarly in the study of
Europe than in the United States of America, (33) and science itself statistics indicate that there may
lower still in Japan and the developing countries. The be things to be explained by detailed study of

16
 the actual mechanisms . . . Needless to say, as explain the relatively poor achievements of some
in all branches of science, the science of sci- research institutes which are not able to transfer
ence can only advance . . . by the growth of a their elderly researchers.
corpus of knowledge which after criticism is But here again Lehman’s findings need to be
accepted by the scientific community. ” interpreted with sociological insight and have been
criticized on the grounds that they ignore the extent
Whilst not everyone might be happy with the expres- to which environmental factors, rather thanpsycho-
sion “science of science”, few workers in this field logical factors, affect creativity. Some psycho-
of studies would dissent from these observations. logists have dismissed them as a statistical arte-
There are many studies which suggest that auseful fact. (45) Wayne Dennis argues that historians and
“corpus of knowledge” can be built up provided the anthologists show a bias in favour of citing works
“actual mechanisms” are subjected to critical analy- done in earlier historical periods and that the ap-
.sis. Some of these have been contributed by parent decline in major contributions with age re-
Dobrov and Mackay themselves. flects the behaviour of historians rather than that
G. M. Dobrov (Director of the Kiev Institute for of scientists. He demonstrates that, because of
the History of Science and Technology) has made the exponential growth of scientific publications, the
important studies of the relative rate of growth of proportion of the total which are cited in histories
various disciplines in the USSR(40) using not only and anthologies is constantly declining:
published papers but also all those unpublished re-
“Since there is a general reduction over time
search reports which are accessible through the
in the likelihood of papers being cited, the later
Soviet scientific and technical information service.
papers of men . . . would be expected to be
He has also made interesting observations on the
cited less often than their earlier papers,
lags in citation of foreign scientific literature, as
simply because of changes in citation practices
compared with “home-produced” scientific papers.
quite apart from any relationship between a e
It would be valuable to extend such studies in order
and quantity of high quality creative works 11. (f5)
to assess such factors as the relative efficiency of
information services, the extent to which scientists Dennis himself demonstrated that 100 eminent
read foreign language literature, the extent to which nineteenth century scientists maintained a fairly
foreign language results have to be “r-e-interpreted” uniform rate of output between age 30 and age 70. (46)
or repeated to be absorbed in the localmainstream His criticisms suggest that for some purposes a
of scientific advance and into technology, and the simple count of papers may be a less misleading
“lags” in the world basic research system. It is index than “major contributions” culled from antho-
often assumed by economists on an over-simplified logies, biographies or science histories. Schmook-
model that “basic research”results areuniversally ler came to avery similar conclusion with respect to
and freely available and that therefore from a indices based on a simple count of patent numbers,
strictly selfish economic (as opposed to a cultural) as opposed to “major inventions” (see Chapter IV).
point of view, there is no need for everycountryto In any case, this controversy once again em-
perform basic research. Such grossly over- phasizes the care, discrimination and insight with
simplified models may be dangerously misleading which these research results must be interpreted.
and this type of research could lead to empirical The institutional factors affecting output of papers
testing of their assumptions. are not only of the greatest importance in compar-
Dobrov has also made interesting comparisons ing age groups in various environments but alsoin
of the relative output of male and female scientists comparisons of the total contribution to science of
in the USSR. Additional results were demonstrated researchers in various sectors of the economy.
by Dodge, who showed that the productivity of women Several inquiries have shown that university scien-
scientists in terms of numbers of papers was about tists make a relatively greater contribution to the
half that of men for employees of higher educational output of scientific papers than industrial research
institutions, and that the discrepancy between the scientists (for example, Table 5). But it would
sexes was greatest at the highest levels of achieve- obviously be absurd to assume from this that uni-
ment(41) .( Table 4). VeTSity researchers are “more productive” than
In another important research area Dobrov has industrial researchers. Any such comparison would
contributed to studies of the age composition of scien- have to take into account the restrictions on publi-
tific teams in relation to laboratory recruitment.(42) cation confronting many industrial scientists, the
H. C. Lehman used the output of outstanding cre- extent to which the output of industrial research
ative works to relate creativity to age. (43) (44) scientists is embodied in other written forms (in-
He found that scientific creativitytypically reached ternal memoranda, patents, etc. ) and the differing
a peak in the late 30’s and then declined, but the objectives and motivation of the two groups.
peak was earlier in highly abstract disciplines such An example of careful assessment of suchfac-
as maths and physics, and later in more empirical tors is the paper by Cotgrove and Box on “The produc-
disciplines. Such findings have been used by re- tivity of industrial scientists”. (47) They found that
search managers to justify transfer of older men 56% of a sample of 400 scientists considered that
from research to other activities, and by others to company policies hindered publication and 42%

17

_.---- .
thought that company security restrictions lowered the 17 institutes with the largest output of papers
their publication rate. This is in line with the find- contributed 300/O of all papers to Indian journals, but
ings of earlier surveys in Britain and the United 50% of all papers appearing in foreign journals
States based upon inquiries to companies rather (Table 6) . To interpret these findings in a way which
than to individuals. These showed that only 140/0of would be useful for policy conclusions would require
174 major United States industrial firms conducting a great deal of additional sociological and economic
basic research published “substantially all” of their research. But Rangarao’s findings provide an essen-
research finding.s and a further 26% “most”of their tial and stimulating first step.
findings. (48) An FBI survey in Britain showed that An example of the sophisticated use of an output
23% of firms followed a “liberal” policy on publica- measure based on numbers of medical discoveries
tion, while 31% indicated “nil” or “very limited” with penetrating sociological analysis of institutional
publication. (4g) Among other reasons given for changes is Ben-David’s study of “Scientific produc-
believing that publication rate was lower than it tivity and academic organization in nineteenth cen-
could be, Cotgrove and Box reported that 25% of tury medicine”. (50) Having demonstrated the out-
their sample thought the company did not allow ade- standing German contribution to medical discovery
quate time for scientists to write up their research in the second half of the century (Table 7), he sought
results for publication and 23% thought that patents to explain this nineteenth century lead in terms of
diminished the need for publication. the earlier professionalization of medical research
In addition to demonstrating the source of sci- in Germany and related organizational factors.
entific papers by sector of origin (Table 5), Rangarao A. Zloczower followed up this study with further
provides interesting data on the degree of concen- independent measures of the structure of German
tration in research output in India; for example, research in the nineteenth century. (51) Neither of
eight universities (out of 68) accounted for 500/O of these studies has been related to a count of numbers
university research papers and 44 institutions (out of papers by national origin and this would be a
of 2, 000) contributed 50% of all papers.. It would be valuable experiment to see how far the results were
useful to have supplementary information on the consistent with Ben-David’s and Zloczower’s find-
relative quality of output from various institutions ings; but Ben-David did make (;;p of publication
and in various disciplines. Rangarao shows that data in his study of pyschology.

NOTES

(22) B. V. Rangarao, “Scientific Research in (33) Derek J. de Solla Price, op. cit. (l), page 45
India: an analysis of Publications”, Journal of (34) Derek J. de Solla Price, “Historical Rela-
Scientific and Industrial Research, Vol. 26, No. 4, tions of Science and Technology”, Technology and
1967, pages 166-176 Culture, Fall 1965, page 562 and The difference
(23) Derek J. de Solla Price, “Nations must Pub- between science and technology, Edison Foundation,
iish or Perish”, Science and Technology, October 1968
1967, page 87 (35) Derek J. de Solla Price, “Research on Re-
(24) S. Dedijer, “Under developed science in search”, from Ed. D. L. Arm, Journeys in Sci-
under-developed countries”, Minerva, Vol. 2, ence, University of New Mexico Press, 1967
No. 1, Autumn 1963 (36) E. D. Brunner, The Cost of Basic Scientific
(25) P. L.> Kapitza, in Pravda, 20 January 1966 Research in Europe: Department of Defence Ex-
perience, 1956-1966, Rand, April 1967
(26) A. J. Lotka, “The Frequency Distribution of
Scientific Productivity”, Journal of the Washington C. Freeman and A. Young, op. cit. (12),
Academy of Sciences, 16 (1926), page 317 page 91-98

(27) Derek J. de Solla Price, op. cit.- (1) (37) OECD, op. cit. (2)

(28) Wayne Dennis, “Bibliographies of Eminent (38) Derek J. de Solla Price, “Nations must Pub-
Scientists”, The Scientific Monthly, September lish or Perish”, Science and Technology, October
1954, pages 180-183 1967, page 87

(29) Derek J. de Solla Price, op. cit. (1)) page 45 (39) G. M. Dobrov and A. L. Mackay, “Not the
Mysticism but the Science of Numbers”, Nature,
(30) Derek J. de Solla Price, op. cit. (1)) page 41 Vol. 219, 1968, page 662
(31) Derek J. de Solla Price, Science Since Baby- (40) G. M. Dobrov, Nauka o Nauke, Kiev, 1966
*, Yale University Press, 1961, page 107
(41) N. T. Dodge, Women in the Soviet Economy,
(32) Derek J. de Solla Price, op. cit. (l), page 53 Johns Hopkins, 1966, Ch. 10

18
(42) G. M. Dobrov, “Scientific Potential as an Ob- (47) S. Cotgrove and S. Box, “The Productivity
ject of Investigation and Control in the Soviet Union”, of Industrial Scientists”, The Technologist, 1967,
Decision-making in National Science Policy, Church- pages 98- 107
hill, 1968, pages 189-201
(48) National Science Foundation, Publication of
(43) H. C. Lehman, Age and Achievement, Prince- Basic Research Findings in Industry, 1957-1959,
ton University Press, 1953 and Science, 1958,
NSF 61-62, Washington 1961
pages 1213-1222
“Men’s Creative Production (49) Federation of British Industries, Scientific
(44) H. C. Lehman,
Rate at Different Ages and in Different Countries”, and Technical Research in British Industry, Table
Scientific Monthly, May 1954, pages 321-325 7, London, 1947

(45) Wayne Dennis, “The Age Decrement in Out- (50) Joseph Ben-David, op. cit. (3) and J. Ben-
standing Scientific Contributions: Fact or Artefact? I’, David and R. Collins, “Social Factors in the Ori-
American Psychologist, 1958, pages 457-460 gin of a New Science”, American Sociological
Review, 1966
(46) Wayne Dennis, “Age and Productivity Among
Scientists”, Science, 1956, page 714 (51) A. Zloczower, Career Opportunities and the
Growth of Scientific Discovery in Nineteenth Cen-
tury Germany, Occasional papers in sociology,
Hebrew University of Jerusalem

19
 Chapter IV

THE USE OF PATENT STATISTICS FOR MEASUREMENT

ciência aplicada vs tralho inventivo

PATENTS, INVENTIVE ACTIVITY developing it to the point of commercial suitability.

AND “APPLIED” RESEARCH Patents do not arise from development work in
this scheme.
In the conventional NSF or OECD scheme of
As with papers, SO with patents, there are severe dif-
classification “inventive work” does not appear as
ficulties inusing crude numbers as output measures.
a separate category, but its content would be di-
Both the publication of scientific papers and the
vided between “applied research” and “experimental
registration of patents have become well-established
development ‘I, and sources outside the R and D
international systems, but there are still important
system. Whereas in Machlup’s scheme patents
national variations in custom, in laws and in cul-
would appear exclusively as the output of “inven-
tural patterns which necessitate great care in at-
tive work”, in the conventional scheme patents
tempting international comparisons and comparisons
would arise both from applied research work -and
over time. For example, in some countries it is
from experimental development. They would pro-
difficult to obtain a patent because of rigorous tests
bably arise in larger numbers from experimental
for originality; in others there is no such test. In
development work than from applied research. In
some countries there are long delays in processing
any case the “input” categories to be related to
patents; in others, little delay, and so forth. There
patent “output” would be the combined inputs into
is consequently the same need for sociological in-
applied research and experimental development. In
sight and historical sense in the interpretation of
some industries Mueller found strong correlations
patent statistics as in the interpretation of output of
between input data for development work and lagged
scientific papers, and similar difficulties in setting
patent data but in others a stronger correlation for
up any general scheme of measurement.
applied research and lagged patents. (52)
There is some degree of overlap between the
In either scheme of measurement, some “in-
two forms of output. The inventor or inventors who
ventive work” would be done outside the professional
take out a patent may often afterwards publish arti-
R and D system altogether. Machlup’s scheme and
cles in technological journals relating to their work.
the variants used by Ames(52) and Schmookler(54)
In many countries prior publication of the idea ina
are useful conceptually but all classification schemes
journal constitutes valid grounds for the refusal of
do violence to the real complexity of the R and D
patent protection. But whereas most published papers
process. Thus Schmookler, who also distinguished
(even in technological journals) provide information
a special category of “inventive activity” whilst
which is not patentable because of its generality,
retaining ” applied research”, wrote:(54)
patents always refer to quite specific applications.
Because they are so specific some economists “Inventive activity is often so intertwined with
prefer to use the category of “inventive work” research and development today that a scien-
rather than ” applied research”, or “applied re- tist or engineer might have trouble deciding
search and experimental development”, and to re- which function he was performing at a given
gard patents mainly as a measure of :he output of moment . . . Thus a scientist, while studying
“inventive work”. Thus, in Machlup’s scheme a given phenomenon, may begin to think about a
(Table 1) “inventive work”refers to a stage preced- particular possible industrial application. At
x development, but not corresponding to applied this point we might say inventive activity has
research. In his scheme ” applied research” dis- begun. He may then attempt to create this ap-
appears as a separate category and would be divided plication on a laboratory scale. If he makes
between “basic research” and “inventive activity”. the attempt and finds that his application does
He uses “development” exclusively in the sense of not work as expected, he may return to do
taking a ready-made invention (from outside) and more research (thereby temporarily

20
 terminating his inventive activity) . . . While use of patent statistics for limited purposes for a
these shifts in r81e may be difficult to keep long time, undoubtedly the major applications
track of as a practical matter, the r8les ... were made by Jacob Schmookler of the Universit
are different: understanding a phenomenon is of Minnesota before hisuntimelydeathin 1967. (55 3
one thing; creating an industrial process or As a result of his investigation into United
product based on that understanding is another.” States patents from 1937 to 1957 he concluded that
non-patenting of inventions was much more common
In the conventional classification this scientist would since 1940 and more common for large than small
probably be classified with “applied research”, but This was attributed to changes in attitude
firms.
these difficulties would apply to all the various of large corporations arising initially from (v) and
schemes of classification of R and D which have been
(vi) above but later affecting (ii) and (iii). Where-
put forward, and most statisticians would agree
as the absolute numbers of patents issuedannually
that the sub-division of R and D into “stages” or
have increased very little in the United States in
“categories” must be regarded as a very rough ap-
the post-war period, they have continued to increase
proximation. The classification depends to some
in most European countries, although more slowly
extent on subjective judgements by respondents on
than R and D expenditure. This suggests that these
the nature of their work and on the degree of spe-
factors have had a much bigger impact in the United
cialization within each organization. The conven-
States than inEurope and Schmookler presents con-
tional system of classification used in input surveys
vincing empirical evidence on the reasons for this.
does have the merit of corresponding a little more
He nevertheless maintained that, used with
closely to the actual division of labour within the
caution, absolute numbers of patents are a better
present-day R and D system. Some laboratories are
guide to inventive output than “important inventions’:
recognizable as being primarily “basic research”
which he also used in his major study. This is be-
or “applied research” or “development laboratories’:
cause information on “important inventions” is
but it would be difficult to distinguish the category
very spotty and subjectively biased by historians,
of “inventive laboratories”. This reflects the in-
and because their “importance” is as variable as
creasingly science-based character of the inventive
that of patents. They are also usually a good deal
process.
less “important” than many lives of great men as-
sume. On the other hand, even though the quality
LIMITATIONS OF PATENT STATISTICS of individual patents does vary enormously, the
proportion of United States patents which are used
Whilst patents are undoubtedly one indicator of ap- commercially has risen from about one-quarter in
plied research and experimentaldevelopment activity the nineteenth century to about half today. Few
(or of inventive work - whichever classification is really important inventions are not patented, but
preferred), they measure only a part of the output. more minor ones are discarded without patent ap-
Some inventions are not patented for a variety of plications. This means that patents represent an
reasons, including the following: inventive output which has great economic signifi-
cance and covers a very large number of contribu-
(i) They are believed by the inventors to lack com-
tions to technical progress. As with scientific
mercial applications;
papers individual quality variations do not exclude
(ii) They are protected by secrecy and the inventors
the possibility of useful aggregate statistics.
believe that security is greater without patent
However, Schmookler is extremely careful not
protection;
to claim too much for patent statistics. In addition
(iii) The inventors believe that scientific and techni-
to pointing out the change in patenting behaviour of
cal leadership is more important than patent
large United States corporations since the Second
protection;
World War, he emphasizes that they are of little
(iv) Peculiarities of patent law affecting certain
value in relation to United States Government ap-
classes of invention;
plied research and inventive activity and that there
The delay, expense or difficulty of patenting;
is a bias in the United States statistics towards
I;!, Legal factors affecting patents, such as anti-
small firms and private inventors, which over-
trust legislation.
estimates their contribution. Moreover, he dis-
It may nevertheless be argued that in spite of these tinguishes very carefully between those contribu-
limitations patent statistics can be used to measure tions to technology which can be patented and those
an important part of the output of applied research which cannot, emphasizing the importance of non-
and experimental development, or of inventive patented general lmowledge in applied science and
work. The problem is to define the limits within engineering.
which they may be used. Whereas most of the work One important difference between American
on scientific papers has been done by sociologists, and some European patent statistics is that the
psychologists and historians of science, most of existence of annual renewal fees permits analysis
the research on patent numbers has been done by of some European statistics in terms of a “weight-
economists, with relatively minor contributions by ing” by years of renewal. Y. Fabian used this
sociologists. Whilst many economists have made method inhis long-term study of innovations in the

21
 iron and steel industry. (58) For obvious reasons D efficiency. This is because there are some dif-
this method can only be used in long-term historical ferences in the “propensity to patent” between dif-
studies. Schmookler’s own studies are also mainly ferent industries. This propensity is highest in
in this category, but for different reasons. those industries in which a technical advance can
Despite all these limitations, he and other very easily be copied by competitors, without much
economists have demonstrated that patent statistics independent development work. An example is the
do have valuable, if limited, applications as output drug industry. It is lowest in those industries in
indicators. As with statistics of scientific papers, which technical advances can be copied only with
it is a question of interpreting experimental statis- great difficulty and with much independent design
tical comparisons with great caution and sociological and development lasting, many years, or where
insight. government contracts play abig role. In such cases
technical leadership may often be maintained with-
out strong patent protection. An example is the
APPLICATIONS OF PATENT STATISTICS aero-space industry. In other cases, differences
in patentability may seriously affect the comparison.
The kind of economic studies in which patent statis- For example, in the computer industry advances in
tics have been used include the long-run changes in hardware are usually patentable, but those in soft-
the amount and direction of inventive output inpar- ware are not. Thus patent statistics can give only
titular industries; the relationship between these a very approximate indication of inventive output in
changes and other long-run economic indicators; each industry. Moreover. .
thev will reflect
”
some
the relative efficiency of company-financed and “inventive output” from outside the industrial R and
government-financed industrial R and D; the con- D system as well as inside and therefore cannotbe
tribution of individual firms to particular areas of directly related to R and D inputs without qualification.
innovative activity; the relative significance of Whilst absolute patent numbers are of limited
foreign and home-generated technology; and the value for inter-industry Rand D output comparisons,
measurement of individual inventive output. they may be more useful for comparisons between
A United Nations study of “The r81e of patents firms in the same industry. But, here too, caution
in the transfer of technology to developing countries” is needed, because propensity to patent also varies
showed that in most of these countries foreign ap- between firms, depending upon the attitude of
plications accounted for more than 80% of total ap- management and’ the particular circumstances in
plications (Table 8)) whereas in the United States which the firm may be placed. Freeman used
theyaccountedforlessthan 20%. (57) AnOECDstudy patent statistics to demonstrate that at critical
showed that patents granted to United States firms periods the outstanding technicalleaders in certain
in Europe were far more numerous than patents industries were ahead both in numbers of “major”
granted to European firms in the United States.(58) innovations and in terms of absolute numbers of
Changes over time in the proportion of patents patents taken out, as well as in volume of R and D
granted to local firms or to firms from various inputs. (81) But he did not attempt to show that
foreign countries could be significant measures of there was any general relationship between the ef-
the comparative success of local and foreign R and ficiency of a firm’s R and D and its patent portfolio.
D in each industry. They would be more reliable Those firms whichuse patent numbers as indicators
than comparisons of absolute numbers of patents of individual output almost always take into account
taken out in each industry in each country, for other factors affecting performance. A man who
reasons which have been mentioned. But even gets many hundreds of patents during his lifetime,
analysis of changing ratios would have to take account as Edison (1, 100) and Lanchester (900)did, is al-
of such factors as the activity of foreign subsidiary most certainly a highly creative engineer or scien-
enterprises in each country and the real possibilities tist. But if a man gets no patents, this does not
of establishing local manufacture in various indus- necessarily mean that he is not productive. More-
tries. Nevertheless, this type of analysis used with over, whether used as an indicator of output by an
discrimination over long periods could provide use- individual or by a firm, patent numbers must be
ful indicators for industrial R and D policy. Par- interpreted over a fairly long period to eliminate
ticipation by the socialist countries in the interna- the “lumpiness ” in their incidence at the “micro”
tional patent system is still too recent to permit level. For this and other reasons, Mueller con-
useful analysis of their experience in this context. cluded that R and D input measures were a more
Several studies in the United States(59) and in satisfactory indicator for inter-industry compari-
Europe(80) have shown fairly strong correlation son than patent numbers, even though he found them
between R and D expenditures classified by industry strongly correlated. (82)
and patent applications classified in the same way. Differences in propensity to patent also com-
Whilst this is interesting confirmatory evidence that plicate the interpreation of findings relating to
the two types of measurement are in some way re- patent numbers by size of firm. Some studies
lated, it unfortunately does not mean that statistics have suggested that smaller firms generate a
of patent numbers can be used directly as output greater number of patents per $ of R and D input
measures for inter-industry comparisons of R and than large firms. (88) But the conclusion that

22
smaller firms’ R and D is therefore likely to be The production of inventions and much other
more productive would not necessarily follow. technological knowledge, whether scrutinized
Schmookler has pointed out that pre-application or not, when considered from the standpoint
commercial testing is much more commoninlarge of both the objectives and the motives which
firms and that their established position, broader impel men to produce them, is in most instances
research base and technological strength will tend as much an economic activity as is the pro-
to reduce their propensity to patent as compared duction of bread. “(87)
with the smaller firm or independent inventor:
His findings, however, related only to three major
“Cross sections of patents granted classified industries.
by industry will tend to be biased downward
in the case of industries dominated by large
firms or with a disproportionately large amount “SCIENCE” AND “TECHNOLOGY” IN
of corporate invention. “(84) RELATION TO PAPERS AND PATENTS
Patent numbers can be a useful guide to the type of
Schmookler’s findings are ample justification for
output of private inventors compared with corporate
the careful use of patent statistics in economic
inventors. Independents were responsible for only
analysis. But it is important to guard against
3% of chemical inventions, compared with 9% of
over-crude interpretation of these findings. They
electrical inventions and 88% of mechanical inven-
do not mean, for example, that basic research is
tions, in the United States. (85) This is valuable
unimportant or that science does not influence the
supporting evidence for the view that professional-
course of invention. On the contrary, Schmookler
ized R and D is dominant in the most advanced
himself emphasizes that basic research provides
technologies.
the “multi-purpose knowledge base” on which in-
Since “propensity to patent” and patentability
ventors are able to draw for whichever practical
are both normally much lower in relation to govern-
purpose they may have in mind. The facility with
ment R and D thaninrelation toindustrial R and D,
which they are able to do so will depend on their
comparisons between sectors have little value. But
own education, imagination and links with those who
there may nevertheless be some merit in compari-
are engaged in enlarging this base. They thus point,
sons between the number of patents generated re-
as to Price’s findings, to the critical importance
spectively by government-financed R and D and
of the interaction between the “science” system and
company-financed R and D where both types are
the “technology” system. Schmookleruses the meta-
performed in industry. Solo has noted the very
phor of two blades of a pair of scissors, while Price
striking differences between the number of patents
makes use of Toynbee’s metaphor of two dancing
arising from the two types of activity in United
partners.
States corporations, (66) and draws a number of im-
But whereas for Price the output of “science”
portant policy conclusions relating to diffusion of
is published papers, whilst that of “technology” is
military-space technology.
artefacts, for Schmookler “technology” includes
From the standpoint of economic analysis, by
published papers as well as other types of informa-
far the most important application of patent statis-
tion and is defined as a “social pool of knowledge
tics has been Schmookler’s demonstration that in-
of the industrial arts”. For Price “the one part
ventive activity in several major United States
has papers as an end product, the other part turns
capital goods industries was demand-induced. He
away from them”. (88) For Schmookler progress
compared the long-run waves in successful patent
in “technology” is derived from four sources: dis-
applications with indicators of output in the same
coveries in engineering, discoveries in the applied
industries and found a high degree of correlation:
sciences (which generalize about classes of pro-
“When time series of investment (or capital ducts and processes), inventions and sub-inventions
goods output) and the number of capital goods (which embody knowledge about specific products
inventions are compared for a single industry, and processes). In his system the first two are the
both the long-term trend and the long swings output of applied research, the third of inventive
exhibit great similarities, with the notable dif- activity and the fourth of experimental development.
ference that lower turning points in major Together they constitute a pool of knowledge, part
cycles or long swings generally occur in capital of which is published freely and part of which is
goods sales before they do in capital goods protected. (89)
patents. At first sight the Price definitions offer great
advantages in simplicity in treatment of input and
He found that trends for the 900 “important inven-
output but it should be noted that Price is able to
tions” which he also analysed exhibited the same
maintain his definitions only by allocating elec-
features. He concludes:
tronics, industrial chemistry and computer science
“the fact that inventions are usually made be- to “science”, instead of “technology”, since inhis
cause men want to solve economic problems or view the publication pattern in these areas resembles
capitalize on economic opportunities is of over- that of “science”. These are, however, the three
whelming importance for economic theory ... most important and fastest growing areas of all

23
modern industrial technology. It seems, therefore, experimental development in any country, there is
that the “output” of new “technology” must be mea- one major advantage: the work is carried out in
sured by an amalgam of several indicators, inclu- order to contribute to the achievement of specific
ding both patents and technological publications of policy goals. It is therefore possible to relate the
various kinds. Important work has been done on whole of the applied research and experimental de-
internal communication channels and patterns of velopment in any particular area to the test of overall
communication between technologists in R and D performance in relation to the policy goal. Inbasic
projects. (See page 23 and references (91) to (96).) research the ultimate applications of new knowledge
But little has so far been done on the published are so diverse and the time lags so great that we are
journal literature of technology. Price has made usually thrown back on publications as the only prac-
interesting observations on the lack of disclosure tical system of measurement, using if possible ad-
in technological journals which he attributes to pro- ditional criteria of academic excellence, citations,
prietary “hoarding” of knowledge. From this point or other quality adjustments. But in applied re-
of view the socialist countries should enjoy major search and experimental development, although
advantages in measurement, since internal techno- “spin-off” and “feed-back” certainly occur and are
logical reports of all kinds, as well as patents, and extremely important, the purpose of the R and Dis
published papers are collected centrally for all R to contribute to specific goals, which may provide
and D projects. a yardstick for rating performance. In the final
However, in considering applied research and chapter of this paper we turn to these criteria.

NOTES

(52) D.C. Mueller, “Patents, Research and De- (61) C. Freeman, op. cit. (60)
velopment, and the Measurement of Inventive
C. Freeman, R. C. Curnow, C. J. E. Harlow and
Activity”, Journal of Industrial Economics, No-
J. K. Fuller, “Research and Development in
vember 1966, Vol. 15, Part I, pages 26-37
Electronic Capital Goods”, National Institute
(53) E. Ames, “Research,Invention, Development Economic Review, No. 34, November 1965
and Innovation”, American Economic Review,
(62) D. C. Mueller, op. cit., (52)
June 1961
(63) F. M. Scherer, “Firm Size, Market Struc-
(54) Jacob Schmookler, op. cit. (6), page 9
ture, Opportunity and the Output of Patented In-
(55) Jacob Schmookler, op. cit. (6) ventions”, American Economic Review, Decem-
ber 1965
(56) Y. Fabian, “Sid&urgie et croissance dco-
nomique en France et en Grande Bretagne (1735-
(64) Jacob Schmookler, op. cit. (6), page 25
1913) - Les Brevets en G-B”, Cahiers de l’ISEA,
February 1965 (65) R. Nelson, J. Peck and E. Kalachek, Tech-
nology, Economic Growth and Public Policy,
(57) United Nations, The R81e of Patents in the
Brookings Institution, 1967, page 58
Transfer of Technology to Developing Countries,
New York, 1964 (66) R. Solo, “Patent Policy for Government-
(58) C. Freeman and A. Young, op. cit. , (12) sponsored R and D”, Idea, Vol. 10, No. 2, 1966
Patent Practices of the Department of Defence,
(59) National Science Foundation, Science and Committee on Judiciary, U. S. Senate, 72757,
Engineering in American Industry, (NSF 56-16), U. S. Government Printing Office, Washington,
Washington, 1956. 1961
J. Schmookler, op. cit. (21)) page 46 (67) Jacob Schmookler, op. cit., (21) page 208
D. C. Mueller, op. cit., (52) (68) Derek J. de Solla Price, op. cit., (34)
(60) Y. Fabian, Measures of Output of R and D, (69) Jacob Schmookler, op. cit., (6)
OECD, 1963
C. Freeman, “The Plastics Industry: a Compara-
tive Study of Research and Innovation”, National
Institute Economic Review, No. 26, November 1963
page 35

24
 Chapter V

INNOVATIONS AND THE R AND D SYSTEM

Invenção ≠ Inovação => inserção no mercado

INVENTION AND INNOVATION of intermediate output must be supplemented by

measures of final output, for most policy purposes.
Schumpeter made the valuable distinction between For ex-post studies the measurement of inter-
“invention” and “innovation” , which has since been mediate output itself often involves reference to
generally adopted by economists, althoughnot quite final output. For example, lists of “important” in-
in the original form. It is avital distinction because ventions or “key” patents can only be made ex-post
m=v “inventions” (and much other “output” of R using the criterion of successfulinnovation to esti-
and D) may never lead to an innovation. Since he mate “importance”. Most histories of “inventions”
regarded “inventions” as exogenous to the economic are actually histories of “innovations”. It is the
system, Schumpeter stressed the decision of the benefit to society, however measured, which is the
entrepreneur to commercialize an invention as the ultimate criterion of R and D output.
decisive step and defined the entrepreneur as the
“innovator”. Today, when a great part of inventive
work is professional, the expression “innovation” COST-BENEFIT ANALYSIS OF INNOVATIONS
is often applied to the whole process from laboratory
to industrial use. But it is also used in the more If a flow of successful innovations is considered as
specialised sense of the first commercial intro- the final output of the R and D system, then these
duction of a new product or process. Taking this may be measured in terms of their contribution to
second meaning, innovation may be regarded as the specific goals, in relation to their cost. The mea-
ultimate aim of most applied research and experi- sures used in this type of assessment may vary
mental development and the success of any R and with the policy goal. For example, those reduc-
D programme may in principle be measured by the tions in mortality attributable to specific innova-
economic benefits arising from the innovations. tions might be a criterion for effectiveness of
When the goal of the innovative process is not an medical R and D, whilst increased destructive
economic one, a similar assessment mayneverthe- power of new weapon systems mightbe the criterion
less usually be made in terms of “effectiveness”. for military R and D.
The distinction between inventions and inno- It is not possible within the scope of this brief
vations is extremely important for policy for science paper to review methods of measuringthe effective-
and technology, as it is theoretically quite possible ness of military R and D. But it is important to
for an R and D laboratory or system to have alarge note that analysis of the cost-effectiveness of new
flow of inventions, or publications, or other infor- weapon systems, during and since the Second World
mation output, but for few innovations to result from War, played a major part in the development of
this flow. If the information output is regarded as techniques of operational research and systems
a goal for its own sake, then this may be a satis- analysis, which have since found much wider ap-
factory state of affairs. But this is rarely the case plications. Here the discussion concentrates pri-
with applied rese,arch and experimental development. marily on economically-oriented R and D and on
Consequently, it is only by measuring innovations “welfare” R and D.
directly or indirectly that the efficiency of the system For economically-oriented research, the ap-
in generating final output can be assessed. propriate criteria might sometimes be cost re-
This is not to say that the measurement of ductions in the case of process R and D, or contri-
intermediate output is unimportant. Some very butions to market growth and profitability in the
productive laboratories may be concerned entirely case of new product R and D. The type of measure-
with “intermediate” output, just as many plants are ment which is relevant will vary with the level of
in the chemical industry. But ultimately, measures decision-making. At the level of the firm a great

25
 deal of detailed information will be available, which to measure the socialbenefits arisingfrom the dif-
can be used both in ex-ante and ex-post project fusion of the innovation. In the case of the potato
evaluation, but cannot be used at a higher level of harvester, Grossfield was able to estimate the total
aggregation. In a market economy the main criterion savings arising from the difference between hand-
may well be profitability of innovations. At the picking and mechanical harvesting, whether these
level of the industry, or the government, it may be benefits accrued to the manufacturers, the farmers
possible to assess external and secondary benefits or the consumers. He was also able to demonstrate
arising from an innovation, or external costs, which and measure approximately the cost of the contri-
are disregarded at a lower level. The method of bution of the National Institute of Agricultural En-
assessment may also vary with the type of economy: gineering and the National Research Development
in a market economy the competitive position of the Council (NRDC) itself to the success of the inno-
firm will be a relevant dimension, but this may not vation. But both he and Griliches encountered
be so in a socialist economy. Just as with industrial difficulties in calculating the cost of “failures”, i.e.
productivity measurement, techniques which are parallel or abortive attempts to make similar inno-
appropriate in one firm or for one type of output vations and research in related fields which made
may not be relevant for another. an indirect contribution. These difficulties of
Unfortunately, very few studies have been pub- selecting, isolating and measuring the appropriate
lished relating to the project of programme evalu- “costs” and “benefits” are multiplied many times
ation of enterorise-level I
R and D. as it is actuallv over in the case of most industrial and agricultural
carried out. There are plenty of papers and books innovations. Researchers usually find that, although
prescribing ways in which it ought to be carried out occasional individual projects readily lend them-
but there is some evidence that these techniques are selves to cost-benefit analysis, most do not.
not very frequently applied in practice, (76) except This is probably one of the main reasons for
perhaps in the socialist countries, where in some the scarcity of practical examples of cost-benefit
industries they are systematically used. Work in studies and rate of return studies at enterprise level
progress at the Department of IndustrialEngineer- in market economies. Other major factors are
ing and Management Sciences at the Northwestern company security and a certain reluctance to subject
University will help to fill this gap for the USA. past experiences to impartial scrutiny. Difficulties
In the Soviet Union “in all planning, design and in quantifying costs and benefits are stressedvery
scientific research institutes and organizations strongly by Mansfield, who has contributed the
there are expert economic sub-divisions which, most important econometric studies in the fieldof
using a more or less uniform method, have to pro- industrial innovation:(76) (77)
vide a detailed justification of the economic effec-
“The productivity of industrial research and
tiveness of each project of an enterprise, of each
development is an extremely important vari-
unit or planned technological process”. (71) Eco-
able which is plagued by unusually difficult
nomic effectiveness is assessed according to a code
measurement problems. “(76)
laid down by the State Committee for the Co-
ordination of Scientific Research in 1963, but the One way of avoiding the problem of isolating Rand
detailed application of the code appears tovary con- D expenditures attributable to a particular project
siderably between different industries and different is to relate the whole of the R and D budget of a
types of project. Whilst these techniques are mainly firm to its technological progress. To calculate
applied at the stage of R and D project selection and marginal rates of return on total R and D invest-
evaluation, that is ex-ante, there is also some ex- ments for ten major firms and for ten manufacturing
post analysis. Apparently development costs are industries, Mansfield attempted to use Cobb-Douglas
often underestimated ex-ante, as in the market production functions(79) on alternative assumptions
economies, but in spite of this, Soviet scientists as to whether technological change is capital-
calculate very high overall rates of return to Rand embodied or disembodied. On the assumption of
D investments. (72) It is not entirely clear whether capital-embodied technological change he found
in making such calculations they include the costs very high rates of return to R and D investment
of generating intermediate output in the earlier for major chemical and petroleum firms but not for
stages of the process, or what type of simplifying the chemical industry.
assumptions are made about other contributions to The difficulties involved inusing Cobb-Douglas
technical progress. A thorough review of the Soviet production functions to estimate the “contribution”
methods has beenmade by R.W. Davies, M.J. Berry of various inputs are notorious. Many economists
and R. Amann. (73) reject altogether the validity of these techniques.(66)
Two much-quoted cost-benefit studies of impor- Mansfield himself stresses the severe limitations
tant individual innovations are those of Griliches(74) of his data and results(61) and states that “not much
and of Grossfield. (75) The frequency of their cita- policy significance should be attached to them”.
tion is a tribute both to their quality and to the ex- The principal finding of the inter-industry compari-
treme difficulty of assembling the relevant data for son of rates of return on Rand D investment is that
such analysis. One of the principal merits of both they were highest in the apparel and furniture in-
these studies was the care with which they attempted dustries in the United States up to 1960.

26
 INDIRECT MEASURES OF THE oftime-lags is animportant indicator of efficiency in
OUTPUT OF INNOVATIONS the whole R and D system.
Hufbauer(35) made a major contribution to the
Of greater interest are Mansfield’s studies of size theory of international trade, elaborating Posner’s
and growth of a firm in relation to innovation, timing “technological gap” concept by relating innovations
of innovations and rate of diffusion of innovations.(32) in synthetic materials to country export perfor-
He points out that, because of data limitations in mance. Vernon(37) and Hirsch(33) made further
calculating rates of return on R and D investment, important contributions, relating the technological
the effect of successful innovation on a firm’s growth gap theory to United States performance in world
rate is one useful alternative measure of R and D trade and the concept of the “product cycle”. Huf-
success. He found that the average effect of a bauer ‘s method was based on identification of
successful innovation in the steel and petroleum major product innovations and measurement of
industries was to raise a firm’s annualgrowth rate “imitation lags” to first production for each country
by 4 to 13 percentage points. Very striking dif- for each new product. These lags were weighted
ferences were also found between the growth rates by the relative importance of each material in the
of innovating firms and those of other firms of com- world economy, to give an “aggregate imitation
parable initial size. In the coal and petroleum in- lag”. Thus all the principal countries could be
dustries he found that the larger firms made a dis- ranked in order of their innovative and imitative
proportionately large share of the principal inno- success in this industry and this in turn could be
. vations in relation to their market share, but not in related to their export performance.
the steel industry. These estimates were made on the Another indirect method of measuring innova-
basis of direct identification and listing of important tion achievements is to use the assessment provided
innovations from technical journals and correspon- by the internationalmarket for licenses and techni-
dence with firms. In the course of his studies he was cal know-how. It is reasonable to assume that
also able to develop useful models designed to ex- such payments will flow mainly to the most success-
plain the diffusion of process innovations. ful innovating firms, industries and countries. The
Other economists have attempted to-relate in- inclusion of “know-how” transactions in the statis-
dustrial R and D expenditures and innovations to tics means that they do not refer only to that part
world market performance. It can be argued that, of R and D output which is patented. Their main
at least for capitalist countries, the world export deficiency is that they do not reflect those trans-
market provides a fairly severe objective test of actions which are conducted on a barter basis or
the innovative success of afirm’s (or anindustry’s) in secrecy. There are also difficulties in connexion
research and development activities for product with the transactions between parent and affiliated
innovations. If a firm is the first in the world to companies, which may take a great variety of dif-
produce a new product, it may enjoy an export ferent forms. Nevertheless, the regular statistics
position based on monopoly supply up to the point maintained by the Japanese Government since 1953
of imitation abroad. Even then it may still be able have demonstrated the value of these figures for
to keep an export lead by technical improvements policy purposes (Table 9). More recently most of
and the introduction of new models, through inten- the leading industrial countries have begun topub-
sive and efficient R and D programmes. Suchsuc- lish them. A recent paper for United Nations Con-
cessful innovations would find their reflection in ference on Trade and Development (UNCTAD) (8g)
exceptionally high shares of the world market for attempted touse these published figures to estimate
particular new products or product groups. world flows and to assess the transfer of technology
Posner(33) had postulated such an explanation to the developing countries through this market.
in theoretical terms and economic historians had This discussion of indirect measures of the
explained the German lead in the nineteenth century final output of R and D emphasizes once more the
chemical and optical instruments industries very difficulty of developing any standardized approach
much on these lines. Freeman(34) attempted to which can be applied across the whole spectrumof
relate world export performance in the plastics and R and D activities. However, one ingenious and
electronics industries to the flow of firm’s inven- original attempt has been made to develop just such
tions and innovations and to R and D expenditures. a method. Maestre(go) has suggested that an
His results indicated the importance of “lead-time” input-output matrix relating a country’s industry
in innovative success, of the threshold levels of to its R and D system could be constructed by a
expenditure which this implied, and some of the novel method of measuring research productivity.
cumulative advantages of large-scale R and D in This method would rank the “utility” of various
particular industries. types of research for each industry and would also
Grossfield’s calculations on the potato harvester take into account the upstream and downstream
confirmed the sensitivity of the “benefits” from R and flows of information within the research systemby
D to the length of lead-time in competitive markets. a separate matrix for the transactions within the
Development lead-times could thus be an important R and D system. Its successful application would
indicator of relative efficiency in R and D output. depend, however, not only on some measurement
Such calculations are also widely used in the Soviet of these difficult intermediate and “feedback”flows,
Union, (3$5) where it is accepted that measurement but also on a large number of peer judgements for

27
 every industry about the “research productivity” which may in turn suggest better methods of
relevant to that industry. Thus in the end the method measurement. By an altogether different route
comes back to dependence on the type of subjective sociologists have come up with interestinghypoth-
evaluationof “quality”and “importance”of research eses which could explain some inter-firm and inter-
output, which bedevils so many output measurement organization differences in research and innovative
techniques. performance. (88) The problems are so complex
Even by this method there is no real escape that they necessitate variety in methods of approach
from the laborious and difficult work of case studies and contributions from many disciplines. Solutions
and systems analysis, industry by industry and will not be found easily or quickly.
even project by project. Such detailed analysis
may ultimately yield the kind of data which could
orçamentação
be applied in the way Maestre suggests. But as he OUTPUT BUDGETING
recognizes, this is a long way off. Progress will
probably depend on fundamental research at amicro- In these circumstances a useful step towards a
level on information channels and information flows satisfactory system of measurement may often be
between individuals, projects and organizations. simple “output budgeting”. This has been de-
Most of the pioneering work in this important field veloped in response to the great difficulties of
has been done at centres of management studies or applying cost-benefit or input-output analysis in
operational research, notably at the Sloan School of many parts of the public sector. Williams has de-
Management (of the Massachusetts Institute of Tech- fined its purpose as follows:(loo)
nology) by D. G. Marquis, T. J. Allen and col-
“Unfortunately, thorough-going cost-benefit
leagues(gl), at Northwestern University Depart-
analysis is not immediately feasible in many
ment of Industrial Engineering and Management
important fields of social policy, because of
Sciences by A. H. Rubenstein et al. (g2), and at the
sheer lack of understanding as to what the rele-
Studiengruppe fiir Systemforschung (Heidelberg) by
vant dimensions of the output are, because of
H. Krauch, H. Rittel, W. Kunz and others. (88) In
difficulty in getting adequate data to measure
the Socialist countries the technique of operations
them even where they are understood suffi-
research has alsobeenused, notab1ybyG.A. Lakhtin
ciently, or because of the inadequacy of our
and I. Male&i, to analyse problems of optimizing
present evaluation techniques in attaching con-
output in research organizations. G. A. Lakhtin
vincing money-values to them. It is, however,
has suggested that the information service withina
possible to move part of the way towards com-
research organization is of great importance(g4),
prehensive cost-benefit analysis by tackling a
but much of the American work points to the crucial
rather more restricted task, that of settingup
significance of informal channels of communication.
an output budget for government activities. The
Of particular interest is Allen’s concept of “tech-
basic idea of an output budget is to relate all
nological gatekeepers”(g5) and Diana Crane’s ap-
cost items to broad functional objectives, by
plications of the gatekeeper concept to science. (g6)
constructing a framework within which it is
It is not possible within the scope of this brief
clear what resources are being devoted towards
paper to review this extremely important literature
what end and with what results. ”
systematically.
Whilst a great deal depends on the progress of Such an approach may often involve complete re-
this work in deepening our fundamental understand- arrangement of traditional forms of maintaining
ing of information systems, and creating new pos- departmental accounts so that all the inputs rele-
sibilities of measurement, meanwhile there is value vant to a particular policy aim may be analysed
in continuing attempts to generalize at the institute together. An illustration of this is shown in Table
level, industry level or country level, crude though 10 for such policy aims as protection of persons
our output measures may remain. Such attempts and treatment of offenders. It will be noted that
may very well yield important guidelines for policy. research expenditures are grouped with many other
For example, Ben-David’s secondary analysis of items as one of the inputs designed to achieve
Jewkes’ data on inventions and some of the OECD specific policy goals. This has important implica-
and National Institute of Economic and Social Re- tions for the reclassification of R and D input
search (NIESR) studies(87) suggest the value of an statistics by goals.
approach which relates measures of inventive or One of the merits of this type of analysis is
research output and input to measures of innovative that it may suggest areas of under-investment or
output. In the military field “Project Hindsight” over-investment in research and experimental de-
studied a large number of “technological events” velopment, which would not otherwise be readily
or sub-innovations and related them to the intro- apparent. For example, from Table 11 it is im-
duction of new weapons systems in terms of cost mediately evident that there is apparently little
and time. (88) This type of comparison may throw government research on treatment of offenders in
up contrasts and unexplained discrepancies between the United Kingdom. This may be justifiable, but
firms or industries, for example, differences in it is the kind of question which should be debated.
lead-times or in “coupling” of social organizations, Whereas the market mechanism may conceivably

28

--.-. -_.. -----.-~. ~-.--.----..-. .- --. _- ___..--- -..-

allocate resources efficiently in some branches of judgements to be made about research output or
industrial R and D, it will certainly not do so in this innovative output, which are essentially non-
area. It is generally accepted that there is a wide economic in character. For example, in the case
area of social policy where similar considerations of a completely new product or service, its intro-
apply. Krauch has pointed out forcibly(l61) that duction may involve considerations of taste, quality
there is a permanent tendency tounder-investment of life and values which cannot be easily expressed
in research in this social welfare area, whilst in economic terms. In the last resort the assess-
Nelson(l02) and other economists have suggested ment of “benefits” of all kinds depends on values.
that the same tendency may operate in relation to Man does not live by bread alone and economic
fundamental research. This tendency to under- growth is desirable not as an end in itself but as a
investment arises from the intangible and diffuse means to other ends. Consequently, it is quite
distribution of many of the social benefits arising legitimate to assert the primacy of non-economic
from this research, the long time lags often involved, values in relation to the output of science and
and the inability of some of the beneficiaries to fi- technology.
nance the expenditures. For this kind of reason, Bernal defined society’s investment in research
both in market and in planned economies, these are as “all inquiries directed to changing the future
areas of accepted social responsibility, usually at state of civilization”. (163) The kind of civiliza-
central government level. Political assessment of tion we want should determine the output of inno-
“benefits” is to some extent both inevitable andde- vations and the way in which we rate theirbenefits.
sirable. Consequently, the direction of investment in new
As we have seen, it is difficult enough in the science and technology must always be the subject
case of economically-oriented R and D to measure of political and ethical debate and choice. The
and relate costs to benefits. Even in cases where change in quality of civilization in the next genera-
good statistics are freely available, there are often tion is the final measure of the “output” of R and D.

NOTES

(70) W. R. Baker and W. H. Pound, “R and D Pro- (76) E. Mansfield, Industrial Research and Tech-
ject Selection: Where we Stand”, I. E. E. E. Trans- nological Innovation, W. W. Norton, New York,
actions on Engineering Management, Institute of 1968
Electrical and Electronics Engineers, 1964, and (77) E. Mansfield, The Economics of Technologi-
A. H. Rubenstein et al, projects listed in A direc- cal Change, W. W. Norton, New York, 1968
tory of Research on Research, Northwestern Uni-
versity, Illinois, 1965 (78) E. Mansfield, op. cit. (76), page 198

(71) Osnovnye Metodicheskiye Polozheniya po (79) E. Mansfield, op. cit. (76), Ch. 4
Opredeleniyu Ekondmicheskoi Effektivnosti (80) OECD, The Residual Factor in Economic
Narechno-Issledovatel’skoi Raboty, MOSCOW, 1964 Growth, Paris, 1964
V. S. Sominskii, Ekonomika Novykh Proizvodstv, (81) E. Mansfield, op. cit. (76), page 201 and
1965, page 40 page 80
(72) V. S. Sominskii, Ekonomicheskaya Gazeta,
1967, No. 10, page 7 (82) E. Mansfield, op. cit. (76), Chapters 5-9

V. M. Petrov, Ekonomicheskie Problemy Sodvuz- (83) M. Posner, “International Trade and Techni-
hestva Nauki i Proizvodstra, Leningrad, 1967, cal Change”, Oxford Economic Papers, October
Page 9 1961

(73) E. Zaleski, J. P. Kozlowski, H. Wienert, (84) C. Freeman, op. cit. (61)

R. W. Davies, M. J. Berry and R. Amann, Science (85) Puti Povysenija Effektivnosti Nauchnogo
Policy in the USSR, Part 5, Section G, pages 457-
Truda, Novosibirsk, 1966, Vol. 2 and op. cit. (73)
487, OECD, Paris, 1969
(86) G. Hufbauer, Synthetic Materials in World
(74) Z. Griliches, “Research Costs and Social
Trade, Duckworth, 1966
Returns: Hybrid Corn and Related Innovations”,
Journal of Political Economy, October 1958 (87) R. Vernon, “International Investment and
International Trade in the Product Cycle”,
(75) K. Grossfield and J. B. Heath, “The Benefit Quarterly Journal of Economics, May 1966
and Cost of Government Support for R and D: a
Case Study”, Economic Journal, September 1966, (88)
. I
S. Hirsch. “The United States Electronics
pages 537-549 Industry in International Trade”, National Insti-
tute Economic Review, No. 34, November 1965

29
(89) C.H.G. Oldham, C. Freeman and E. Turkcan, T. J. Allen “Communications in the R and D La-
The Transfer of Technology to Developing Coun- boratory” Technology Review 70, November 1967
tries, with Special Reference to Licensing and
(96) Diana Crane, “The Gatekeepers of Science”,
Know-how Agreements, UNCTAD, November 1967
American Sociologist, November 1967
(90) Claude Maestre, “Vers une Mesure des
Echanges Intersectoriels entre, la Recherche et (97) J. Ben-David, Fundamental Research,
l’Industrie”, Le Progres Scientifique, No. 102, OECD, Paris 1968
November 1966 OECD, Gaps in Technology, Paris 1969
(91) D. G. Marquis, Research Programme on
C. Freeman, op. cit. (61)
the Management of Science and Technology,
Massuchesetts Institute of Technology, 1968 (98) C. W. Sherwin and R. S. Isenson, First In-
terim Report on Project HINDSIGHT, Office ofthe
T. J. Allen “The Performance of Information Director of Defence Research, (ODDR) , Clearing-
Channels in the Transfer of Technology, Industrial
house for Scientific and Technological Information,
Management Review, 8.1966 AD 642400, 1967
D. G. Marquis and T. J. Allen “Communication- (99) T. Burns and G. M. Stalker, Management of
Patterns in Applied Technology”, American Psycho- Innovation, Tavistock, 1961
logist, 1966, pages 1052-1060
G. Gordon, “The Problem of Assessing Scientific
D. G. Marquis and W. H. Cruber, Factors in the a Potential Solution”,
Transfer of Technology, Massachusetts Institute of
Accomplishment: -IEEE
Transactions, Institute of Electrical and Elec-
Technology, 1969
tronics Engineers, December 1963
(92) A. H. Rubenstein, “Programme of Research N. Kaplan, “Some organizational factors affecting
on the Management of Research and Development”, creativity”, IRE Transactions, March 1960
Annual Reports 1966 et seq, Department of Indus-
trial Engineering and Management Sciences, (100) A. Williams, Output Budgeting and the Con-
Northwestern University, Illinois. and op. cit. (70) tribution of Micro-economics to Efficiency in
Government, HMSO, London, 1967
(93) H. Krauch, Studiengruppe fur systemsfor-
schung, Jahresberichte 1966 et seq, Heidelberg. (101) H. Krauch, Sqcial costs and benefits in
R and D, paper submitted to first Frascati Con-
(94) G. A. Lakhtin, “Operational Research in ference, OECD, 1963
Research Management”, Minerva, Summer 1968
(102) R. Nelson, “The Simole Economics of
G. A. Lakhtin, “Nauchno-issledovatel’skaya Basic Scientific Research”, -Journal of Political
Deyatel’nost i Materialovoe Proizvodstvo”, Economv. 1959
M’ ~~~-
Ekonomischeskie Nauki, January 1966
R. Nelson, “The Allocation of R and D Resources”,
K. Malecki, Popytka Otsenki Parametra Effektiv- in Ed. R. Tybout, The Economics of R and D,
nosti Nauchnyhh Issledovanie, (Polish-Soviet Ohio, 1965
Symposium), Academy of Sciences, Moscow 1967
(103) J.D. Bernal, op. cit. (9)
(95) T. J. Allen, Managing the Flow of Scientific
and Technical Information, Ph. D. Thesis, Massa-
chusetts Institute of Technology, 1966

30
 Chapter VI

SUMMARY AND CONCLUSIONS

mão de obra gastos

1. The rapid growth in the resources devoted 8. Numbers of papers and patents have often
to research and experimental development work has been collected and analysed independently of the
led to a growing demand for measures of the effici- corresponding input measures, whilst input data
ency with which these resources areutilized. This have usually been assembled without any attempt to
involves in one form or another, direct or indirect obtain any corresponding output indicators. It is
measurement of outputs in relation to inputs. often extremely difficult or impossible to relate
2. I~JL& measures of scientific and engineering one to the other. This is a grave deficiency of both
manpower and R and D expenditures have now be- series.
come official statistical series in many countries, 9. It would be valuable if Unesco and other
although there are still severe problems of defini- interested organizations could promote experi-
tion and international comparability. mental work in particular countries enabling a sys-
3. Output measures are still in the experi- tematic comparison of input and output indicators
mental stage and do not lend themselves readily to in particular sectors and industries. For example,
any standardization on a national or international respondents to R and D input surveys could be asked
scale. For example, there is no prospect of sub- also to enter information relating to publications,
stituting any “output” series for the “input” values patents, licensing and know-how income and inno-
now used in some national accounts systems. vations. Abstracting journals and scientific journals
4. The argument that output measurement is could also be asked to collect and publish syste-
intrinsically unattainable is rejected and the litera- matically data on the place and institution, where
ture is briefly reviewed to demonstrate some appli- research leading to a paper was carried out and the
cations of output measures. citizenship of the authors.
5. One major problem in attempting to measure 10. The final output of the R and D system is
output is the interdependence of the whole R and D a flow of innovations. These may be measured
system. If the final output of the system is con- either by making lists of important innovations or
sidered as a flow of innovations, then the inter- in a more sophisticated way by measuring the
mediate output of many parts of the system is a ‘benefits” or “effectiveness” of each innovation.
flow of new information. This information flows “Cost-benefit” analysis of innovations, although it
in many different directions and is used with vary- has been successfully applied in particular cases,
ing time-lags. Only a part of it is published. is however very time-consuming and sometimes
6. Limited success has been achieved inusing impossible because of incomplete information on
numbers of scientific papers and patent statistics the costs and benefits. A particular difficulty is
as indicators of intermediate output for parts of the to make allowance for all the “intermediate” output
R and D system. Important contributions to eco- of information which led up to the innovation and
nomics and sociology have been based on such for all those experiments which failed but which
measures. Nevertheless, great caution is neces- nevertheless by their negative results contributed
sary in applying these measures, especially in to ultimate success.
international comparisons. 11. One way of getting round these difficulties
7. The use of patent statistics and counts of is to relate innovations to total R and D budgets
scientific papers, although justifiable for specific rather than to particular project budgets. In this
limited purposes, must usually be heavily qualified way a “rate of return” may be calculated for invest-
because of quality variations. These are generally ment in R and D in particular organizations. But
less significant at higher levels of aggregation, but this still leaves unresolved the problem of infor-
they may also be important in international compari- mation flows derived from other parts of the R
sons, because of institutional factors. and D system. Measures of relative efficiency in

31
 innovative performance maybe derived from “lead- which improves the quality of life, such as medical
times”, from operational research techniques and research or research on the urban environment
case study methods. must often be rated by essentially political judge-
12. The “benefits” of innovations are often ments. This is quite legitimate, since economic
difficult to isolate by direct measurements, but may growth too is not an end in itself but only a means.
sometimes be measured indirectly by their contri- Both inputs and outputs of R and D should therefore
bution to the growth or export performance of a be related to the principal policy objectives which
firm or an industry. In the case of non-economic they are designed to serve. Such “output budgeting”
benefits, alternative criteria must be applied which techniques may identify areas of relatively low in-
will vary with the policy objective. Research puts or outputs.

32
 Table 1 The Flow of Ideas through the Stages of Research,
Invention and Development to Application

INPUT OUTPUT
Stage
Intangible Tangible Measurable Intangible Measurable

I 1. Scientific knowledge Scientists Men, man-hours A. New scientific knowl- Research papers
” Basic (old stock and output Technical aides Payrolls, current edge : hypotheses and memoranda ;
Research ” from I-A) Clerical aides 1 and deflated and theories formulas
(Intended 2. Scientific problems Laboratories Outlays, current B. New scientific prob-
output : and hunches Materials, and deflated lems and hunches
” Formulas ” ) (old stock and output fuel, power Outlay per man
C. New practical prob-
from I-B, II-B, and -
lems and ideas
III-B)

1
II 1. Scientific knowledge Scientists Men, man-hours A. Raw inventions :
” Inventive (old stock and output Non-scientist Payrolls, current technological recipes
Work ” from I-A) inventors and deflated a. Patented inventions a. Patent applica-
(Including 2. Technology Engineers b. Patentable inven- tions and patents
minor (old stock and output Technical aides tions, not patented b. Technological
improvements from II-A and III-A) Clerical aides but published papers and
but excluding c. Patentable inven- memoranda
3. Practical problems Laboratories Outlays, current
further tions, neither pat-
and ideas Materia’ls, and deflated
development ented nor published c.
(old stock and output fuel, power Outlay per man
of inventions) d. Non-patentable, in- d. Papers and
from I-C, II-C, III-C ventions, published
(Intended memoranda
and IV-A) e. Non-patentable
output :
” Sketches ’ ) inventions, not
published -
f. Minor improvements F: -
B. New scientific prob-
lems and hunches -
C. New practical prob-
lems and ideas

III 1. Scientific knowledge Scientists Men, man-hours A. Developed inventions : Blueprints and
’ Development (old stock and output Engineers Payrolls, current blueprints, specifi- specifications
Work ” from I-A) Technical aides and deflated cations, samples
(Intended 2. Technology Clerical aides
B. New scientific prob-
output : (old stock and output Laboratories Outlays, current lems and hunches
” Blueprints from III-A) Materials, and deflated
and Speci- C. New practical prob-
3. Practical problems fuel, power Outlay per man
fications “) lems and ideas
and ideas Pilot plants Investment
(old stock and output
from I-C, II-C, III-C
and IV-A)
4. Raw inventions and
improvements
(old stock and output
from II-A)

IV 1. Developed inventions Entrepreneurs A. New practical prob- New-type plant

‘New-type (output from III-A) Managers lems and ideas producing
Plant Financiers and a. novel products
2. Business acumen and
Construction” bankers b. better products
market forecasts
Builders and c. cheaper products
(Intended 3. Financial resources contractors t investment in
output :
4. Enterprise (venturing) Engineers new-type plant
’ New-type
plant “) Building materials
Machines and
tools

Source : F, Machlup. The Production and Distribution of Knowledge, Princeton, 1962.

34
Table 2 Schematic 7’able Showing Numbers of Authors of
Various Degrees of Productivity (in papers per lifetime)
and Numbers of Papers so Produced1

--
Papers/man Papers Remarks

1 100 100
2 25 50 . The 75 per cent of men
11.1 who are low scorers
3 33.3
produce one quarter of
4 6.2 25 all papers.
5 4 20
6 2.8 16.7
7 2 14.2
8 1.5 12.5
9 1.2 11.1
10 1 10 . . . . Subtotal: 10 men produce
10-11.1 1 lot more than 50 per cent of
all papers.
11.1 - 12.5 1 ll.l+
12.5 - 14.2 1 12.5 +
14.2 - 16.7 I 14.2 +
16.7 -20 1 16.7 +
20-25 1 20 t
25 -33.3 1 25 +
33.3 - 50 1 33.3 +
The top two men produce
50-100 1 50 + one-quarter of all papers.
Over 100 1 100+

Total 165 586 t

Average papers/man = 586/165 = 3.54

1. Table constructed on basis of exactly 100 men with a single

published paper. Other entries computed from Lotka’s law.

Source: Derek J. de Solla Price, Litlle Science, Big Science,

Columbia Univ. Press, 1763, p” 45.

35
 Table 3 Origin of Science Abstracts

Percentages of World Total

Country Share of GNP Share of Share of Share of
1964 rt6yls. Abstr. Chem. Abstr. Population
1965 1964

(i) Larger participants :

USA 32.8 31.6 28.5 5.9
USSR 15.6 15.6 20.7 7.0

I
.W. Germany 5.2 6.3 1.8
6.2
E. Germany 0.8 2.2 0.5

UK 4.8 13.6’ 6.7 1.6

France 4.5 6.3 4.5 1.4
Japan 3.6 7.8 7.3 2.9
Italy 2.6 3.4 2.7 1.5

Canada 2.2 1.1 2.0 0.6

India 2.2 1.8 2.2 14.4
Poland 1.6 1.5 2.9 0.9
Australia 1.1 0.5 1.2 0.3

Romania 1.0 0.6 0.9 0.5

Spain 0.9 0.2 0.4 1.0
Sweden 0.9 0.7 0.9 0.2
Netherlands 0.9 5.2l 0.8 0.4

Belgium 0.8 0.3 0.6 0.3

Czechoslovakia 0.7 0.9 1.6 0.4
Switzerland 0.7 1.0 1 .o 0.2
Hungary 0.5 0.5 1.0 0.3

Austria 0.4 0.2 0.5 0.2

Bulgaria 0.4 0.2 0.5 0.2

(ii) Smaller participants and non-participants : -

All other countries 15.8 0.8 4.6 57.5

1. Data known to be swollen because of one or more large international journals

published from this nation.

Source: Derek J. de Solla Price, ” Research on Research “, Journeys in Science:

Small Steps - Great Strides, Report of the 12th Airforce Office of Scientific
Research Science Seminar, University of New Mexico Press, 1967.

36

--- __--- --. _ ------_.I--.. .- _- -.-_-___

Table 4 Percentage of women on the prolessional staffs o/higher educational
institutions in 1947 compared with percentage of articles in professional
journals contributed by women in 1940, and 1960.

1947 1940% 1960+

Discipline Percentage Percentage Sample Percentage Sample

of Women of Articles Size of Articles Size
in Field by Women by Women
Physics-mathematics 21.1 11 (247) 7 (1270)
Chemistry 45.3 26 (206) 40 (320)
Biology 47.8 26 (384) 36 (565)
Geology 23.4 9 (186) 25 (315)
Engineering 10.5 4 (868) 9 (1877)
Agriculture 23.0 14 (156) 21 (409)
Veterinary medicine 22.3 6 (274) 16 (331)
Medicine 48.0 24 (203) 39 (549)
Physical culture 27.6 9 (127) 13 (115)
Social-political and
philosophy 31.0 8 (131) 10 (180)
Economics 16.3 9 (168) 11 (447)
History 30.1 21 (154) 14 (188)
Geography 28.7 9 (105) 12 (125)
Philology 67.7 24 (134) 21 (227)
Arts 34.0 11 (216) 37 (207)
Law 12.7 6 (121) 10 (203)
Pedagogy 40.0 22 (294) 26 (639)

Total 35.0 16 (3974) 19 (7967)

Source: Norton T. Dodge, Women in the Soviet Economy: their ro^le in economic. scientific and
technical development, Johns Hopkins Press, Baltimore, 1966, page 228.

37
 Table 5 Distribution of Research Publications and their Origin

Total Research Letters Technical Symposia Review Case Others

no. of papers to editor reports and and reports
Institution publications and short conference informative
communi- papers articles
cations

Universities1 3816 2492 613 6 40 353 287 25

CSIR* 766 537 115 22 23 69 -
Medical laboratories
and hospitals 1 023 543 108 - 203 157 12
Agricultural
laboratories 1187 764 291 4 1 123 4
Industrial and other
laboratories 3 1545 957 277 43 83 177 3
Other institutions,
individuals 1122 485 99 5 7 460 60 6
From abroad 345 170 20 6 34 102 13

Total 9804 5948 1 523 86 193 1 487 524 43

1. Including medical, engineeringand agricultural colleges.

2. Including research association laboratories supported by CSIR.
3. Including research organizations of all ministries not covered by the above groups, private maintained laboratories,
etc.

Source: B.V. Rangarao, ” Scientific Research in India : an Analysis of Publications “, journal o/ Scientific and
Industrial Research, 1967, Vol. 26, No. 4.

38

-..
-. --_l_---l ..-. - _...... -.---- --~I .___
Table 6 Institutions from which more than 100 Papers were Published during the Year under Analysis

Total no. No. in No. of disciplines

foreign in which contribution
journals is
>lO 10-5 <5

University of Calcutta l 279 50 a 3 1

Agra University 260 10 5 1 4

Indian Agricultural
Research Institute 227 12 3 1
Banaras Hindu
University 226 22 6 3 2
Delhi University 206 58 6 1 4
Indian Institute of
Science 194 26
Rajasthan University 177 32
University of Madras 2 174 16
University of Bombay 162 7
Panjab University 160 la
Andhra University 154 20
Atomic Energy
Establishment 133 26
Kerala University 128 4
University of Allahabad 110 49
University of Lucknow 110 15
Tata Institute of
Fundamental Research 107 63 2 4
Vikram University 107 2 3 5

All Institutions (17) 2 914 430

1. Excluding Saha Institute of Nuclear Physics.

2. Excluding Agricultural College and Research Institute, Coimbatore.

Source: B.V. Rangarao, “Scientific Research in India: an Analysis of Publications “, Journal of

Scientijic and industrial Research. 1967, Vol. 26, No. 4.

39
 Table 7 Number of Discoveries in the Medical Sciences by Nations, 1800-1926

Year U.S.A. England France Germany Other Unknown Total

1800-09 2 a 9 5 L 1 27
1810-19 3 14 19 6 2 3 47
1820-29 1 12 26 12 5 1 57
1830-39 4 20 18 25 3 1 71
1840-49 6 14 13 28 7 - 68
1850-59 7 12 11 32 4 3 69
1860-69 5 5 10 33 7 2 62
1870-79 5 7 7 37 6 1 63
1880-89 la 12 19 74 19 5 147
1890-99 26 13 18 44 24 11 136
1900-09 28 18 13 61 20 a 148
1910-19 40 13 a 20 11 7 99
1920-26 27 3 3 7 2 2 44

Source : J ~ Ben-David, ” Scientific Productivity and Academic Organitation in 19th Century Medicine ” American
Sociological Review, December, 1960.

Table 8 Patents granted to Foreigners as a Percentage of Total Patents, 1957-61

Country %

Belgium 85.55
Canada 94.65
Ceylon 83.39
Czechoslovakia 12.78
Federal Republic of Germany 37.14
Finland 78.07
France 59.36
India 89.38
Ireland 96.51
Italy 62.85
Pakistan 95.75
Poland 37.94
Sweden 69.30
Switzerland 64.80
Turkey 91.73
U.S.S.R. 00.72
U.S.A. 15.72
Yugoslavia 60.50

Source: United Nations, The Role of Patents in the Transjer o/

Technology to Developing Countries, (Dept. of Economic &
Social Affairs), New York, 1964, Annex E.

40

-_-..- _ _ ._...._
_^_..~.
..._.-_ .-_._ _.__-.. _.... ~--- _ .-.___. - . -- ---_.- _.
Table 9 Approved Enterprise-to-Enterprise Agreements for the Transfer of Technology in lapan

A-type contracts1 B-type contracts 2 Total payments

for contracts
Year (A + B)
Payments Payments
Number Number us pooo
$‘OOO Million ye” Jl’OOO Million ye”

1950 27 501 .l 180.4 49 2 133.6 768.1 2 634.7

1951 101 4 841.9 1 743.1 87 1 854.2 667.5 6 696.1
1952 133 8 156.1 2 936.2 110 1 779.7 640.7 9 935.8
1953 103 11 466.9 4 128.1 133 2 411.9 868.3 13 878.8
1954 82 13 011.1 4 684.0 131 2 782.2 1 001.6 15 793.3
1955 72 17 963.1 6 466.7 113 2 033.6 731 .l 20 222.3

1956 144 28 416.9 10 230.1 167 4 922.8 1 772.2 33 339.7

1957 118 39 438.9 14 198.0 136 3 181.1 1 145.2 42 620.0
1958 90 44 370.0 15 973.2 152 3 479.4 1 252.7 47 849.4
1959 153 54 196.1 19 510.6 225 7 659.4 2 757.4 61 815.5
1960 327 83 466.1 30 047.8 261 11 421.4 4 111.7 94 887.5
1961 320 98 229.7 35 362.7 281 17 441.4 6 278.8 115 671.1
1962 328 104 491.1 37 616.8 429 10 474.4 3 770.8 114 965.5
1963 564 134 380.0 48 376.8 573 11 464.7 4 127.3 145 844.7
1964 500 139 000.0 50.000.0 541 16 402.0 5 900.0 155 402.0
1965 472 486 160 000.0

Total 3 534 642 454.1 281 454.5 3 874 83 042.5 35 793.5 1 040 899.1

1. An A-type technological assistance contract is a” agreement whose duration ot period of royalty payment is one year
ot more and whose royalty is to be paid in foreign currency. These are governed by the Foreign Investment Law.
2. A B-type technological assistance contract is a” agreement whose duration or period of royalty payment is less than
one year ot whose royalty is to be paid in Japanese currency. These come under the Foreign Trade and Exchange
Control Law.

Source: Table reproduced from: C. H. G. Oldham, C. Freeman, E. Tutkcan, The Transfer of Technology to Developing
Countries with Special Reference to Licensing and Know-how Agreements, United Nations Conference on Trade
and Development (UNCTAD), November 1767, page 21”
 UNESCO PUBLICATIONS: NATIONAL DISTRIBUTORS

Afghanistan Panusai, Press Department, Royal Afghan Ministry of Education, KABUL.

Albania N. Sh. Botirneve Naim Fasheri. T~ANA.
Alpe*ia Institut pddagogique national, II, rue Ali-Haddad (ex-rue 2aBtch.s). ALGER.
Argentina Editorial Sudamericana S.A.. Humberto I n.” 54s. T.E. 30. 7518. BUENOS AIRES.
Australia Publications : Educational Supplies Pty. Ltd., Box 33, Post Office, Brookvale eroo, N.S.W. Perio-
dicnls : Dominic Pty. Ltd., Box 33. Post Office, Brookvale aroo, N.S.W. Sub-agent: United Nations
Association of Australia, Victorian Division, 4th Floor, Askew House, 364 Lonsdsle St., MELBOURNE
(Victoris) 3000.
Austria Verlag Georg Fromrne & Co., Spengergasse 39. Wrtt~ 5.
Belgium All publications: Editions ‘Labor’. 34s. rue Royale, BRUXELLE~ 3 ; N.V. Standaard Wetenschsppelijke
Uitgeverlj, Belgiglei, ANTWBRP~ I.
For ‘The Courier’ and slides only: Jean De Lannoy,, I I 2. rue du TrBne, Bguxet.~ss 5,
Comisi6n National Boliviana de la Uneaco, Mimsterio de Educaci6n y Cultura. casills de corre~ “0 4107.
LA Paz. Libreria Universitaria. Universidad San Francisco Xavier, apartsdo sts, SVCRH.
Funda$o Getulio Vargas, caixa postal 4081-ZC-05. Rro DH JANE~RO(GUANABARA).
Raznoiznos, I Tsar Assert, SOFIJA.
Librsirie Albert Portail, 14, avenue Boulloche. PHNOM-PGNH.
Papeterie modeme, Mailer et Cie, B.P. 495, YAOUNDB.
The Queen’s Printer, OTTAWA (Ont.).
Lake House Bookshop, Sir Chittsnrpslam Gardiner Mawata, P.O. Box 244. COLOMBO z.
Ah publicntiom: Editorial Universitsria S.A., casilla reuse, SANTIAGO.
Fm ‘The Courier’: Cotnisidn National de la Unesce, Mat-Iver 764, dpt”. 63, S~~rt~oo.
Chins The World Book CO. Ltd., 99 Chungkiig South Road, section I, TAIPM (Taiwan/Formosa).
Colombia Libreria Buchhols ,Galet+a, avenids Jimenez de Quesada 8-40. BOGOTA; EdicionesTercer Mundo, spsrtsdo
aereo 4817, BOOOTA; Distrilibros Ltda., Pio Alvonso Garcia csrrers 4.s. n.0~: 36-119 y 36-rs5, CARTI\~ENA
J. Germ&~ Rodriguez H., oficina 101, Edificio Bsnco de Bogota’, apartado “scions1 83, GIRARDOT.
Cundinamarca; Lihreria Universitarla. Univenidad Pedag6gica de Colombia, TUNJA.
Congo (Dun. Rep. of) La Librairie. Institut politique congolais, B.P. 2307, KINSHASA.
Costa Rica Libreria Trejos, S.A., apartado 1313. SAN Jose. T&forms z&5 y 3200.
For ‘The COURT’: CURIOSVslerinSa’en~& CO. Ltds., ‘ElPalacio de la Revistas’, apartsde 1924. SAN Josd.
Cuba Instituto de1 Libm, Departamento Econ&nico, Err&a y San Pedro, Cerre, LA HAB~A.
CYPrt” ‘MAM’. Archbishop Makarios 3rd Avenue. P.O. Box 1722. NICOSIA.
Czechoslovakia SNTL, Spsle”s 51. PRAW r (Permanent dzsplay); Zahranicni literatura, I I Soukenicka, PRAHA 1.
Denmark Ejnar Munksgaard Ltd., 6 Nerregade. t 165 KBBENHAVN K.
Dominican Republic Librerla Dominicana, Mercedes 49. apartado de correos 656, SANTO DOMINGO.
Ecuador Csss de 1s Culturs Ecusterisns, Nuclei de1 G~ayas, Pedro Moncayo y 9 de Octubre, casilla de corre~
354s GUAYAQUIL.
El Salvador Librerla Culture1 Salvadorena, S.A., Editicio San Martin. 6.. calle Oriente n.0 t 18, SAN SALVADOR.
EF’E;$ National Commission for Unesco, P.O. Box 2996, A~ors ABABA.
Akateeminen Kirjakauppa, s Keskuskatu. Hasmrtr.
Fle”l2.s Librai& de l’U”es~~, place de Fontenoy, 75 P-7’. CCP rs598-48.
French West Indies Libnirie J. Becage. sue Lsroir. BP. ~8. FORT-D&FRAN(g (Msrtinique).
G-“Y (Fed$& R. Oldenbourg Verlsg. Unesco-Vertrieb f0r Deutschland, Rosenhehnerstrssse 145. MONCHEN 8.
Methodist Book Depot Ltd., Atlantic House. Connncrcisl Street, P.O. Box roe. CAPE COAST.
G*.XCe Librairie H. Ksulhnann, 28, rue du Stade, ATHENAI; Libraitie Elefthereudakis. Nikkis 4. ATHENAI.
Guatemala Comisi6n Nacionsl de la Unesco. 6.’ Calle 9.17, mns I, GUATEMALA.
Haiti Librsirie ‘Ale Csrnvelle’, 36, rue Roux, B.P. I I I, PORT-AU-PRINCE.
“=wfgr; Swindon Book Co.. 13-15 Lock Road, Kow~oo~.
Akademiai K6nyvesbolt V&i u a% BUDAP~ V.
A.K.V. Konyvtavosok Boltja, Ndpk6zdvsss&g utja 16 BUDAPEST vt.
Iceland Snaebjiim Jonsson & Co. H. F.. Hafnarstracti 9, R~~~~JAvIK.
India Orient Longmans Ltd.: Nicol Road, Ballard Estate BoMBAv I; 17 Chittsmnjan Avenue, CALCUTTA 13;
36s Mount Road, MADRAS ‘2; 3/5 Asaf Ali Road, NEW DELHI I.
Sub-depots: Oxford Book& Stationery Co., r7Park Street, CALCITITA 16; and Scindia House, NEW DELHI.
India” National Commission for Unesce. stt. : The Librarian. Ministry of Education, ‘C’ Wing, Reem
No. 214. Shastri Bhswan. NEW DELHI 1.
Indonesia Indira P.T.. Dil. Dr. Sam Ratulanaic 37, DJAKARTA.
Iran Cornntissio” n&ion& irsnienne pour I’Unesco. svenue du Mu&e. TPH!&AN.
Iraq McKenzie’s Bookshop, Al-Rsshid Street BAGHDAD ; University Bookstore, University of Bsghdsd. P.0
BOX 75. BAGHDAD.
Ireland The National Press, e Wellington Road, Ballsbridge, DUBLIN 4.
Isteel Emanuel Brown fornterly Blunrstein’s Bookstores: 35 Allenhy Road and 48 Nshlst Benjamin Street,
TEL AVIV.
Libreris Connnissionsria Sansoni S.p.A., via Lsmsnn~rs 45. cssells postale 55z,501st FIRENZE; Libreris
Intemssionale Rizsoli, Gal&a Colonna, Lsrgo Chigi, ROMA; Libreria Zanichelli, Piazza Gslvsni t/h,
BOLOGNA: Hoepli, via Ulrico Hoepli 5. MILANO; Librsirie frsrwise, piazza Castello 9, TORINO:
Diffusione Edizioni Anglo-Americane, via Lima 28, 00198, ROMA.
Ivory Coast Centre d’edition et de difhts’on africaines, bolte postale 4541. ABIDJAN PLATEAU.
Jamaica Ssngster’s Book Stores Ltd.. P.O. Box 366, mr Water Lane, KINGSTON.
JsPs” Msruren Co. Ltd., P.O. Box 5050, Tokyo International, TOKYO.
Jordan Joseph I. Bahous & Co., Dar-ul-Kutub, Salt Road, P.O. Box 66, AMMAN.
ESA Bookshop, P.O. Box 30167. NNROSI.
:zz: Korean National Commission for Unesco, P.O. Box Central 64. SEOUL.
Kuwait The Kuwait Bookshop Co. Ltd., P.O. Box 2942. KUWAIT.
Lebanon Librsiries Antoine, A. Naufal et Freres, B.P. 656. BEYROUTH.
Liberia Cole& Ysncy Bookshops Ltd., P.O. Box 286. MONROVIA.
Libya Agency for Development of Publication and Distribution, P.O. Box 261, TRIPOLI.
Liechtenstein Eurecs” Trust Reg., P.O.B. 5, SC~AAN.
Luxembourg Librairie Psul Bruck. sf GrsndeyRue. LLJXSMBOVRO.
Madagascar &&wbeb;gxtr: Comrmsston natmnale de !a Rdpublique malgache, Minis&e de I’Education natto”:.!c
Fm ‘The &tier’: Service des ocuvres post et p&i-scolaires, Mini&m de I’Education nation&
TANANARIVE.
 Malaysia Federal Publications Sdn. Bhd., Balai Ber~ta. 3r J&n Riong, KUALA LUMPUR.
Mali Librairie populalre du Mali, B.P. 28. BAMAKO.
Malta Sapienza’s Library 26 Kingsway. VALLETTA.
Mauritius Nalanda Co. Ltd., 30 Bourbon Street, PORT-Lows.
Mexico Editorial Hermes, Ignacio Mariscal41. MEXICO D.F.
MO”8CO British Libraw. 10. boulevard des Moulins. MONTE-CARLO.
MOIOCCO AlI public&k> “Lihrarie ‘AUX belles im&s’, 281 avenue Mohammed V, RABAT (CCP 68.74). For ‘The
Courier’ (for teachers) : Commission nationale marocaine pour I’Unesco. 20. Zenkat Mourabitine. RABAT
(CCP 307-63).
Mozambique Salema & Carvalho Ltda.. caixa postal 192. BEIRA.
Netherlands N.V. Marthus Nijhoff. Lange Voorhout, 9 ‘~GRAVENHAOB.
Netherlands Antilles G. C. T. Van Dorp & Co. (Ned. Ant.) N. V., WILLEMSTAD (Cura~o. N.A.).
New Caledonia Reprex, avenue de la Victoire. lmmeuble Painbouc. NOUMCA.
New Zealand Government Printing Office, Govetnment Bookshops: Rutland Street, P.O. Box 5344, AUCKLAND; 130
Oxford Terrace. P.O. Box 1721. CHRISTCHURCH; Alma Street, P.O. Box 857, HAMILTON ; Ptinces Street,
P.O. Box 1.04. DUNEDIN; Mulgrave Street, Private Bag, WELLINGTON.
Nicaragua Libreria Cultural Nicaragiiense, calle 15 de Septiembre y avenida Bolivar ,apartado “.o 807. MANAGUA.
Norway Allpub/icicotionr: A. S. BokhjGmet, Akersgt. 41, OSLO 1.
For ‘The Courin’: A. S. Narvesens Litteratuqeneste. Box 6125, OSLO 6.
Pakistan The West-Pak Publishing Co. Ltd., Unesco Publications House. P.O. Box 374, G.P.O.. LAHORG.
Showmoms: Urdu Bazaar, LAHORB, and 57-58 Mutee Highway, G/6-1. ISLAMABAD. Pakistan Publications
Bookshop : Satwar Road, RAWAI.PlNol ; Parlbagh, DACCA.
Paragg Melcbor Garcia, Eligio Ayala 1650, kWNCI6N.
Distribuidora INCA S.A.. Emilio Althaus 470. Lime. apartado 3115. LIMA.
Philippines The Modern Book Co., 928 Rizal Avenue, P.O. Box 632, MANILA.
Poland Osrodek Rozpowzechniania Wydawnictw Naukowych PAN, Palac Kultury t Nauki. Warszawa.
Pcrtugal Dins & Andrade Ltda., Libmria Portugal, rus do Carmo 70, LISBOA.
Puerto Rico Spanish English Publications, Eleanor Roosevelt I 15, apartado 19r2, HATO REY.
Southern Rhodesia Textbook Sales (PVT) Ltd. 67 Union Avenue, SALISBURY.
Romania Cartimex, P.O. Box 134-r35. 126 c&a Victonci, BUCURESTI. (Telex: 226.)
Senegal La Maison du Livre. I 3. .wen”e Roume, B.P. 20-60. DAKAR.
Singapore Federal Publications Sdn Bhd., Times House, River Valley Road, SINGAPORE 9.
South Africa Van Schaik’s Bookstore (Pty.) Ltd., Libri Building, Church Street, P.O. Box 724, PRBTORIA.
Spain All publications: Libreria Cientifica Medinaceli, Duque de Medinaceli 4, MADRID 14.
For ‘The Courier’: Ediciones Ibemameticanas S.A.. calle de Onate 15, MADRID; Ediciones Liber. apartado
de correos 17, ONDARROA (Vizcaya).
Suds” AI Bashir Bookshoo. P.O. Box I I 18. KHARTOUM.
Sweden Al1publicotim.s: AjB. C. E. Fritzes i<“ngl. Hovbokhandel. Fredsgatan t, STOCKHOLM 16.
For ‘The Courim’: The United Nations Association of Sweden, Vasagatan 15-17. STOCKHOLM C.
Switzerland Europa Verlag, Rtimistnase 5. ZL~ICH; Librairie Payot, 6. rue Grenus 121 I GEN~VE II.
Syria Libra& Sayegh, Immeuble Diab. rue du Parlement, B.P. 704. DAMAS.
Tanzania Dar es Salaam Bookshop, P.O. Box 9030. DAR FSSALAAM.
Thailand Suksapan Panit, Mansion 9, Rajdamnem Avenue, BANGKOK.
Turkey Librnitie Hachette, 469 Istiklal Caddesi, Beyoglu, ISTANBUL.
Uganda Uganda Bookshop. P.O. Box 145, KAMPALA.
U.S.S.R. Mezhdunarodnaja Kniga. MOSKVA G-zoo.
United Arab Republic Libra& Kasr El Nil, 38. rue Kasr El Nil, Ls CAERE, Sub-depot: Ls Renaissance d’dgypte, 9 SR. Adly
Pasha, CAIRO (Egypt).
United Kingdom H.M. Stationery Office. P.O. Box 569. LONDON S.E.r ; Government bookshops: London, Belfast.
Birmingham, Cardiff. Edinburgh. Manchester.
U.S.A. UnescoPublicationa Center. P.O. Box 433, New York. N.Y. 10016.
Um3”aY Editorial Losads Uruguaya. S.A../Libreria Losada, Maldonado ro92/Colon1a 1340, MONTEVIDEO.
Venezuela Libreria Historia, Monjas a Padre Sierra. Edificio Oeste 2. n. o 6 (frente al Capitolio). apartado de correos
73.20, CARACAS.
Republic of V&-Nom Libnirie-Papeterie Xuan-Thu, 185-193 rue Tu-Do. B.P. 283. SAIGON.
Yuu0slav1a Juaoslovensk;l KnJiga, Terpzije 27. BEOGRAD.Druvnn Zaluzbs Slorenijc Matni Trg. 26 LJUBLJANA.