Approaches in the Area of

Measurement Uncertainties

Introduction
Measurements
Uncertainties
Operations on measurands
and corresponding uncertainties

62.15 Kg

62.40 Kg
2

Introduction (cont.)
Measurements : difference = 62.40 - 62.15 = + 0.25 Kg
??? Can a jogging (or gym) session have such an
effect (i.e. putting weight on ! )???
Is that realistic (or even possible) ?
Effect of Uncertainties in everyday life !
Digital readings are misleading
Sources of uncertainties to be identified
What is the real effect of making the same (or a similar)
measurement twiceand then making calculations on
the values found ?

The world of Radio measurements ...

History
In the days before the R&TTE Directive and the CEPT
policies
Equipment had to be type-approved in the various
European Countries
Manufacturers would travel across Europe
with equipment from Lab to Lab
and have their equipment measured

Panic (very) different results were often found

Can pass or fail be a random variable !
4

An initial Methodology had been developed (before the 1990s)

Detailed description of the methods of measurement
Equipment measured in the same way in the various
European Countries
Results to be presented in the same way

Including the characteristics of equipment to be used in

the measurements
Requirements concerning the type of equipment to be used
(instrumentation)
Requirements concerning the minimal performance
(of instrumentation)
5

Drawbacks of this old way of handling uncertainties

Lack of freedom for the industry
In terms of type of equipment to be used
Prevents innovation
Benefits from having better equipment in one subset of the test
set-up lost

Performance in terms of uncertainty could not be part of a

commercial relationship
Restrictive in terms of competition

How to relate the required performance in terms of

equipment used for the measurement into parameters to be
used for :
Systems deployment (operators ( need also to take into
account dispersion in the characteristic of equipment )) ?
Compatibility and Sharing studies (ITU, CEPT) ?
6

Targets for a New Approach (i.e. TR 100 028)

Allow for a continued enhancement of the performance
in term of uncertainties
Allow for positive effects of improvements in terms of
performance of instrumentation
New concept : uncertainty budget.
Decouple new editions of Standards from the evolution
of Technology in the area of Instrumentation
(avoids having to update standards just to keep up with
progress in the area of instrumentation ! )

Enhanced transparency
Realistic uncertainty figures support, in particular :
the optimization of system deployment
avoiding discrepancies between results obtained by various
partners (e.g. under the R&TTE regime)

Supports enhancement of the uncertainty figures.

7

The New Approach (in accordance with TR 100 028 V 1.4.1)

Theoretical approach for the evaluation of
uncertainties
Evaluation of the measurement uncertainty for each
measurement
Agree on pass/fail criteria
Define maximum values for the uncertainty (e.g. in
Harmonized Standards under the R&TTE Directive)
Prepare supporting documentation (e.g. TRs, forms)

The New Approach (in accordance with TR 100 028 V 1.4.1)

(STEP 1)
Theoretical approach for the evaluation of
uncertainties
Evaluation of the measurement uncertainty for each
measurement
Agree on pass/fail criteria
Define maximum values for the uncertainty (e.g. in
Harmonized Standards under the R&TTE Directive)
Prepare supporting documentation (e.g. TRs, forms)

Statistical ?

Worst case ?

First choice :
Worst case
statistical (usage of random variables)

Criteria
Is a worst case approach possible at all ???
Impossible if non-finite contributions
Gaussians can often be found

Is it representative and realistic ?

10

 Mapping of the measurement set-up into :

random variables (sources of uncertainty =
contributions) and
operations between these random variables

Evaluate (or define) the probabilistic properties of

each of the above random variables (e.g. rectangular
distribution)
Find out the mathematical properties of the
measurement set-up, or intermediate steps (based
upon the properties of the distributions mapped to the
various uncertainty sources).
11

Examples of sources of uncertainties

Related to instrumentation :
Levels of signal generators
Readings (e.g. value of a frequency)

Related to the environment or to the EUT (Equipment Under

Test):
test conditions (e.g. effect of temperature)
Relations between antenna polar diagrams and EUT

Related to parts of the measurement set-up :

cables and hardware (e.g. attenuators)
mismatch ...

12

Example of combinations
The table found in TR 100 028, part 2 Annex D
section D.3.12 provides for usual operations :
the resulting distributions
the values of the means and standard deviations ...

13

 and also support ,in particular,

for conversions between linear terms and dBs ...

14

 Possible simplifications
Differentiation
1st order approximations

Usage of sub-blocs (sub-systems)

Useful in the case where specific parts of a
measurement system are often used together (e.g.
automated test systems)
Useful when different units are used in different parts of
the measurement set-up (e.g. Volts and dBs).
15

The basic trick about standard deviations

The table in D.3.12 shows, in particular, that :
a number of usual operations translate into simple operations on
standard deviations

As a result, one possible approach is the following :

list all contributions and characterize the corresponding standard
deviations
combine all standard deviations in accordance with the rules given
in the table (which provides a combined uncertainty, given by its
standard deviation)
Invoke the Central Limit Theorem which provides for a
Gaussian having the standard deviation found above
IN ORDER TO AVOID HAVING TO
Perform the direct calculation .
16

Result :
The distribution corresponding to the combined uncertainty
(i.e. the probability of error r)

Gaussian curves

1
r 2
p(r )
exp ( 2 )
2
2

Standard deviation

17

Changing Confidence Levels

p( x )

The standard deviation corresponds

to 68 % (Normal distribution), i.e.
the probability of the error being
within the two bounds (plus and
minus sigma).
2
x
1 exp 2
p( x)

2
2

( )

Expansion factor (k) : the factor allowing to change

from one confidence level to another
k=2 is an usual value ; provides approximately 95 %
in the case of Gaussian (Normal) distributions
i.e. the probability of the error being within the two
bounds (plus and minus two sigma).

18

The New Approach (in accordance with TR 100 028 V 1.4.1)

(STEP 2)
Theoretical approach for the evaluation of
uncertainties
Evaluation of the measurement uncertainty for each
measurement
Agree on pass/fail criteria
Define maximum values for the uncertainty (e.g. in
Harmonized Standards under the R&TTE Directive)
Prepare supporting documentation (e.g. TRs, forms)

19

 List of the contributions (complete not forgetting

cables, mismatch, etc.)
Define units, shape and characteristics for each
distribution
Find any natural assembly (i.e. definition of sub-sets)
Possible help : ETSI TRs (including spreadsheets)

find the right uncertainty value(s)

(for the right confidence level or factor k )!

20

The New Approach (in accordance with TR 100 028 V 1.3.1)

(STEP 3)
Theoretical approach for the evaluation of
uncertainties
Evaluation of the measurement uncertainty for each
measurement
Agree on pass/fail criteria
Define maximum values for the uncertainty (e.g. in
Harmonized Standards under the R&TTE Directive)
Prepare supporting documentation (e.g. TRs, forms)

21

Pass or fail ??? (1)

Found in the appropriate standard :
limit value for each parameter
maximum acceptable measurement uncertainty for each parameter

The measurement provides :

the measured value
the estimation of the uncertainty
for a certain confidence level (i.e. 95%) or
for a certain expansion factor k (i.e. k= 1,96 or 2,00)

The shared risk approach

measurement uncertainty stated together with measured value
directly compare value measured to limit (found in the standard)
measurement uncertainty has to be better than the maximum acceptable
uncertainty (found in the standard).

(Usually) Harmonized Standards published by ETSI under the R&TTE

Directive clearly state the pass/fail criteria (i.e. the shared risk
approach).
22

Pass or fail ??? (2)

Other approaches :
Make the limit harder to pass by the amount of the uncertainty (H)
Make the limit easier to pass by the amount of the uncertainty (E)

(E)

(H)

Limit

Uncertainty

Measured
value
Shared risk
PASS

FAIL

PASS

Target
23

Pass or fail ??? (3)

Why prefer the shared risk :

Comparing directly the result of the measurement to the limit is
a natural approach was used before taking uncertainties into
account
Used for a long time and widely accepted
The only approach where the interest of all parties is to
improve the uncertainty of the measurements
The maximum acceptable uncertainty is given in the standards,
therefore, the risk is known by all parties beforehand and can
be taken into account (in the design, planning or sharing)
Has been used safely by the industry (under both
the type approval and
the R&TTE regime).

24

Pass or fail ??? (4)

How can a manufacturer avoid the statistical risks :

Target design = 1 maximum acceptable measurement
uncertainty (or more) away from the limit
Dispersion of characteristics of equipment (due to the production
line) to be also taken into account

In all the approaches, there is always a possibility of failing by a

fraction of a dB if the margin indicated above is not taken into
account !

25

Pass or fail ??? (5)

(Other) Problems with approaches other than the shared risk
The measured value has to be or may be one (maximum acceptable)
measurement uncertainty away from the stated limit.
Therefore, the actual value of that parameter may be TWO (maximum
acceptable) measurement uncertainties away from the stated limit
(if the uncertainty is + 6dB , equipment may be found at + 12 dB from the
limit).
The expansion factor is arbitrary (ETSI has been using 1,96 or 2) If the
value of the uncertainty is taken into account de facto, in the pass/fail
criterion, that criterion contains also implicitly the effect of some arbitrary
value.
Furthermore the additive combination of a finite number of finite
distributions can only result in a finite distribution and therefore not in
any Gaussian shaped curve. As a result, it may occur that an expansion
factor of 2 results in an estimation of the uncertainty beyond the worst case
(e.g. by 15 %). Under such situation, it makes no sense to use that over estimated value to, de facto, relax or tighten the limit.
Recent experience (TFES) has shown that the evaluation of the uncertainty
may depend upon frequency and other parameters (e.g. size of the EUT)
Possible evolution, in time, of the maximum acceptable uncertainty
Incorporating the uncertainty implicitly in the limit humpy limits !
(risk of problems for the manufacturers due to both effects).

26

More precisely ...

One rectangular distribution ...

2
x2

-A

27

More precisely ...

One rectangular distribution ...
The usage of an expansion factor of 1,96 or 2 makes the
expanded uncertainty go beyond the worst case !

2
x2

-A

A
3

2 A
1,15
3
28

More precisely ...

Two rectangular distributions ( A > B )

-A-B

A+B

A>2B

-A+B

2
x2

-A

A
3

2 A
1,15
3
29

More precisely ...

Three rectangular distributions ( A > B > C )

-A-B

A+B

A>2B
B>2C

-A+B

2
x2

-A

-A-B-C = worst case

A
3

2 A
1,15
3

It may occur that the expanded uncertainty exceeds the worst case30

Conclusion ...
This example shows that approaches other than the
shared risk approach (which is based upon a direct
comparison between the result of a measurement and
the corresponding limit) may generate wrong
decisions.

(The shared risk approach is the pass/fail criterion

recommended in TR 100 028.)

31

The New Approach (in accordance with TR 100 028 V 1.4.1)

(STEP 4 and 5)
Theoretical approach for the evaluation of
uncertainties
Evaluation of the measurement uncertainty for each
measurement

Agree on pass/fail criteria

Include the maximum acceptable values for the
uncertainty (and the limits e.g. in Harmonized
Standards under the R&TTE Directive)
Prepare supporting documentation (e.g. TRs, forms)

32

Examples and support documentation

(e.g. TRs, forms)
ETSI has published a number of harmonized standards
using the statistical approach together with the shared risk
approach. These Harmonized Standards include the
maximum acceptable uncertainties and have been listed in
the OJ of the EU, which has given them the appropriate
legal status.

ETSI has developed a number of documents

e.g. TR 100 028 (version 1.4.1 published end 2001) in
support of the methodology.
TR 100 028 includes spread sheets as well,
(V 1.4.1 also includes a presentation (the .ppt file)).
ETSI has also developed forms (e.g. test report forms)
Bibliography
...Bureau International des Poids et Mesures
(BIPM method)

33

Conclusion
The usage of the probabilistic approach (as defined in
TR 100 028), on a global basis is expected to :
Facilitate global roaming while simplifying the legal
aspects (e.g. in R&TTE geographical areas)

Allow for an enhancement of the performance in terms

of uncertainties, as the performance of instrumentation
increases, without a need to change the standards
Increase transparency and clarity and ease both :
system deployment
Compatibility and sharing studies between services.

34

Conclusion (2)
Further steps :
Make it clear and public that the ETSI preferred
approach is the shared risk
publish a general standard
circulate other documents (LS, etc )

Spread out the usage of the statistical / shared risk

approach on a global basis (RAST, GRSC).

35