18 Performance-Related Metrics

in the ISO 9126 Standard

Steffen Becker

University of Karlsruhe, Germany

ISO 9126 [243, 244, 245] is a standard which can be used to describe the quality of soft-
ware systems. It is based on a quality model that is illustrated in part one of the standard
[243]. This model distinguishes quality attributes into internal and external attributes.
Internal metrics depend on knowledge on the internal details of the respective software.
External metrics can be measured without knowing internal details. Furthermore, the
quality model introduces characteristics and sub-characteristics which are abstractions of
the actual attributes. For example, Usability is an abstraction of Learnability, Understand-
ability, and Operability which each itself again abstracts from the different attributes.
The ISO 9126 standard has no characteristic Performance. The closest characteristic
to our definition of performance is Efficiency. It is divided into two sub-characteristics:
time behaviour and resource behaviour. Some people say this distinction is artificial as
time is also a (rare) resource. Nevertheless, the timing behaviour is separated in ISO
9126. The important attributes of Efficiency in ISO 9126 are being described in the
external metrics specification. Hence, the following is restricted to the second part of
the standard describing the external metrics [244].

18.1 General Comments

The introduction to the efficiency metrics in ISO 9126 contains some interesting general
comments. First, it is remarked that most of the metrics for efficiency vary over time
and can only be measured with certain jitter. Therefore, the introduction recommends
to use statistical means to characterise the respective attributes. It is recommended that
at least the maximum and the distribution should be taken during measurements.
Additionally, the influence of the hardware and software environment is pointed out.
Software running on different hardware platforms and/or different software platforms
(operating systems, application servers, middleware platforms, ...) has different effi-
ciency characteristics. The timing behaviour varies with the speed of the underlying
system layers. Hence, ISO 9126 recommends to specify the environment on which the
measurements have been performed exactly.
In some of the metrics a so called task is referenced. Task should be defined in the
context of the application and should somehow correspond to typical tasks of the sys-
tem. Examples stated are business or database transactions, sending/receiving a com-
munication package, the processing and output of a chunk of data for a user application.

18.2 Time Behaviour Metrics

ISO 9126-2 defines the following timing related metrics:

I. Eusgeld, F.C. Freiling, and R. Reussner (Eds.): Dependability Metrics, LNCS 4909, pp. 204–206, 2008.
c Springer-Verlag Berlin Heidelberg 2008
 Performance-Related Metrics in the ISO 9126 Standard 205

Response time. The response time should measure the time consumption for complet-
ing the specified task. It should be recorded as the time span between the start of
the task and its completion.
Response time (Mean time to response). The mean time to response should record
the average response time under a specified system load in terms of concurrent
tasks and system utilization. It is estimated by taking the response time several
times and dividing the sum of all runs by the amount of measurements.
This can again be divided by the required mean response time so that the result
is the ratio of fulfilling the requirements. The ratio should be less than 1.0, lower
being better.
Response time (Worst case response time). The worst case response time is calcu-
lated using the ratio of the maximum response time of a set of measurements di-
vided by the required maximum response time. Again, the value should be less than
1.0, lower being better.
Throughput. The throughput describes the amount of tasks which can be performed
over a given period of time.
Throughput (Mean amount of throughput). For an amount of concurrent runs of the
specified task calculate the sum of each of the throughputs divided by the amount
of runs. Then divide this by the required throughput to get a ratio. The ratio should
be less than 1.0, lower being better.
Throughput (Worst case response time). For an amount of concurrent runs of the
specified task, take the maximum of the measured throughput values and divide
this by the required throughput to get a ratio. The ratio should be less than 1.0,
lower being better.
Turnaround time. The turnaround time is the waiting time an user experiences after
issuing an instruction to start and complete a group of related tasks. It is the time
span from starting the tasks until the last task finishes.
Turnaround time (Mean time for turnaround). For an amount of measures of the
turnaround time under a given system load in terms of concurrent tasks and sys-
tem utilization take the sum of all measures divided by their number. This value
is again divided by the required average turnaround time to get a ratio. The ratio
should be less than 1.0, lower being better.
Turnaround time (Worst case turnaround time ratio). For an amount of measures
of the turnaround time under a given system load in terms of concurrent tasks and
system utilization take the maximum of all measures divided by the required worst
case turnaround time to get a ratio. The ratio should be less than 1.0, lower being
better.
Waiting time. The waiting time characterises the proportion of the time the users spent
waiting of the total time a set of concurrent scenarios takes to execute.

18.3 Resource Utilisation Metrics

We briefly summarize the resource utilisation metrics in the following. The resource
utilisation metrics are divided into three main groups: I/O device utilisation, memory
resource utilisation and transmission resource utilisation.
206 S. Becker

I/O Device Utilisation

The I/O device utilisation section offers several metrics to characterize the load of the
specified resources with respect to the tasks defined. It contains metrics for the device
utilisation, load limit, I/O related errors and the waiting time of the user resulting from
device response times.
Note, that several of the metrics have to be taken under fixed conditions with respect
to the concurrency of the tasks involved. Hence, those metrics can be adjusted to several
situations and their interpretation highly depends on the conditions under which they
have been measured.

Memory Resource Utilisation

The memory resource utilisation metrics can be used to determine the memory con-
sumption during the execution of the specified tasks. The standardised metrics contain
metrics for the maximum amount of memory consumed, the mean occurrence of mem-
ory errors and the ratio between memory errors and execution time.
Note, that the maximum memory utilisation is determined by evaluating the maxi-
mum number of memory related error messages during a set of experiments which is
different from common sense (see ISO9126-2 [244, p.48]).

Transmission Resource Utilisation

The transmission resource utilisation is supposclsed to describe the load of commu-
nication related transmission channels. Metrics of this group contain the maximum
transmission utilisation, the media device utilisation balancing, the mean occurrence
of transmission errors and the mean of transmission errors per time.
Note again, that the maximum transmission utilisation is evaluated using the max-
imum of the transmission error count over a set of experiments (see ISO9126-2 [244,
p. 49]). Additionally, the device utilisation balancing is supposed to capture the delay
introduced by synchronisation constraints between the different media types.

18.4 Open Issues

The metric definitions in the ISO 9126 are quite vague. Additional information has to
be supplied together with the metric: the system, on which the value was measured,
workload, and the task performed. This sounds reasonable, however, it is a tough task
to compare the results taken by one person with results from another person. In order
to allow the comparison of the measured values, there is the additional need to have a
common reference platform, reference setup and reference measurement instructions.
Additionally, those instructions have to be clear and reproduceable, so that any two
experts perform the same experiment. In so doing, the results become comparable and
can be verified by third-parties.