Appendix 6: Validation parameters considered in the context of methods in microbiology.

SANS 17025 specifically refers to the following parameters: Accuracy, Selectivity, Detection limit, Linearity, Robustness, Repeatability, Reproducibility, Uncertainty of
Measurement, Cross-sensitivity against interference from the matrix. It follows that these parameters should be considered when planning validation experiments.
The points below provide information on how validation parameters could be interpreted for methods in microbiology.

Parameter Parameter considered in the context microbiology Hypothetical examples to illustrate the practical investigation
methods of validation parameters.
Accuracy: closeness of the For microbiology methods there are obvious challenges Lyophilized units of a certified reference material are used to spike E.coli
agreement between a test result associated with determining the true number of organisms in a into ten drinking water samples. The certificate of analysis for the
and the accepted reference value sample. Therefore the best estimate of the true value will often reference material indicates that each lyophilized unit contains 100 E.coli
(SANS 13843:2000). have to be used. This could be derived from: 1) A certificate of organisms. If an average of 95 E.coli /100mL were detected in the ten
analysis for a quantified reference materials, 2) the consensus samples accuracy could be expressed as follows.
value of a proficiency testing scheme or 3) the result of an
alternative reference method. The average number of E.coli detected was 95 cfu/100mL
Ultimately the selection of a true
value will require an element of judgement on the part of the
laboratory. The best estimate of the true value is 100cfu/100mL.
Once the accepted reference value has been selected
accuracy can be expressed using the formula below.
95

When drafting validation reports it would be ideal to document any

bias observed. Bias may be considered to be the systematic
measurement error or its estimate, with respect to a reference
quantity value. (VIM-3rd edition, ISO international vocabulary of
basic and general terms in metrology).

Selectivity: The ability of a method For microbiology methods the ability of a method to determine A method using membrane filtration and selective chromogenic media is
to determine accurately and accurately and specifically the analyte of interest might best be used to quantify E.coli in ten 100mL surface water samples. A total of 100
specifically the analyte of interest in expressed using the concepts of sensitivity and specificity below. colonies were counted.
the presence of other components in
a sample matrix under the stated Sensitivity is the Proportion of positive targets (colonies, tubes, Seventy of the colonies had a presumptive E.coli phenotype. Of these 60
conditions of the test. Eurachem wells) correctly assigned by the method. were confirmed to be E.coli using biochemical tests (true positives) while
Guide (1998). The Fitness for 10 were found to be of another species (false positives).
Purpose of Analytical Methods. A
laboratory guide to method Thirty colonies did not have a typical E.coli phenotype. Of these 25 were
validation and related topics. found to be of another species (true negatives) while 5 were shown to be
Copyright LGC (Teddington) Ltd E.coli using biochemical tests (false negatives).
1998 Specificity is the proportion of negative targets (colonies, tubes,
wells) correctly assigned by the method. From the example above the true positive count was 60 and the false
negative count was 5. Therefore Sensitivity = 60 / (60 +5) = 0.92

SANAS Page 20 of 25
 From example above the true negative count was 25 and the false
positive count was10. Therefore Specificity = 25 / (25+10) = 0.71

SANAS Page 21 of 25
TR 28-01
Limit of detection (LOD): The
lowest number of microorganisms
that can be detected, but in numbers
that cannot be estimated. EA Guide
EA-04/10: 2002,
Accreditation in microbiological
laboratories
Linearity: Ability of a method to
obtain test results proportional to
the concentration of the analyte.
Eurachem Guide (1998). The
Fitness for Purpose of Analytical
Methods. A laboratory guide to
method validation and related
topics. Copyright LGC (Teddington)
Ltd 1998
Attempts to determine the LOD for microbiology methods are A commercial lyophilized reference material is obtained with an average
complicated by the difficulties associated with preparing low count of 30 E.coli per unit. The material is used to spike ten 100mL water
concentrations of target organisms. This concept is captured in samples. E.coli are then enumerated in the samples using a multiple tube
the following points taken from SANS 13843:2004 section 6.1. fermentation assay.
The points assume that microorganisms in a perfectly mixed
matrix have a Poisson distribution. If E.coli is detected in each of the ten samples it can be stated that the
method has demonstrated the ability to detect the target organism at a
 “Random uncertainty increases rapidly as the colony concentration of 30 E.coli per 100mL. It may also be stated that the
count decreases ” actual detection limit may be lower but cannot be investigated due to
 “In the count range below about ten, which happens to be practical constraints around the accurate preparation of spiked samples
of considerable public health interest, single with concentrations of E.coli below 30 organisms per sample.
measurements are so imprecise that they can hardly be
characterized as better than semi-quantitative.”
 “At very low particle concentrations all microbiological
methods, MPN and colony count included, become
essentially P/A methods.”
 “Colony numbers such as 20, 25 or 30 have been
traditionally considered the lowest statistically reliable
counts.”
Given the challenges associated with the preparation of low
numbers of microorganisms experiments which attempt to
demonstrate an LOD below 30 organisms may not generate
statistically significant results.
When executing methods to assess linearity it may be appropriate A lyophilized and quantified reference material was used to spike six
to simultaneously define the upper working limit of the method. 100mL drinking water samples. The amount of reference material added
Essentially this would be the highest concentration of target was adjusted so that the concentrations of E.coli spanned a range
organisms in a sample which falls within the methods linear consistent with the intended purpose of the method.
range.
E.coli were enumerated in each sample. These results were plotted
against the amount of reference material spiked into each sample. A
commercial spreadsheet program was used to assign a linear trendline to
the data set. A random distribution about the trendline confirmed the
linearity of the method. A systematic trend of data points away from the
trendline would have indicated a departure from linearity.
SANAS Page 22 of 25
TR 28-01
Robustness: A measure of an
analytical procedure’s capacity to
remain unaffected by small, but
deliberate variations in method
parameters and provides an
indication of its reliability during
normal usage. Eurachem Guide
(1998). The Fitness for Purpose of
Analytical Methods. A laboratory
guide to method validation and
related topics. Copyright LGC
(Teddington) Ltd 1998
Repeatability: Closeness of the
agreement between the results of
successive measurements of the
same measurand under the same
conditions of measurement. [VIM:
1993 ISO International vocabulary of
basic and general terms in
metrology]
Despite efforts to execute methods consistently there will always
be slight variations in the test conditions. In the context of
microbiology two of the most important parameters are incubation
time and incubation temperature. Others include slight differences
in the concentration of media prepared, age of the media used
and sample holding time. For any method a degree of judgement
is required to identify those experimental parameters which could
influence the results.
Once key experimental variables have been identified the extent
to which they are expected to vary should be defiend (eg
incubation time could vary between 20 and 24 hours). Validation
experiments should then be conducted to examine the impact of
the variable. The concept is best illustrated with an incubation
time example.
Repeatability gives an indication of the degree of variation in
results that can be expected when one analyst executes a method
over a short space of time using the same consumables, media
and equipment.
It is important to recognize that for microbiology methods
repeatability will be a component of reproducibility. Therefore if
the reproducibility has been thoroughly examined and deemed to
be acceptable, there may be little value in estimating the
repeatability of the method separately.
SANAS Page 23 of 25
A method is being validated for the identification of E.coli in untreated
water samples using membrane filtration technology and a commercially
available media. The manufacturer of the media suggests that incubation
should proceed for a period of between 20 and 24 hours.
An experiment to demonstrate the robustness of incubation time may be
executed as follows.
 Thirty raw water samples are collected.
 Each sample is split so that there are two equivalent sets.
 The first set is processed and incubated for 20 hours.
 The second set is processed and incubated for 24 hours.
 A statistical test such as a Student t-test can be used to
compare for the two incubation times.
 The method may be considered robust if there is no significant
differences in the results for the two incubation times.
Repeatability can be estimated using the same approach as that
provided for reproducibility below. However, variation in the experimental
conditions should be minimized as far as possible. The same analyst
should execute all the work using one set of consumables and
instruments over a short space of time.
TR 28-01

Reproducibility: Closeness of the Reproducibility gives an indication of the degree of variation in Example of a hypothetical experiment to determine the relative standard
agreement between the results of results that can be expected when different analysts execute a deviation of reproducibility (RSDR) for a quantitative method using split
measurements of the same method at different times using different batches of consumables, samples. This example was adapted from the Health Protection Agency
measurand carried out under media and equipment. (2005) National Standard Method QSOP 4 Issue 5 Appendix A. It is
changed conditions of recommended that the original reference be consulted for greater detail
measurement. [VIM: 1993 ISO Due to the labile nature of microorganisims the same samples on the methodology and equations used.
International vocabulary of basic cannot typically be stored over a long time period of time and
and general terms in analyzed using different instruments, analysts and consumables. The experiment considers a method used to enumerate E.coli in water
metrology] This can be partly overcome by expressing reproducibility as a samples. It runs over four days. On each day a single water sample is
Relative Standard Deviation (RSD) derived from the results for collected and split into two aliquots. The aliquots are analyzed separately
split samples. The concept is illustrated in the example. with as much variation in the analytical conditions as permitted by the
method (eg different analysts, different consumables and different
equipment). Common logarithms (log10) are taken for the counts. Relative
Standard Deviation RSDR is determined for each paired count. A
hypothetical data set is presented below.

Results for split samples

E.coli cfu/100mL
Day Sample
Split Result 1 Split Result 2
1 1 1089.00 1211.00
2 2 122000.00 142000.00
3 3 32500.00 29000.00
4 4 28000.00 35020.00

Common logarithms (log10) are taken from the data

above. Relative Standard Deviations are then calculated
for each data pair.

Relative
Log10 values for counts Standard
Day Sample
Deviations
(RSDR)
Split 1 Split 2
1 1 3.037028 3.083144 0.010656
2 2 5.086360 5.152288 0.009106
3 3 4.511883 4.462398 0.007798
4 4 4.447158 4.544316 0.015281

An estimate of the combined reproducibility RSDRC is obtained by

determining the quadratic mean of the relative standard deviations (RSDR)
for each pair. An RSDRC estimate of 0.011 would be derived using the
data in the tables above.

SANAS Page 24 of 25
TR 28-01

Uncertainty of Measurement: EA Guide EA-04/10: 2002, Accreditation in microbiological This example was adapted from the Health Protection Agency (2005)
Parameter associated with the laboratories makes to key points which place uncertainty estimates National Standard Method QSOP 4 Issue 5 Appendix A. It illustrates how
result of a measurement that in microbiology in context. an estimate of Uncertainty of Measurement (UM) can be derived from the
characterizes the dispersion of the reproducibility data presented in the example above. It uses a
values that could reasonably be EA Guide EA-04/10: 2002, clause 5.2 states: “Microbiological hypothetical scenario where a result has been obtained for E.coli in water
attributed to the measurand tests generally come into the category of those that preclude the of 6.76 x 104 cfu/100mL.
Eurachem Guide (1998). The rigorous, metrologically and statistically valid calculation of
Fitness for Purpose of Analytical uncertainty of Measurement. It is generally appropriate to base The following formula is used to determine uncertainty for a given result.
Methods. A laboratory guide to the estimate of uncertainty on repeatability and reproducibility
method validation and related data alone, but ideally including bias (e.g. from proficiency testing  Upper UM estimate = log10 (result) + k x RSDRC
topics. Copyright LGC (Teddington) scheme results).”  Lower UM estimate = log10 (result) - k x RSDRC The
Ltd 1998
EA Guide EA-04/10: 2002, clause 5.4 states: “The concept of following data are substituted into the UM equation.
uncertainty cannot be applied directly to qualitative test results
such as those from detection tests or the determination of  The result is 6.76 x 104 cfu/100mL.
attributes for identification. Nevertheless, individual sources of  The RSDRC from the reproducibility example above was 0.011.
variability, e.g. consistency of reagent performance and analyst  A coverage factor (k) of 2 is selected (See Health Protection
interpretation, should be identified and demonstrated to be under Agency (2005) for further guidance on selection of coverage
control.” factors). 4
 Upper UM estimate = log10(6.76 x 10 ) + 2 x 0.011 = 4.8519
The example provided illustrates how an estimate of uncertainty 4
 Lower UM estimate = log10 (6.76 x 10 ) - 2 x 0.011 = 4.8079
can be derived from reproducibility data.
The result above provides an uncertainty estimate on the log 10 scale. If a
result is required on the natural count scale then the antilog of these two
values should be determined as follows

 Upper UM estimate = 104.8519 = 7.11 x 104 cfu/100mL

 Lower UM estimate = 104.8079 = 6.43 x 104 cfu/100mL

SANAS Page 25 of 25
TR 28-01

ADDENDUM 1: AMENDMENT RECORD

Proposed By: Section Change

STC Converted from a technical guidance (TG 26) to a technical requirements (TR 26)
document.
QM 2.1 Included references to SANAS R80 and ILAC G9:2005
STC 2.1 Included references to: Eurachem Guide (1998), Ludwig Huber, Health Protection
Agency (2005) and the PALCAN Guidance

accuracy

Determination - Prepare a suspension of microorganisms at the upper end of

the range of the test
and serially dilute down to the lower end of the
range of the test.
At least five suspensions across the range of the test should be analyzed.
Calculate
each of the suspensions as a percentage dilution of the original.
The result obtained by the sample at the upper range of the test should be
referred to as
100%. Compare the result,
i.e., actual versus expected, obtained by each of the other suspensions, i.e.,
100%, 75%, 50%, 25% and 10% of the original culture, against the result
expected from the dilution; present these as percentage recoveries.
Acceptance criteria - The new method should give equivalent or better
results than the current method. Percentage recoveries of the order of 30%
can be expected for microbiological methods.
The acceptance criterion is at least 70% recovery

Repeatability
refers to the use of the microbiological method within the same laboratory
over a
short period of time using the same SANAS
analyst with Page 26 of 25
the same equipment. Reproducibility refers to the
use of the microbiological method withinthe same laboratory over a short
period of time
using different analysts with the same equipment.
Determination –
Prepare a suspension of microorganisms at the upper end of the range of the
test and serially dilute down to the lower end of the range of the test. At least 2
suspensions across
the range of the test should be analyzed. For each
suspension at least 10 replicates should be assayed
in order to calculate statistically significant estimates of the standard deviation
or relative standard deviation (coefficient of variation).
Acceptance criteria - Generally, a coefficient of variation (relative standard
deviation) in the 15 to 30% range is acceptable for microbial counts
[iii] Specificity
TR 28-01

Definition - The specificity of a microbiological

method is its ability to detect a range of microorganisms which demonstrates
that the method is
fit for purpose. Method compatibility with the
types of sample matrices with which the method
will be used should also be proven.
Determination - Screen the method against a representative range of
microorganisms appropriate
to the method. Screen the method against a representative range of sample
types.
Acceptance criteria - All microorganisms selected
as representative are successfully isolated and enumerated from the sample
matrices.
[iv] Limit of Detection
Definition - The limit of detection is a parameter of a limit test.
It is the lowest number of microorganisms in a sample that can be detected,
but not
necessarily quantified, under the stated experimental conditions.
A microbiological limit test determines the presence or absence of
microorganisms.
Due to the nature of microbiology, the limit of detection refers to the number of
organisms present in the original sample, before any incubation step, not the
number of organisms present at the point of assay.
Also, the amount of sample tested and the dilution of that sample may
determine the Limit of
Detection.
For example, when 10 grams of test material is diluted in 90 mL of diluent and
1 mL is
plated, the absence of colonies on the plate would be reported as <10 cfu per
g.
Determination - As it is not possible to consistently obtain a reliable sample
containing a single microorganism, it is essential that the limit of detection of
an assay is determined from a number of replicates (n ≥5) for the standard
compendial microorganisms.
Acceptance criteria - The best statement that can be made is that if a single
organism is present in the sample, it will be detected during the time frame of
the assay.

[v] Limit of Quantification SANAS Page 27 of 25

Definition - Limit of quantification is a parameter of quantitative assays for
low levels of microorganisms in sample matrices. It is the lowest number of
microorganisms which can be determined with acceptable precision and
accuracy under the statedexperimental conditions
Determination - As it is not possible to obtain a reliable sample containing a
known number of microorganisms, it is essential that the limit of quantification
of an assay is determined from a number of replicates (n ≥5) at each of at
least five different points across the range of the assay.
Acceptance criteria - The best statement that can be made is that if a single
organism is present in the sample, it will be quantified during the time frame of
the assay.
Note: Limit of Qualification of the new method
should be equivalent or better than the existing method.
[vi] Linearity
Definition - The linearity of a microbiological test method is its ability to elicit
TR 28-01

results which are proportional to the concentration of microorganisms present

in the sample within a given range.
Determination - As it is not possible to obtain a reliable sample containing a
known number of microorganisms, it is essential that the linearity of anassay is
determined from at least duplicates at each of at least five different points
across the range of the assay.
Acceptance criteria - Correlation coefficient r2 = 0.9 or better with the slope
not diverging more than 20% from 1.0, i.e., r2 = 0.8 to 1.2. Another statistical
tool that could be used is the test for goodness of fit.
[vii] Range
Definition - The range is the interval between the upper and lower levels of
microorganisms that have been demonstrated to be determined with
precision, accuracy, and linearity using the method as written.
Determination - The range of the method is validated by verifying that the
analytical method provides acceptable precision, accuracy and linearity when
applied to samples containing analyte at the extremes of the range as well as
within the range

Acceptance criteria - This will depend on the performance characteristics of

the method.
[viii] Ruggedness
Definition - The ruggedness is the degree of reproducibility of test results
obtained by analysis of the same samples under a variety of normal test
conditions, such as different analysts, different instruments, different lots of
reagents, etc.
Ruggedness is normally expressed as the lack of influence on test results of
operational and environmental variables of the microbiological method.
Ruggedness is a validation parameter best suited to determination by the
supplier of the
test method with easy access to multiple instruments and batches of
components. Data supplied by the test method manufacturer are entirely
admissible to prove validation of ruggedness.
Determination –
Prepare a suspension of microorganisms and test at least 5 replicates against
each assay variable in order to be able to calculate statistically significant
estimates of the standard deviation or relative standard deviation (coefficient
of variation).
For a quantitative method, the range of the test method should also be
covered. Attention should be paidSANAS
to the inherent instability of microbiological
Page 28 of 25
suspensions and
experimental protocols randomized to eliminate bias.
Acceptance criteria - Generally a coefficient of variation in the 10 to 15%
range is acceptable.
[ix] Robustness
Definition - The robustness of a microbiological method is a measure of its
capacity to remain unaffected by small but deliberate variations in method
parameters. It provides an indication of
its reliability during normal usage. Robustness is a validation parameter best
determined by the
supplier of the test method. Data supplied by the test method manufacturer
are entirely admissible
to prove validation of ruggedness.

Determination - The manufacturer investigates changes up to 20% on the

TR 28-01

critical reagent concentrations, instrument operation parameters and

incubation temperatures.
Acceptance criteria - The results need to be reviewed against the
manufacturer’s quality assurance requirements and the instruction for use.

SANAS Page 29 of 25