Accuracy in Quantitative Risk Assessment?

13th International Symposium on Loss Prevention

J.R.Taylor
Dome Oillfield Services, Abu Dhabi
JRT@ITSA.DK

Abstract

This paper describes the results of a study stretching over 25 years, to determine how
accurate the risk assessment process can be made for process plants. During that time, risk
assessment has become a standard working tool for safety engineering, not just in land use
planning, but in design of safety shutdown and interlock systems, gas detection systems,
building blast resistance, and fire protection systems. It becomes highly desirable that QRA’s
should be just as accurate as calculations in other areas of engineering.

The study used several methods of approach including QRA followed by a 20 year serial
study of a number of plants; comparison of results from several methodologies and a large
number of models against each other; comparison with actual accident reports and statistics;
and third party reviews of QRA’s.

The main results are that while models have been developed which agree very well with
experiment, they are not always relevant to real life; that several very important accident
types are generally ignored; that the accuracy within the plant and especially at short
distances is much poorer than the accuracy at a distance; that the calculation methodologies
most commonly used introduce unnecessary additional errors; and most importantly, that
most of the deficiencies in current practice can be fairly easily corrected.

Acknowledgements

The authoer gratefully acknowledges the help in completing this study from Herlinde
Beerens, Y Weber, Abid Sayyed, Tamir Said, Shastri Ranjiputri

Accuracy in QRA accident frequencies

The most important property of any practical application of scientific theory is that the results
should agree with observation. Secondarily, the results should be repeatable.
Until a few years ago, even repeatability in QRA was not achievable to better than two orders
of magnitude (ref. 1). This was a saddening result, considering that QRA methodology has
been under development for over 30 years. The cooperative study organised by RIVM and
published in 2004 (ref. 2) showed that experienced analysts, using the same data, and a
carefully made guideline, could produce result in agreement to within a factor of 2, even
when using different software packages. This was encouraging in that it showed that
consistency, at least, can be achieved.

Even at the best, though, such calculations can only be “accurate on average”, that is, they
are accurate for an average plant. If you have an average plant, then you are in luck.

The data that are used in practice today for frequency determination are largely drawn from
just a few sources, and to a considerable extent, from the North Sea oil and gas production
systems (e. g. ref. 3). Others are often based on the original engineering judgements in the
Rijnmond study (ref. 4, with updates through the years, ref.5). Given that actual failure rates
for some items of equipment have vary between different plant types by up to three orders of
magnitude (ref. 6), this means that “accuracy on average” may be very inaccurate. An
example of this is the urea reactor on older urea plants. The historical frequency of vessel
rupture is about 2.6*10-3 per year (based on US experience), whereas generic vessel
frequencies in tables are of 5*10-5 to 5*10-7 per year i.e. a factor of 1000 inaccuracy.

To get over this problem, the RELBASE release frequency data base was developed (ref. 6)
this is a very extensive collection of data from in all about 2,500 plants of different types. It
allows plant specific release frequency values to be used, with dependency on technology,
feasible causes, and process fluid. It does not, so far, allow process safety management
variations to be taken into account.

Plant specific frequencies of major accidents derived using this approach were compared
with actual major hazard accident frequencies, for those plant types for which sufficient data
are available. Comparisons of prediction and observed frequencies are shown in Table 1,
with accident frequencies taken from ref. 7, and also derived as part of this project.

Unit type Frequency of UVCE, per year

Statistics from Fyman, ref 7 Predicted Ratio
and from ref. 6 Predicted/observed
Crude unit 4.9*10-4 5.45E-04 1.1
Alkylation unit 5.1*10-4 1.56E-04 0.31
Ammonia 0.21, all sources 0.036 0.051
synthesis, 0.12, excluding the 3 worst
ammonia release plant
0.07 excluding 3 worst plant
and ammonia truck loading
Ammonia releases, 0.057 0.04 0.70
large
Explosions 0.053 0.033 0.62

Plant specific data may be undesirable in regulatory circumstances, such as in land use
planning, because of the complexity of deriving fairly precise values. However, at least the
large variations arising from different technologies and different process fluids should be
taken into account. It simply does not make sense to use the same values for failure of piping
for the case of simple carbon steel and for 360 SS steel for example.

Note that the issue here is not uncertainty in failure rate values. The generic values used are
simply wrong for many cases, and this error is independent of any statistical uncertainty.

Looking to the future, many companies are already using risk based inspection as a tool, and
this is already generating a lot of plant specific failure rate data.

The curious case of human error

It is generally agreed that human error is one of the most important causes of major
accidents. This was confirmed from the accident reviews made as part of the present study.
Yet curiously, risk analysts working in the process industries spend large amounts of time
counting pipes, flanges and valves, and generally none at all in counting opportunities for
human error. By contrast, HRA is regarded as a standard part of nuclear power plant risk
assessments.

The kinds of operator and maintenance errors which appear as causes of major hazards
accidents are a fairly small set (ref. 8). It is possible to determine the frequency on a
statistical basis, given the number of accident reports available. The US RMP data which
was used as part of the input to the RELBASE database includes a significant percentage of
operator and maintenance related accidents. Alternatively human reliability analysis methods
can be used for analysis of the typical tasks which can give releases. The task risk analyses
(TRA’s or JSA’s) carried out by many companies provide a useful basis for such analyses.
It has been argued that release frequency data used in QRA’s already include an element of
human error. Examination of the sources of data used in commercial risk analyses show that
this is not actually the case. In any case, operator and maintenance errors often give
consequence which are quite different from leaks and pipe ruptures recorded in QRA
reference databases.

The impact of including human error into petroleum and chemical plant QRA have not so far
been investigated as part of the project.

By far the biggest human error contributions to major hazards accidents are design error and
management error (ref. 9). Accounting for these presents a philosophical problem – if you
can predict these errors it is best to remove them, not waste time calculate their frequency.
We did not find any good way out of this deadlock so far. Some possibilities for investigation
are described in ref. 9.

Consequence modelling

In order to assess the importance of modelling on QRA results, several approaches were
used:

1. Several commercial consequence modelling packages were obtained, and results

compared, using as examples the cases from the Dutch Yellow Book (ref. 10). This
approach does not guarantee accuracy, but at least it allows anomalies to be
identified, and the sources of these can then be investigated.
2. A large proportion of the, published models were implemented in one QRA package
(ref.11) and the results compared.
3. The model results were compared with published experimental data.
4. The model results were compared with accident results.
5. The impact of model differences was assessed by completing QRA’s for six reference
plants, investigating the difference.

The results from this exercise were encouraging. At distances over 100 to 200 m., many of
the models agreed with experiment to within a factor of 2 or better. The least consistent of
the models were those for which turbulence is important, i.e. gas dispersion and explosion,
but agreement with experiment to better than a factor 2 in 90% of cases seemed to be the
usual case.

Agreement at short distances, as needed for in-plant risk assessment for hazards to
employees and for domino effects, were less good. There were largely due to the common
use of oversimplified or inappropriate models. Examples were:
 • Modelling jet fires as simple straight flames (in fact, horizontally and obliquely directed
flames always bend upwards)
• Neglect of flame drag and flame dip in tank fire and pool fire models.
• Use of a single surface radiation intensity for pool and jet fires, rater than varying
intensity
• Neglect of gas jet impingement
• Neglect of cross wind dispersion
• Assumption that all liquid releases occur on flat, non absorbent ground (in fact most
chemical plants have a slope of about 2˚, which has a very large effect on pool size.
Reduction in pool size is the prime purpose of the slope.)

Table 2 shows examples of accidents with their coverage in standard QRA methodologies.

Accident Mechanisms Included in Models

standard available ?
methodologies

Texas City 2005 KO drum overflow, spraying vent, HSCE N Y

Tosco Avon 1997 Hydrocracker reactor runaway, HSCE N Y

Valero Sunray 2008 Dead leg, jet fire Y Y

Formosa Plastics Fork lift truck crash, UVCE Y Y

2005

Totale, Buncefied Overflow, HSCE, splashing flow, UVCE N Y

2005

Conoco Phillips, Injection point corrosion, RBI, UVCE Y Y

Humber

CAI, Danvers 2006 Confined VCE N Y

Texaco, Milford HMI, KO drum overflow, hammer, UVCE Hammer N, Y

Haven, 1994 UVCE Y

Giant Industries, HMI on valving, flash fire, jet fire N Y

Jamestown

Motiva Enterprises, Acid tank explosion N Y

Delaware

Pennzoil, Rouseville, Vapour release near hot work, tank N Y

1995 explosion
Table 2, examples of accidents, and their treatment in standard methodologies, ref. 10 and
19

Models for all of these effects were found in the literature. Especially the model reviews
performed by Deaves, Rew et al. for the UK HSE proved more accurate at short ranges
(refs.12, for example). (Perhaps not surprising since the work is relatively recent and could
draw on extensive earlier work).Omission of these effects was found to be important in many
cases. Most of the simplifications of the models result in underestimate of risk, so the
cumulative effects are important.
One perhaps surprising feature of modelling which became obvious from the comparisons
with actual accidents was that hazard zones for all pool fires, flash fires and explosions are
calculated as circles in commercial QRA packages. Not all accidents are circular. Examples
of accidents with extended linear geometries are the fire at Bellingham (ref. 13), Cubatao
(ref. 14), and Buncefield (ref.15), and San Rafael de la Laya.

It proved relatively straightforward to implement this kind of extended geometry, using

models developed for environmental protection spills assessments (ref. 16) for the liquid
flows, using terrain sensitive gas dispersion models for gases, and using superposition of
acoustic waves for calculation of explosion pressures.

One problem that requires considerable further research was identified from the comparisons
of models with accident data. That is the prediction of unconfined vapour cloud explosion
pressures. Flame acceleration is the key problem, and many different mechanisms for this
have been identified, only one of which has well researched full scale experimental support.

Lacunae in consequence modelling

Many of the phenomena which occur in practice in process plant are not included at all in
standard methodologies and model sets. Examples are:

• Splashing releases of volatile liquids, as at Buncefield

• Spraying releases of volatile liquids, as in most liquid releases from small and
medium size holes in pipes, and as illustrated dramatically in the St. Herblain
accident (ref. 16) (Note that spraying releases of liquefied gases are normally
included in QRA’s)
• Boilover and slopover
• Fire induced tank explosion, (as occur fairly frequently, see ref. 6 , and as illustrated
dramatically at Port Edouard Heriot, Lyon, in ref. 17)
• Reactor runaway explosions, both for continuous and batch reactions.
• Vessel rupture explosions (without fire) including boiler explosions
• Fired heater explosions
• Confined explosions, for example in fired heaters
• Dust explosions
• Indoor process accidents (releases, evaporation, fires and explosions) as needed for
many fine chemicals, coatings, and chemical waste plants.

Models were available for all of these except fire induced tank explosion, which had to be
developed. The error induced by these lacunae varied from a significant percentage to a very
large factor, depending on the case.

It could be argued that the omitted scenarios are exceptional cases, which could be
calculated separately and incorporated into QRA’s. However, it appears especially that
spraying and splashing releases are the norm for liquids, rather than exceptions. A review
was made of major hazard accidents investigated by the US Chemical Safety Board and the
UK HSE. For petrochemical plant, less than 20% would be calculated by standard model
sets such as the Yellow and Green Books, or by commercial QRA calculation tools. For the
full range of major hazard accidents, only a small percentage would be calculated using
standard methodologies and tools, see table 2.
Choice of methodology

Methodologies for QRA specify which scenarios should be calculated, and often also many
of the parameters which should be used. Typical methodologies in use today are the World
Bank guideline(ref. 18), the Dutch purple Book (ref. 5) and the CCPS guideline (ref.19). For
this study, characteristics of methodologies were investigated in order to determine the effect
of methodology choice on results (ref. 21). The parameters investigated were:

• Choice of scenarios, just leaks and spontaneous ruptures, or a full range of scenarios
which could be identified by hazop studies such as overpressuring, overflow, confined
explosion etc. To keep the analyses comparable, the lacunae scenarios described in
the previous section were not included.
• The range of hole sizes considered (two, three, four or five, as provided for in
different methodologies. To support this study, an analysis of sizes for several
hundred cases was made).
• The location of holes along pipes (one representative location, two, three or four)
• The selection of vessels to be analysed, and whether each vessel should be
analysed, or groupings of vessels in an isolatable section.
• Whether to include domino effects at all, to include just calculation of the frequency of
domino effects, or the demanding calculation of both the sizes and frequency of
domino effects.
• Whether to include mitigation calculations in the methodology.

Some conclusions from these comparisons are:

• Focussing on a standard tables of release frequencies given in the literature as the

basis of the methodology results in overlooking a large fraction of the total, and
worse, makes it very difficult to make rational recommendations on risk reduction.
Proper ALARP analysis including prevention measures requires that the accident
scenarios in the QRA are related to hazop or high quality hazid studies.
• Applying vessel size cut offs, so that only the larger vessels are calculated, or
analysing only for isolatable sections, makes accident escalation calculations
impossible.
• Use of four hole sizes is sufficient to give reasonable accuracy in the risk result.
• Use of three locations for holes along pipes is sufficient to achieve reasonable
accuracy for in plant analyses, with pipe runs typically 10 to 50 m. The result depends
on the use of usual pipe sizing and pump sizing rules. More locations may be needed
to allow accurate assessment of escalation. For long inter unit pipes with large
pressure drops, pipeline release profiles may be needed.
• Domino effect calculations have moderate effect on risk to the public, increasing risk
by only about 50%. This compares well in comparison with the 100 largest incidents
published by Marsh (ref. 21) showed that only about 1/3 of these involved escalation.
Domino effects though are critical for determining safe evacuation distances and fire
fighting procedures. For petrochemical plant asset risk, domino effects dominate the
risks.
• Since the ALARP criterion is increasingly being used, mitigation calculations are
essential.
• The biggest source of inaccuracy in analyses is the accident types which are
overlooked or neglected.
Most of these conclusions imply a considerable increase in the amount of analysis work.
However even the most demanding of the analyses with all vessels and major pipes in a
refinery (450 vessels and associated piping, 200 inter-unit pipes), with four hole size classes,
three hole locations for each vessel and pipe required only a few hours of calculation on a
modern laptop computer. (The calculations are automated, but not “black box”. All results for
each stage of the calculations are tabulated).

Conclusions

Quantitative risk assessment has become an important tool in plant safety engineering.
Earlier guidelines such as refs. 8 and 18 have provided an open and well researched basis
for the use of QRA. These methodologies, however, were developed during the 1990’s for
land use planning. The present study indicates that accuracy can be much improved by
using more recent models, using more thorough methodologies, and particularly, increasing
the scope of accident scenarios covered.

Can this make the analyses truly accurate, to the same level as other engineering
disciplines? It appears difficult to improve models for gas dispersion much until better
experimental results become available, which will need better instrumentation that used until
now; or possibly, improvements in CFD techniques will allow a priori calculation of
dispersion. Similar problems arise for explosion calculations. For now though, “factor of two
accuracy” seems the best to be achievable for most consequence calculations.

It also appears possible to improve frequency calculations, though factor of two accuracy
seems difficult to guarantee except under the best circumstances. Factor of five accuracy
seems achievable in a wide range of cases. Uncertainty bands in results have also narrowed
considerably as more data have become available.

Is accuracy really necessary? Could it be just accepted that repeatability and consistency is
sufficient, as is the case for regulatory models? After all, the criteria for risk acceptance can
always be adjusted to take into account an estimated factor of error. The main problem with
this approach is that it does not provide a good basis for risk reduction and ALARP analysis.
Also, the regulatory model approach is very inflexible, which means that it will lead to hazard
types being neglected unless the model set is complete. The most important improvement is
to make sure that the accident scenarios calculated are the ones which occur in practice.

The price for improvement in accuracy is that more thorough assessments are required. The
additional cost of calculations themselves is not a significant factor. The collection of data as
a basis for the calculations, including process parameters for all vessels containing
hazardous materials in a plant, however, represent a significant, though manageable effort.

References

1. Amendola, Contini, Ziomas, Uncertainties in Chemical Risk Assessment,: Results of

a European bench mark exercise J. Hazardous Materials, 1992
and Christou, M.; Lauridsen, K.; Amendola, A.; Markert, F.; Kozine, I.O.,
ASSURANCE: Assessment of uncertainties in risk analysis of chemical
establishments. In: Proceedings. Vol. 1. 5. International conference on probabilistic
safety assessment and management (PSAM 5), Osaka (JP), 27 Nov - 1 Dec 2000.
 Kondo, S.; Furuta, K. (eds.), (Universal Academy press, Tokyo, 2000) (Frontiers
Science Series) p. 369-374
2. B.J.M Ale et al., Benchmark risk analysis models, RIVM report 6100660015/2001
3. HSE (2002), “Offshore Hydrocarbon Release Statistics 2001”, HID Statistics Report
HSR 2001 002, Health & Safety Executive
4. Rijnmond public Authority Risk Analysis of Six potentially hazardous Objects in The
Rijnmond Area, Reidel, 1982
5. B Ale et al. Guidelines for quantitative risk assessment, Purple Book, Director
General for Social Affairs and Employment 1999
6. J.R.Taylor Hazardous Materials Release and Accident Frequencies for Process Plant,
Taylor Associates, 2004
7. Fryman, C.E., A Screening Methodology for Assessing Adequacy of Blast Protection
for Control Rooms and Other Onsite Buildings, Report No. 1993, 221031, BP
International Ltd, Sunbury, 1993, cited in Guidelines for Evaluating Process Plant
Buildings for External Explosions and Fires, CCPS, 1996
8. J.R.Taylor, Human Error in Process Plant Operations and maintenance, QRA Quality
Project, Vol 8, 2009
9. J.R.Taylor, Understanding and combating design error in process plant design, Safety
Science (2006), doi:10.1016/j.ssci.2006.08.014
10. TNO (1996), “Yellow Book”, Methods for the Calculation of the Effects of the Escape
of Dangerous Materials, Dutch Ministry of Labour, 1978, 1996
11. J.Taylor QRA Pro, Technical Reference Manual 7 edition, 2007
12. Rew P J, Deaves D and Maddison T (1995), Current Status and Advances in Flash
Fire Modelling”, Proceedings of International conference on modelling and mitigating
consequences of accidental releases of hazardous materials, CCPS, New Orleans.
13. Pipeline Rupture and Subsequent Fire in Bellingham, Washington June 10, 1999,
NTSB/PAR-02/02
14. Lees, F.P., Loss Prevention in the Process Industries, Butterworth/Heinemann
15. Buncefield Major Incident Investigation Board, The Buncefield Incident 11 December
2005, HMSO 2008
16. Lechaudel J.F. and Mouilleau Y. Assessment of an accidental vapour cloud explosion
- A case study : Saint Herblain, October the 7th 1991, France, Proceedings of the 8 th
International Loss Prevention Symposium, pp 333-348, Antwerp,Belgium.
17. J. Mansot, The Blaze at Shell Storage at Port Edouard Heriot Harbour, 2 and3 June
1987, 6th Int. Symp. Loss Prevention and Safety Promotion in the Process Industries
18. Kayes, P.J. (ed.), 1985: Manual of industrial hazard assessment techniques. World
Bank, London, England
19. Chemical Process Risk Assessment, CCPS, 2000
20. Marsh McLennan, 100 Largest Accidents, 1985 and 2002 editions
21. J.R.Taylor, The QRA Quality Project Vol 1 to 7, 2005 to 2008, Available on request