A methodology for evaluating the performance of seismicity indicators at El Teniente Mine,

Chile.

Javier Vallejos
Department of Mining Engineering, University of Chile
Advance Mining Technology Center, University of Chile
Rodrigo Estay
Department of Mining Engineering, University of Chile
Rodrigo Zepeda
El Teniente Division, Codelco, Chile
Patricio Jorquiera
El Teniente Division, Codelco, Chile

Abstract
In many Block Caving mines, induced seismicity and the associated rock burst problem must be managed
to ensure the safety of mine workers and production. As a part of a study to provide an integral approach
for managing the effects of seismicity at El Teniente Mine, a methodology for evaluating the performance
of seismicity indicators was developed. The procedure relies on accepted statistical techniques
(contingency matrix, skill scores) for measuring the forecasting efficiency of a particular seismicity
indicator. The developed methodology is applied to evaluate the performance of the seismic alert indicator
currently used at El Teniente Mine. This indicator is based on event frequency, and is used to delineate
areas in which the seismic patterns are changing, which may lead to an uncontrolled response of the rock
mass. The currently used seismic indicator presented a state of alert in 56% of the relevant magnitude
cases studied previous their occurrence. The presented methodology is currently being applied to other
seismic indicators to evaluate their effectiveness and improve the alert criterion used at El Teniente Mine.
Introduction
A common characteristic of deep Block Caving mines in hard rock is induced seismicity. This results from
stress changes and rock failure around mining excavations. Some seismic events cause damage to
excavations and injury to personnel due to ejected rock, and are called rockbursts. The unpredictability of
seismic events makes management of their effects difficult. To ensure the safety of mine workers and
production, it is necessary to establish a procedure that allows managing the seismic activity effects. In
this context, some indicators have been developed to summarize the seismic response of the mass rock to
the applied mining conditions. For example, the decay patter in seismicity following blast and large
magnitude seismic events has been used to develop guidelines for the development of re-entry protocols
(Vallejos and McKinnon, 2008, 2009, 2010, and 2011). The purpose of a seismic indicators is to generate
a state of alert when the seismic patterns are changing, which may lead to an uncontrolled response of the
rock mass. An alert condition indicates that the seismicity indicator history should be followed in order to
perform the corresponding actions in the mining process for modifying the response of the seismic system.
In this paper a study related with the forecasting performance of seismicity indicators at El Teniente mine
is presented.

El Teniente Mine overview

El Teniente Mine is the world largest copper-molybdenum underground mine owned by Codelco, Chile. It
is located in the Andes range in the central zone of Chile, about 70 km South-Southeast from capital city,
Santiago. The current production is 140,000 tonnes/day. Sectors in operation are shown in Figure 1. In
particular, this study focuses on Reservas Norte (RENO) sector.

Figure 1: Sectors in operation at El Teniente mine (Daz & Morales, 2008)

Figure 2 presents the rockbursts occurrences at El Teniente mine during the period 1982-2009 in
connection with the effective production in tons per day (tpd). It can be observed that the production has
increased over the years but the number of rockbursts has decreased. This is a direct effect of the
geomechanical and operational safety related actions implemented at the mine.
 Zone
Teniente 4 Sur

Number of rockbursts
x103 tons per day
Teniente Sub 6
Isla Martillo
Esmeralda
Isla LHD

Year

Figure 2: Rockbursts v/s production for years 1982 to 2009.

Current seismic alert indicator

The seismic alert indicator currently used at El Teniente Mine is based on event frequency that occurs in a
given polygon/zone. The mean seismicity frequency is defined by

(1)

Where is the number of events occurred in a time period . This indicator involves the comparison of
the seismic event frequency occurred in a period of time with a seismic band which is defined as
the normal levels of seismicity (Figure 3a). The upper and lower bands are defined using the event
frequency occurred in a period of time (with ), and the 2% and 95% percentiles of the
Poisson probability function (Figure 3b). If lies outside the bands then the system is on state of alert.
For short term day, days is used, while for the medium term days,
days. The process is repeated and updated every 15 minutes.
Cumulative ascending distribution
Number of events

Date
Figure 3: Seismic alert indicator currently used at El Teniente mine. (a) Comparison between the
daily mean event rate and a seven day Poisson band. Cyan lines are upper and lower b, blue line is
the 7 day mean event rate and red line is the daily mean event rate (b) Determination of the lower
(LI) and upper (LS) bands for a 7 day mean event rate of 32 events/day.
The analysis is performed by selected all events in a previously defined polygon/zone. Figure 4 shows
polygons and levels defined for analyzing the RENO sector.

Figure 4: Definition of: (a) polygons and (b) levels, for the analysis of the RENO sector.

The study period consists of four years with approximately total of 280,000 seismic events (Table 1).

Table 1: Number of events by polygon and year.

Number of events
Polygons
2007 2008 2009 2010 Total
1 8,291 4,524 14,939 81,704 109,458
2 22,879 13,225 24,577 36,593 97,274
3 12,774 4,643 6,650 21,737 45,804
4 2,659 943 2,713 21,010 27,325
Total 46,603 23,335 48,879 161,044 279,861

Seismic indicator assessment

Condition times
Figure 5 shows the definition of the time intervals of possible states according to the actual seismic alert
indictor. Three possible scenarios are identified: alert by high t i , alert by low t i , and normal t i .
aa ab n

The inspection of the resulting cumulative distribution functions of t iaa and t iab , indicated that 15%,
50% and 95% of the population of time intervals on state of alert by high are lower than 0.25, 2.3 and 40
hours respectively, while for the state alert by low are 0.25, 2.3 and 49 hours, respectively. This specifies
that only 15% of the cases where the seismic indicator is on alert (by high or low) the duration of such
alert is just 0.25 hours, which is coincident with the updating time window of t 15 minutes.
 Numbers of events

Date

Figure 5: Definition of the time intervals of possible states: alert by high t iaa , alert by low t iab ,
and normal t in .

The resulting total time periods for each possible state are indicated in Table 2. This analysis shows that
for the time period of analysis (T), the seismic alert indicator remains at an alert condition (low and high)
33% of the time. Additionally, it can be observed that the total time of states of alert by high is higher than
the total time of state of alert by low. This indicates a lack of symmetry in the selection of the percentiles
of the Poisson distribution for the definition of the bands and the alert condition.

Table 2: Resulting total time periods for each possible state.

2007 2010 (T=1454 days)

Polygon T aa t iaa T ab t iab T n t in
(days) (days) (days)
1 264 246 944
2 311 277 866
3 284 229 941
4 176 102 1177
Mean 25959 21377 981135

Startup time alert-relevant magnitude event

The purpose of this section is to develop statistics for the time between the start of a state of alert and the
occurrence of a seismic event of relevant magnitude (Moment magnitude, M w 0.7). Figure 6 shows
schematically the time between the start of a state of alert and the occurrence of a seismic event of
relevant magnitude t a R .
 Seismic events / day

Date

Figure 6: Time elapsed between the onset of the seismic alert and the occurrence of a seismic event
of relevant magnitude. The colors of the lines correspond to those described in Figure 3.

For the RENO sector 99 events of relevant magnitude were identified, 46 of which occurred during states
of alert (hits). The probability of detection of seismic events of relevant magnitude during an alert
condition for the current seismic indicator is 49/99=46%. Table 3 presents a summary of these events.
Note that in this evaluation all hits have the same score, regardless if they occurred with t a R =0.25
hours.

Table 3: Seismic events of relevant magnitude that occurred during alert condition. RENO sector.

Seismic events Hits

Polygon Alert Alert Hit ratio
( M w 0.7) Total
by high by low
1 25 15 4 19 19/25 = 0.76
2 48 16 4 20 20/48 = 0.42
3 5 1 1 2 2/5 = 0.40
4 21 4 1 5 5/21 = 0.24
Total 99 36 10 46 46/99 = 0.46

Table 3 indicates that a higher number of hits are associated with the state of alert by high (36/46=78%)
compared to the state of alert by low (10/46=22%). However, this result indicates that the alert condition
by low is a necessary component of the seismic alert criterion. The cumulative distribution function of
t a R indicates that 26%, 61% and 74% of hits satisfies that t a R 8 hours, 6 hours and 24 hours,
respectively.
Forecasting performance
In order to quantify the forecasting performance of seismicity indicators, a methodology was developed
using accepted statistical techniques, such as the contingency matrix and skill scores. The seismic events
of relevant magnitude ( M w 0.7) are used as targets. The following four possible scenarios must be
considered (Figure 7a):
1. The seismic indicator is on alert and a relevant event did occur TP: True positive
2. The seismic indicator is on normal and a relevant event did occur: FN: False negative
3. The seismic indicator is on alert and a relevant event did not occur: FP: False positive
4. The seismic indicator is on normal and a relevant event did not occur: TN: True negative
These four scenarios are grouped to build the contingency matrix (Figure 7b).

(a) 1: TP (b)
2: FN
3: FP
Numbers of events

4: TN

Date

Figure 7: (a) Example of the possible scenarios considered for the evaluation of the contingency
matrix. The red arrows indicate the occurrence of seismic event of relevant magnitude. The colors
of the lines correspond to those described in Figure 3. (b) Contingency matrix

The following performance metrics are associated to the contingency matrix (Fawcett, 2006):

TP
TPR
TP FN
(2)
FP
FPR
FP TN

TPR determines the percentage of significant magnitude events that occur during states of alert, while
FPR defines the percentage of false alarms from all negatives occurrences (FP + TN). A perfect indicator
satisfies that TPR=1 and FPR=0. Also, the following Skill Scores (Peirce, 1884; Heidke, 1926) are used to
evaluate the forecasting performance of seismicity indicators:

Peirce: PSS TPR FPR

(3)
2TP TN FP FN
Heidke: HSS
TP FN FN TN TP FP FP TN
These metrics provide a direct comparison between the method used and a random one, taking on values
between -1 and 1. Values equal to 1, 0 and -1 indicate that the method is perfect, random or worse than
random, respectively.

For quantifying the scenarios of the contingency matrix the following methodology was developed:
1. For a given time t i determine whether the indicator is on alert or not.
2. Verify the occurrence of relevant magnitude events in a time window ti t f , with t f fixed.
3. Assign a value of 1 to the corresponding scenario of the contingency matrix, according to the
following criteria:
If the seismic indicator is on a state of alert at t i and a relevant magnitude seismic event
occurred on t i t f , then TPi = 1.
If the seismic indicator is on a normal state at t i and a relevant magnitude seismic event
occurred on t i t f , then FNi = 1.
If the seismic indicator is on a state of alert at t i and a relevant magnitude seismic event
do not occurred on t i t f , then FPi = 1.
If the seismic indicator is on a normal state at t i and a relevant magnitude seismic event
do not occurred on t i t f , then TNi = 1.
4. Time is updated. i.e. ti ti t and return to 1.
With this methodology it is possible to assign scores to the four scenarios of the contingency matrix, and
evaluate the effectiveness of an indicator of seismic events of relevant magnitude in a future window of
time t f . In case of El Teniente mine t f 8 hours and t 15 minutes was used.

Table 4 presents the results of the assessment using the methodology described above. 56% of the relevant
magnitude event studied cases presented a state of alert before their occurrence. However, the Skill Scores
indicate that the current indicator is comparable to a random rule, i.e., when the indicator is on alert
condition, it is not possible to state that a relevant magnitude event will occur. This low performance is
associated with the high number of false alarms (FP) detected.

Table 4: Contingency matrix and performance metrics. RENO sector.

Polgono TP FN FP TN TPR FPR PSS HSS
1 501 278 48448 90356 0.64 0.35 0.29 0.01
2 832 514 55607 82630 0.62 0.40 0.22 0.01
3 74 86 49188 90233 0.46 0.35 0.11 0.00
4 248 399 26381 112555 0.38 0.19 0.19 0.01
Total 1655 1277 179624 375774 0.56 0.32 0.24 0.01
Concluding remarks

A methodology for evaluating the performance of seismic indicators has been presented. The proposed
approach has the following three statistics:
1. Condition times
2. Startup time alert-relevant magnitude event
3. Forecasting performance
These statistics are relevant for evaluating the performance of competing seismic indicators. In addition,
alert thresholds can be calibrated by optimizing the performance of the seismic indicator evaluated.

The developed methodology was applied to evaluate the performance of the event frequency seismic alert
indicator currently used at El Teniente Mine, Sector RENO. The following conclusions were provided:
1. The indicator remains at an alert condition 33% of the time.
2. A higher number of relevant magnitude events are associated to the state of alert by high (78%)
compared to the state of alert by low (22%).
3. 56% of the relevant magnitude event studied cases presented a state of alert before their
occurrence (Hits). 26% of these hits occur within a time period of 8 hours measured from the
onset of alert.
4. Skill Scores indicate that the current indicator is comparable to a random rule, i.e., when the
indicator is on alert condition, it is not possible to state that a relevant magnitude event will occur.
This low performance is associated with the high number of false alarms (FP) detected. However,
the scope of an alert procedure has to be reminded. They are mainly used to to delineate areas in
which the seismic patterns are changing, which may lead to an uncontrolled response of the rock
mass.
The presented methodology is currently being applied to other seismic indicators to evaluate their
effectiveness and improve the alert criterion used at El Teniente Mine.

Acknowledgements
The authors wish to acknowledge the permission of the mine to publish this work.

References
Daz, G., & Morales, E. (2008). Tunneling construction for 140.000 tonnes per day El Teniente mine
Codelco Chile. In: 5th International Conference and Exhibition on Mass Mining, Lulea, Sweden, 83-96
Fawcett, T. (2006). An Introduction to ROC Analysis. Pat. Rec. Lett., 27, 861-874.
Heidke, P. (1926). Berechnung des Erfolges und der Gte der Windstrkevorhersagen im
Sturmwarnungsdienst. Geogr. Ann., 8, 301-349 (In German).
Peirce, C.S. (1884). The numerical measure of the success of predictions. Science, 4, 453-454.
Vallejos JA, McKinnon SD. Guidelines for development of re-entry protocols in seismically active mines.
In: Proceeding of the 42nd US Rock mechanics symposium, San Francisco, California, ARMA/USRMS,
2008, paper 08-97.
Vallejos JA, McKinnon SM. Re-entry protocols for seismically active mines using statistical analysis of
aftershock sequences. In: Rock engineering in difficult conditions. Proceedings of the 20th Canadian rock
mechanics symposium, Toronto, Canada, 2009, paper 4028.
Vallejos JA, McKinnon SD. Omoris law applied to mining-induced seismicity and re-entry protocol
development. Pure Appl Geophys 2010; 167(1):91-106
Vallejos, J.A. and McKinnon, S.D. (2011). Correlations between mining and seismicity for re-entry
protocol development. International Journal of Rock Mechanics & Mining Science, 48, 616-625.