Anda di halaman 1dari 86

Enterprise Survivable Servers (ESS): Architectural Overview

written by Greg Weber and Aaron Miller

Section 1 - Introduction
1.1 What is ESS ?
Enterprise Survivable Servers (ESS) is an offering which takes an existing Avaya Communication Manager (CM) system to a higher level of availability and survivability. ESS achieves this by allowing media servers to be used as alternate controllers within a system by leveraging IP control of port network gateways and being completely independent of the main servers both functionally and geographically. ESS protects the communication system against a catastrophic main server failure and, at the same time, provides service to port network gateways that have been fragmented away from their current controlling entity.

1.2 How is this paper organized ?


Section #1 The Background section gives the technical background needed to fully understand and appreciate ESS by going into a deep dive of how ACM works today in a non-ESS environment. It provides detailed definitions of some basic infrastructure terms and goes into tremendous depth of how media servers control port network gateways. If the only objective of reading this paper is to obtain a high level overview of ESS, then this section can be skipped. However, if the desire is to understand the whys of the ESS operation, then this section builds that foundation. Section #2 The Reliability Single Cluster Environments section covers many of the methods service can be supplied to a port network gateway while still only having a single controlling cluster. The ESS offering does not replace these methods, but rather builds on top of them. The methods described in this section attempt to restore service if faults occur within the communication system. Only if all of these methods, including server interchanges, IP Server Interface (IPSI) interchanges, control fallback, and more, are unsuccessful in restoring service will the ESS offering begin to take effect. Section #3 The Reliability Multiple Cluster Environments section covers some of the current methods of protecting the communication system against main server failures. All of these offerings, such as Survivable Remote Processors (SRP), ATM WAN Spare Processors (ATM WSP), and Manual Back-up Servers (MBS) are being replaced by ESS. This section provides a brief overview of the operation for each one of these offerings, making it clear that ESS provides at least the same level and in most cases a higher level of protection. Section #4 The ESS Overview section introduces the ESS offering by showing, at a high level, how ESS servers increase overall system availability by providing protection against catastrophic main server failures and extended network fragmentations. In addition, this section emphasizes that the ESS offering is one of last resort and that it works in conjunction with all existing recovery mechanisms by reviewing a number of failure scenarios which are resolved by methods other than ESS. Section #5 The How Does ESS Work section does a deep technical dive into the operational steps required to make ESS operate. The section describes how ESS servers register with and acquire translation updates from the main cluster. Furthermore, it also covers, in detail, how IPSIs prioritize various ESS clusters within a system and when it will use this ranking to get service. The information in this section gives the hows and whys of the ESS offerings operation.

Section #6 The ESS in Control section examines what happens to existing calls when port networks failover under ESS control. While there is no feature debt incurring or performance degradation running under ESS control, there are subtle operational differences. The section concludes with a discussion of these differences including possible call-flow changes and access to centralized resources and voicemail. Section #7 The ESS Variants Based on System Configuration section inspects the subtle differences that occur when a system needs to utilize ESS clusters for control based on various port network connectivity configurations. For each type of port network connectivity configuration available, failure examples, including catastrophic server outages and severe network fragmentations, are simulated and thoroughly explained. Section #8 The ESS in Action - A Virtual Demo section allows the reader to experience the ESS offering in a real world environment. The demo is given on a communication system that consists of three geographically separate sites interconnected in various ways. After a thorough explanation of the setup and how the system operates in a non-faulted environment, catastrophic server failure and network fragmentations are introduced. Each stage of the demo is clearly explained and the viewpoints of the system from the main servers and the ESS servers are shared through administration status screens. The demo concludes with an execution of the steps required to restore the PBX switch back to normal operation after the networks is healed and the main cluster fixed. Section #9 The More Thoughts and Frequently Asked Questions section covers various topics involving the ESS offering. These topics include details about license files, re-use of MBS servers as ESS servers, the alarming differences between main servers and ESS servers, how non-controlling IPSIs work with ESS, and much more.

Section 2 - Background
2.1 What is an Avaya Media Server ?
The Switch Processing Element (SPE) is the controlling entity of an Avaya Communcation System. The SPE responsibilities range from making any and all intelligent decisions to control of every endpoint within the switch. In a traditional DEFINITY PBX, the SPE software resides on proprietary hardware (UN/TN form factor circuit packs) and runs on a proprietary operating system called Oryx/Pecos. Avayas next generation Communication System needs to have the ability to efficiently integrate faster processors as they become available. One of the main objectives of the Avaya Communication Manager was to transport this powerful, feature-rich, reliable SPE onto an off-the-shelf server running an open operating system. This objective was realized by the migration of the SPE onto media servers running the Linux operating system. The following table shows the different types of Avaya media servers which are currently available.
Server Name S8720 S8710 S8500 S8700 S8400 S8300 Chip Set AMD Opteron 2.8 GHz Intel Xeon 3.06 GHz Intel Pentium-4 3.06 GHz Intel Pentium-3 850 MHz Intel Pentium-M 600 MHz Intel Mobile Celeron 400 MHz Duplicated Yes Yes No Yes No No Port Network Support Yes Yes Yes Yes Yes No

Table 1 Types of Media Servers

In the table above, the server types are listed in descending order of processing power. This is a function of the chip set on which they run. Duplication is another distinguishing property of the server types. The S87XX series of media servers are always duplicated, where each server in the server pair has an SPE running on it. The standby SPE, running on one of the servers, is backing up the active SPE on the other server in the pair. Refer to the What is a Server Interchange section for more information on server duplication.

The Avaya product line has many different types of gateways including H.248 gateways, H.323 gateways, traditional DEFINITY cabinetry, and 19 rack mountable cabinets. Port networks (explained in the section below), which are made up of traditional DEFINITY cabinets and 19 rack mountable cabinets, are not supported by all media servers. Since the ESS project is designed to provide survivability for port networks, the S8300 media server will be omitted throughout the rest of this paper unless specifically noted since it does not support the ESS offering. Also, the scope of the ESS product does not include support for the S8400 media server at this time.

2.2 What are Port Networks ?


All endpoints connect to the communication system through circuit packs when utilizing the S8720, S8710, S8700, or S8500 Media Servers. For example, digital phones are wired directly to digital line circuit packs (TN2124), PRI trunks terminate at DS-1 circuit packs (TN464), and IP endpoints utilize CLAN circuit packs (TN799) as their gatekeepers. All of these TN form factor circuit packs are housed in cabinets. These cabinets, listed in the table below, support circuit packs by providing them power, clocking, and bus resources. The bus resources, a time division multiplexed (TDM) bus and a packet bus, are used as both a control conduit to the circuit packs and a bearer interconnectivity medium.
Cabinet Type MCC/SCC G600 G650 Duplicated IPSI Support Yes No Yes EI Support Yes No Yes Rack Mountable No Yes Yes

Table 2 Types of Cabinets

A port network is the grouping of cabinets that have physically connected buses or, in other words, a port network is a collection of cabinets that share the same TDM bus and the same packet bus. In addition, since all circuit packs within a port network share a TDM bus, they need to be synchronized. For that reason, each port network contains one active tone clock residing on either a dedicated tone clock board (TN2182) or an IP Server Interface (IPSI) card (TN2312, described later in this section).

2.3 How are Port Networks Interconnected ?


The interconnected buses, within a set of cabinets making up a port network, provide a direct communication medium between all endpoints associated with that port network. However, endpoints in disperse port networks also need to be able to establish a communication medium when needed or, in other words, there needs to be support of Port Network Connectivity (PNC). The PNC provides an interconnection of port network buses when required. There are a variety of PNC types supported by ACM and are shown in the table below.
PNC Type Center Stage Switch (CSS) Direct Connect ATM IP Port Network Interconnectivity Device EI Board (TN570) EI Board (TN570) ATM EI Board (TN2305/6) MEDPRO (TN2302) Ability to Tunnel Control Yes Yes Yes No Resource Manager SPE SPE ATM Switch IP Network Max # of Port Networks 45 3 64 64

Table 3 Types of PNC

When a communication path is needed between two port networks, the SPE will set up the path. For the PNC types which have the SPE as the resource manager such as Center Stage Switch (CSS) and direct connect, the SPE will not only inform the Expansion Interface (EI) boards which path to use, but also create the path through the PNC. For example, to set up a communication path between two port networks using the CSS PNC, the SPE will inform the EI in port network #1 to use path A, set up path A through the CSS, and inform the EI in port network #2 to also use path A. When the SPE needs to create the actual communication path, the PNC is referred to as SPE managed. For the PNC types which do not have the

SPE as the resource manager such as ATM and IP, the SPE will only need to tell the ATM EI boards (or MEDPRO boards) to interface with their peer ATM EI board (or MEDPRO board) and the creation of the actual path is left to either the ATM switch (if ATM PNC) or the IP network (if IP PNC). For example, to set up a communication path between two port networks using the ATM PNC, the SPE will inform the ATM EI in port network #1 to communicate with the ATM EI in port network #2 and the ATM switch figures out how this is done. When the SPE does not need to create the actual communication path, the PNC is referred to as self-managed.
PN #1
EI

PN #2
EI

PN #1
EI

PN #2
EI

PN #1
EI

PN #2
EI

PN #1
MEDPRO

PN #2
MEDPRO

SNI

SNI

ATM SWITCH ATM PNC

IP NETWORK IP PNC

Direct Connect

Center Stage Switch

Figure 1 Connectivity of Different Types of PNCs

2.4 How are Port Networks Controlled ?


As described above, a port network consists of a number of circuit packs and two transport buses, a TDM bus and a packet bus. Each circuit pack is locally controlled by resident firmware called an angel. Therefore, in order to utilize the functions of a circuit pack, the SPE needs to interface with that circuit packs angel. Some circuit packs contain enhanced angels, specifically EI boards and IPSI boards (described later in this section), which have the capability of becoming an arch-angel when activated. An activated angel, or arch-angel, provides the communication interface to all other angels in the port network over the TDM bus for the SPE. Hence, in order for the SPE to control all the circuit packs and the TDM bus in a port network, the SPE must be able to establish a communication path to the port networks archangel. Some other circuit packs in the system, such as CLAN boards for example, need additional control links for various reasons. These control links traverse the port networks packet bus in the form of Link Access Protocol Channel-D (LAPD) links to the circuit pack. The SPE utilizes a Packet Interface (PKTINT) to manage the port networks packet bus and terminate the other end of the LAPD links. Therefore, in order to control these types of circuit packets, a LAPD link needs to be established between the PKTINT and the circuit pack and the SPE needs a communication path to the PKTINT. The communication path between the SPE and the port networks arch-angel takes the form of a special LAPD link called an Expansion Arch-angel Link (EAL). An EAL begins in the PKTINT and terminates at the port networks arch-angel. Consequently, the SPE can control everything it needs to be in command of in a port network, all control links and both buses, if it has a communication path to a PKTINT that can serve that port network. In a traditional G3r DEFINITY system, the SPE and the PKTINT both reside in the Primary Port Network (PPN) and are connected via the carriers backplane. The PKTINT serves all of the Expansion Port Networks (EPN) by having EALs go through the PNC (either CSS, ATM, or direct connect) and terminate to arch-angels on each port network. EI cards contain these enhanced angels which can be activated into arch-angels. The figure below shows how the SPE located in a DEFINITY PPN can control its EPNs.

SPE PPN PKTINT EI


EAL #1 EAL #2

EPN #1 EI

EPN #2 EI

PNC

Figure 2 SPE Control of EPNs in Traditional G3r DEFINITY System

However, in the new CM configuration, the SPE no longer resides on the proprietary hardware in the PPN, but rather externally on the media servers. Therefore, the SPE cannot communicate with the PKTINT through the carriers backplane. To solve this problem, the Server Interface Module (SIM) was created to provide a physical connection to the PKTINT and an IP interface to communicate with the SPE. The IP Server Interface (IPSI) card (TN2312) is a conglomeration of a number of components mentioned thus far: a PKTINT, an enhanced angel (which can be activated into an arch-angel), a SIM, and a tone clock. The figure below shows how the SPE located on media server can control its EPNs.
SPE

LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #3

EPN #1 EI

EPN #2 EI

EPN #3 EI

PNC

Figure 3 SPE Control of EPNs in an ACM Configuration (IPSIs in all EPNs)

While the above figure shows every EPN in the system having an IPSI, this does not necessarily have to be the case. Every control link in the system goes between a PKTINT and a circuit pack containing an archangel. If all of the control links went through a single PKTINT, as it does in the traditional G3r DEFINITY system, a bottleneck would be created. However, the PKTINT resident on the IPSI card has more than enough capacity to support its own port network control links and control links for other EPNs. If the PNC supports tunneled control, then not every port network is required to have an IPSI for control. The figure below shows the control of EPNs with some having IPSIs and others not. Based on IPSI processing capacity and reliability issues, which is discussed in the next section, it is suggested that there be one IPSI for every five port networks with a minimum of two per system.

SPE LAN

IPSI
EAL #1

IPSI
EAL #2

EPN #1 EI

EPN #2 EI

EPN #3 EI

PNC

EAL #3

Figure 4 SPE Control of EPNs in an ACM Configuration (IPSIs in some EPNs)

2.5 What are SPE Restarts ?


The SPE can go through many different types of restarts which can be grouped into three categories cold restarts, warm restarts, and hot restarts. Each type of restart has different memory state requirements and different end-user effects. The most drastic restart of an SPE is a cold restart. The cold restart clears all memory which deals with call state and re-initializes the entire system. The end-user effect of a cold restart is that every call in the system is dropped and that all endpoints are reset. A cold restart has virtually no memory requirements in order to achieve this type of restart. A much more graceful SPE restart is a warm restart. This restart re-initializes some of the system, but keeps all memory which deals with call states intact. The end-user effect of a warm restart is much less severe than during a cold restart; all calls that are stable remain up, however, some stimuli (e.g. end-user actions or endpoint events) may be lost. This means that an end-user simply communicating on a call over an established bearer channel is unaffected, but an end-user in the process of dialing may lose a digit. Therefore, a warm restart has a memory requirement where all the call state information is valid and up-todate in order to preserve calls. While a warm restart has some minor effects on end-users, a hot restart of the SPE does not. A hot restart does not alter or clear any memory and will only restart the SPE software processes using the intact memory. There is no end-user effect of a hot restart; all calls are preserved and all stimuli occurring during the restart are received by the SPE and appropriate actions are taken on them. To accomplish this, a hot restart has the requirement that all of its memory must remain completely intact through the restart. The following table summarizes all the different types of restarts.
Restart Type Hot Warm Cool Cold Reboot End-user Effect None Minimal Minimal High High Stable Calls Survive ? Yes Yes Yes No No Approximate Time for Completion <1 second 5-9 seconds 45-60 seconds see below see below Executable ACM Command reset system 1 reset system 2 reset system 3

Table 4 SPE Restart Summary

The cool restart introduced in the table above is a form of a warm restart that can be used across different versions of software. It allows the SPE to be actively running and get reset into a new software version with all stable calls remaining up through the restart. A cool restart is the mechanism to upgrade the SPEs release without losing any calls. As previously stated, cold restarts clear all call state memory but, based on the level of the cold restart, it may or may not reload the translations. A low level cold restart clears all call state memory, while keeping its translation memory intact. A high level cold restart, or a reboot, clears both the call state memory and reloads translations. This implies that if an administrative action is done on the switch, it will be preserved through a cold restart. However, if the same administrative action is done and the translations are not saved before a reboot occurs, the changes are lost since the translations are reloaded from the disk. Restart times are measured from the beginning of the restart until all endpoints are operational and the SPE is again processing end-user stimuli. Hot restart, warm restarts, and cool restarts never have any endpoints go out of service and need only wait for the SPE to begin processing once again. This has the implication that the restart time for these types of resets does not vary too much on the size of the system or the number of endpoints and therefore approximate restart times can easily be determined without any knowledge of the system. Cold restarts and reboots reset all endpoints and cannot be deemed completed until all of them are operational. Determining an approximate restart time mainly depends on three major factors the types of endpoints (analog, digital, BRI/PRI, IP ), how many of each is in the system, and the processing speed of the server. For very small systems running on S87XX media servers, the restart completion time is on the order of minutes, one to three. For larger systems with a low number of IP & PRI endpoints, the restart completion is approximately three to six minutes. For systems with a large number of IP & PRI endpoints, the system may not be fully operational for up to ten to fifteen minutes. During this time, all other endpoint types will be functioning while the remaining IP & PRI endpoints are coming back into service.

2.6 What are EPN restarts ?


The SPE restarts, described in the previous section, affect the entire system. For example, a cold restart of the SPE causes all endpoints in the system to be reset and every call to be dropped. SPE restarts are used as a recovery method for a subset of failures, such as a server hardware malfunction or a software trap. The type of failure dictates which restart level is needed for recovery. However, there are other types of failures which need recovery where an entire system restart is not warranted. For example, if a single port network has an issue which needs a restart to return to normal operation, then only that PN should be reset as opposed to the entire system. These types of restarts are referred to as EPN restarts. While EPN restarts are implemented differently than system restarts, the same classifications exist for them hot, warm, and cold. An EPN cold restart is the most extreme PN restart. In an EPN cold restart all circuit packs within the PN are reset which causes all associated endpoints to reset and all calls to be dropped. The basic concept of an EPN warm restart is to resynchronize the SPE and the EPN after some event occurs that caused them to become slightly out of sync in terms of resource allocation. That said, a warm restart of a PN preserves all established talk paths that it is supporting and does not cause its associated endpoints to be reset. However, any end-user actions that took place during the time of the fault through the completion of the EPN warm restart are lost. If, for example, an IPSI temporarily loses its connectivity with the SPE for less than 60 seconds, it requires an EPN warm restart to return to normal operation. Any calls in progress within and through that PN would be unaffected (unless of course the connectivity that was lost was also being used for the bearer traffic), but any end-user stimuli (e.g. dialing or feature activation) that happened during that time are lost. A hot EPN restart is used to interchange IPSIs within a PN in a non-faulted environment. IPSI duplication and IPSI interchanges are described in the next section in detail, but it is important to understand that there are no adverse effects on any endpoints or any talk paths through a hot EPN restart.

Section 3 - Reliability Single Cluster Environments


3.1 What is a Cluster ?
As described in the What is an Avaya Media Server section, some of the available servers can be duplicated. This duplication allows one server to run in active mode and the other server to run in standby mode. The What is a Server Interchange section below describes how the standby server could take over for the active server if needed. If the standby server is in a refreshed state immediately before the failure, it could take over without affecting any of the stable calls. In other words, the standby server shares call state with the active server via memory shadowing and can take over the system in a way that would preserve all stable calls. A cluster is defined to be a set of servers that share call state. In the case of a server which can only be simplex, the server itself is a cluster. In the case of a server which can be duplicated, the cluster is the combination of both of the servers.

3.2 How do single cluster environments achieve high availability ?


The basic structure of the Avaya Communication Manager consists of a decision maker (the SPE), an implementer (the port network gateways), and a communication pathway for the decision maker to inform the implementer what actions to carry out. If the SPE ceases to operate or fails to communicate with its port networks, then the PBX will be inoperable since the SPE makes all the intelligent decisions for the PNs. For example, if the SPE was powered off, then all of the control links to the arch-angels would go down. If an arch-angel has no communication path to the SPE, then all messages sent to it from angels on other boards in the port network would be dropped. If the angels on the TN boards cannot send messages anywhere to be processed, then endpoint stimuli get ignored. If all endpoint stimuli are dropped, then service is not being supplied to the end user. Higher system availability is achieved by providing backups to the systems basic structural components. In the case of the SPE, if the servers support duplication, then the standby SPE within a cluster can take over for its active SPE in fault situations. Once a stable active SPE is achieved, a control link is brought up to each port network. These control links can traverse different pathways to compensate for communication failures and can terminate at different locations to compensate for TN boards failures. IPSI interchanges and EI-control fallback are two methods of achieving this and are described in detail at the end of this section.

3.3 What is a Server Interchange ?


In the What is an Avaya Media Server section above, a number of server types were listed along with some of their characteristics. One of these characteristics was duplication. Understanding the catastrophic nature of an SPE failure, it is important to be able to provide SPE duplication to avoid a single point of failure. If the servers are duplicated, then one server runs in an active mode and the one server runs in a standby mode. The server in active mode is supporting the SPE which is currently controlling the Communcation System. The server in standby modes SPE is backing up the active SPE. If there is a failure of the active SPE, then the standby SPE takes over the system. One feature of duplicated servers is the ability to shadow memory from the active SPE to the standby SPE. One technique of achieving this is by having a dedicated memory board (DAJ or DAL board) in each server and interconnecting them with a single-mode fiber. This method of memory shadowing is referred to as hardware memory duplication. Another means for shadowing memory between a pair of servers is to transmit the information over a high-speed, very reliable IP network. This method of memory shadowing is referred to as software memory duplication and is not available for all types of media servers. For performance reasons and bandwidth considerations, only a percentage of the systems total memory is duplicated between the servers. During steady-state operation, any changes made to the active servers call state memory are transmitted to the standby server. After these transmitted changes are applied to the current state of the standby servers memory, the standby servers memory will be up-to-date with the active servers memory. The key, however, is that transmitted changes are applied to the current state of

the standby servers memory and for that reason, if the current state was not up-to-date with the active server before the changes were made, the transmitted changes have no context and therefore are useless. If a standby servers memory is up-to-date with the active server, the system is refreshed. If the standby servers memory is not up-to-date, the system is in a non-refreshed state. A standby server becomes nonrefreshed if the communication between the servers has a fault or if either SPE goes through a restart. If the system is non-refreshed, the active and standby servers go through a coordinated refreshing process to get the standby server up-to-date and prepared to accept transmitted changes. If the standby SPE is required to take over the system, it needs to transition to become the active SPE. The process of a standby SPE becoming active is done through an SPE restart. As described earlier, there are various types of restarts with each having different end user effects. The ideal restart level for an SPE interchange is a hot restart, whereby the interchange would be completely non-service affecting. However, this type of restart requires that the standby server have 100% of the memory state of the active server before the failure. Unfortunately, the system does not achieve 100% memory duplication which implies that a hot restart interchange is not supported between SPEs running on media servers. However, enough memory is shadowed to allow a warm restart during the interchange process. This implies that a standby SPE can take over for an active SPE through a system warm restart if the standby servers memory is current (or refreshed). However, if the standby server is not refreshed, the standby SPE can take over for an active SPE, but only via a cold restart. In other words, if a standby server is refreshed, then it can take over for an active server without affecting any stable calls in the system. If the standby server is not refreshed, it can take over the system, but the interchange process will cause all calls in the system to be dropped.
SPE A SPE B SPE SPE B

LAN

LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #1

IPSI
EAL #2

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #1 EI

EPN #2 EI

EPN #3 EI

EAL #3

EAL #3

PNC SPE A in Active Mode Figure 5 SPE Interchange

PNC SPE B in Active Mode

3.4 What is Server Arbitration ?


The decision to determine which server should be active and which server should be standby within a cluster is the job of the Arbiter. An Arbiter is local to each server within a cluster and instructs its coresident SPE which mode to run based on a number of state-of-health factors. Arbitration heart-beats are messages passed between two peer arbiters via the IP duplication link and the control networks (for redundancy) containing the state-of-health vector. The state-of-health vector is primarily comprised of hardware state-of-health, software state-of-health, and control network state-of-health. The hardware stateof-health is an evaluation of the media servers hardware (e.g. fan failures, major UPS warnings, server voltage spikes, etc.) done by the Global Maintenance Manager (GMM) and reported to the Arbiter. The

software state-of-health is an evaluation of the SPE software processes (e.g. process violations, process sanity failures, etc.) done by the Software Watchdog (WD) as well as the Process Manager and reported to the Arbiter. The control network state-of-health is an evaluation of how many IP connections to IPSI connected port networks the media server has compared to how many IP connections it expects to have. This information is determined by the Packet Control Driver (PCD) and reported to the Arbiter. While the decision tree that an Arbiter goes through has many complex caveats (including anti-thrashing measures, comparison algorithms for state-of-health vectors, tie-breaker procedures, etc.), it can be summarized by three basic rules: 1) If an Arbiter and its peer determine that both SPEs in a cluster are in active mode, one Arbiter in the pair, the one which has been active the longest, will instruct its associated SPE to go into standby mode. 2) If an Arbiter cannot communicate with its peer, it will instruct its associated SPE to go into active mode. 3) If an Arbiter has a better state-of-health vector than its peer, it will instruct its associated SPE to go into active mode (if it is not already) and ensure that the peer SPE is no longer running active mode.

3.5 What is an IPSI Interchange ?


In order for the SPE to be able to service a port network, a LAPD control link, an EAL, must be established and functioning. A port networks EAL begins at a PKTINT, which may or may not be present in its own carriers, and terminates at an arch-angel, which must reside within the port network itself. If a failure occurs on a PKTINT which supports the port network, another one must be found to support the EAL if the port network is to remain operable. There are two basic ways to find another supporting PKTINT for a port network - transition to another port networks PKTINT (which is referred to as fall-back and is discussed in the next section) or interchange to another PKTINT resident in the port network (which is referred to as an IPSI interchange and is described below). In the What are Port Networks section above, a number of cabinet types were listed along with some of their characteristics. One of the traits mentioned was the ability to support duplicated IPSIs which implies the ability to support duplicated PKTINTs since the PKTINT is one of the components of an IPSI. The SPE chooses which IPSI will be active and which IPSI will be standby within a pair. The active PKTINT supports all of the LAPD links (including the EALs and other control links) that are needed to service the port networks that it supports. The standby PKTINT is available to take over support of the LAPD links if needed. There are two types of IPSI interchanges that exist, each of which has different end user effects - a planned migration and a spontaneous interchange. A planned IPSI migration will not have any effect on the end users and can occur during periodic maintenance or via an administrators command. A planned migration occurs in non-fault situations and carefully transitions link support from the active PKTINT to the standby PKTINT in a method whereby no messages are lost. The standby PKTINT is transitioned to an active PKTINT via a hot restart. A spontaneous IPSI interchange will have no effect on stable calls, but will possibly drop end user actions during the switchover. A spontaneous IPSI interchange is an action that SPE takes if an active PKTINT failure occurs or a network fault occurs between the SPE and the active IPSI. A spontaneous IPSI interchange will inform the standby PKTINT of all existing links and then transition it to the active PKTINT via a warm restart. The left side of Figure #5 below shows the control links for port networks #1 and #2. Port network #1 (PN #1) has a resident IPSI and the controlling link for this port network, the EAL, goes from the IPSIs PKTINT to the IPSIs arch-angel. The other port network does not have its own IPSI and therefore is utilizing another port networks PKTINT for service (the next section explains how this is possible). PN #2s EAL goes from port network #1s IPSIs PKTINT to the EI board residing in its own carrier. When a failure occurs on the active PKTINT, a communication fault in this example, the SPE will transition over to the standby IPSIs PKTINT in PN #1. The right side of Figure #5 shows the new instantiations of the

10

control links after the spontaneous IPSI interchange. Notice that for the non-IPSI controlled port network, PN #2, the EALs near end termination point has shifted from the A-side PKTINT to the B-side PKTINT, but the far end remains at the same location, on the EI board. However, both termination points of the EAL for the IPSI controlled port network, PN #1, have shifted. Like the non-IPSI port networks EAL, the near end shifted from the A-side PKTINT to the B-side PKTINT, but unlike the non-IPSI port network, the far end has also shifted from the A-side IPSI to the B-side IPSI. Due to historical reasons, if both IPSIs and PKTINTs are healthy, SPE maintenance preferences back to the A-side IPSI (unless it is locked down to the B-side) even though there is no performance effect from being on one side or the other.
SPE SPE

LAN

LAN

IPSI-A

IPSI-B
EAL #1

IPSI-A EPN #2 EI
EAL #2

IPSI-B
EAL #1

EPN #1 EI

EPN #1 EI

EPN #2 EI
EAL #2

PNC A-Side IPSI Active

PNC B-Side IPSI Active

Figure 6 - IPSI Interchange

Also, the determination of which IPSI will be active and which will be standby is completely independent of which server is in active mode and which is in standby mode. Since there is no longer a tight association between the active server side and the active IPSI side (e.g. the B-side server can be in active mode and the A-side IPSI can be in active mode at the same time), the SPE can independently select which side (A or B) will be active for each IPSI pair (e.g. some IPSI pairs could have the A-side be active while others are concurrently active on the B-side).

3.6 What is EI Control Fallback ?


In the section above which describes how port networks are interconnected, different types of PNCs are discussed. Some of the PNC types not only support bearer traffic between port networks, but also provide a medium for tunneling control from one port network to another. This tunneling ability allows for some port networks to be supported by a PKTINT which is located in another port network. In these cases, the port network has an EAL control link which goes from another port networks PKTINT to an arch-angel residing on an EI board within the port network itself. Port networks which are being serviced in this fashion are referred to as EI controlled. Port networks which are being serviced from a co-resident PKTINT are referred to as IPSI controlled. EI fallback is a recovery method whereby a port network transitions from IPSI controlled to EI controlled due to a fault. If the port network has a simplex IPSI, a fault of that board will cause an EI fallback (provided the PNC type supports it). If the port network has duplex IPSIs, a fault of the active board will cause an IPSI interchange. However, if the newly activated board has an additional fault and the pair IPSI board has not returned to an error-free state, the port network will fallback to EI control.

11

The left side of Figure #6 shows three port networks getting controlled before the introduced fault. Port network #1 and port network #2 have simplex IPSIs and currently have their EALs go from their coresident PKTINTs to their co-resident arch-angels (both of which are contained within the IPSI). Port network #3 does not have an IPSI and is getting its control indirectly from the SPE through PN #2. In this example, if a fault occurs in PN #2s PKTINT, then the EALs for PN #2 and PN #3 need to transition off of the faulted PKTINT. The near end of Port network #3s EAL shifts from the PKTINT in PN #2 to the PKTINT in PN #1, but the far end remains at its EI board. Port network #2 moves from IPSI controlled to EI controlled by having the near end of the EAL shift from PN #2s PKTINT to PN #1s PKTINT and having the far end of the EAL shift from the IPSIs arch-angel to the EI boards arch-angel. In order to shift the port networks EAL from the IPSI to the EI board, a PN warm restart is required.

SPE

SPE

LAN

LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #1

IPSI EPN #2 EI
EAL #2

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #1 EI

EPN #3 EI

EAL #3

EAL #3

PNC Normal Operation Figure 7 - EI Fallback Recovery

PNC Faulted Fallback Operation

After the fault in the PKTINT is cleared, the SPE attempts to control that port network through its IPSI once again. The automated process of maintenance shifting a port networks EAL from the EI board back to the IPSI is called fall-up. Fall-up, like fall-back, requires a port network warm restart to shift the far end termination point of the EAL.

Section 4 - Reliability Multiple Cluster Environments


Thus far, a complete failure of the controlling cluster would render the communication system inoperable. As shown in the Reliability - Single Cluster Environments section, there are reparation methods for many types of failures that keep the system operational and port networks functioning. For instance, a failure of an active SPE can be addressed by having the standby SPE, within its cluster, takeover for it. This method works on the premise that failures of the active SPE and the standby SPE are independent. If there is a long mean time between failures of SPEs and one SPE is backing up the other one, then statistically there will always be a functional SPE. Therefore, given a functional SPE and a viable control path to the port networks, the system will be operational. However, true independence of SPE server failures within a cluster cannot usually be achieved due to the lack of geographical separation. For example, all the hardware and software reliability numbers calculated for the SPE servers are completely meaningless if the data room supporting them has an incident whereby everything in the room is destroyed.

12

There is a direct correlation between the availability of a port network and the number of different pathways that can be used by the SPE to control the PN. While there are recoveries which try different pathways for controlling purposes, there are situations when all of them fail. For example, if a port network is in a building and all communication pathways into that building are non-functional, then the systems controlling cluster is unable to support that isolated port network. In this case, the port network ceases to provide service to its endpoints and is useless until a control link is re-established. The ESS offering addresses theses types of cataclysmic failures. This section covers the previous offerings which provide the Communcation System protection from catastrophic main cluster failures and port network isolation using multiple clusters. All of these offerings, including Survivable Remote Processors, ATM WAN Spare Processors, and Manual Back-Up Servers, are being replaced with ESS.

4.1 How long does it take to be operational ?


The first question always asked in reference to survivability is How quickly does the PBX provide service to end users after a fault has occurred? Before answering this question in terms of an exact time, which is done in each of the survivable options sections below, the question What are the critical factors in determining a recovery timeline? is addressed. A recovery timeline for any problem can be broken down into three logical segments fault detection time, recovery option evaluation time, and recovery action time. The fault detection time is the period of time it takes to determine that something is not operating normally within the system. Some of the survivability options have automated fault detection. For example, the ATM WAN Spare Processors are continually handshaking keep-alive messages with the main server complex. If it does not receive a keep-alive response message in an appropriate amount of time, then the ATM WSP determines there is a fault. Other survivability options, such as Manual Back-up Servers, require manual detection that a fault has occurred. For automated detections, the time frame for fault detection is usually on the small order of seconds; however, no such time frame can be easily characterized for manual detections. After a fault is detected, a number of issues need to be resolved before taking any recovery actions. The first issue that needs to be addressed is determining if the fault detection is a false positive. For example, if two entities are continually handshaking over an IP network, they could detect something is awry if an abnormal round trip delay is detected. However, if this is a one time occurrence (which is completely normal for IP networks), then the fault detected is not really a fault and nothing should be done. The next issue to deal with is weighing recovery speed against the effect on end users. If a port network and its controlling server have a communication fault, then no service is supplied to end users supported by that PN. As previously discussed, if the fault is remedied within 60 seconds, then the server and the port network will continue operation without dropping any active calls. If a Survivable Remote Processor takes over a port network, it will cold restart that PN and, therefore, drop all calls supported by that PN. The question becomes, if a real fault is detected between the server and the port network, should the PN be immediately taken over by the Survivable Remote Processor and drop all calls in the process or should the main SPE be given the chance to re-establish connectivity with the PN and thereby possibly not dropping any calls if accomplished fast enough. The general rule of thumb is that if the recovery process will not disrupt service any further, then the recovery option evaluation time can be relatively short as it is with recovery methods shown in the Reliability - Single Cluster Environments section. If the recovery process does not preserve calls, as is the case with ESS and the offerings below, then the recovery option evaluation time should be long enough to give the non-call effecting recoveries every chance. The last issue that needs to be addressed is the trade off between recovery times versus being a fragmented system. Running in a fragmented mode has some drawbacks discussed at the end of the Reliability Multiple Cluster Environments section. If a fault prevents the server from communicating with a PN for an extended period of time, then that PN goes out of service. All the recovery options at this time involve resetting the port network and tearing down all existing calls, so fragmentation becomes the only issue. If it was known that the fault would be fixed within two minutes for example, it may be wise to prevent the

13

PN from being fragmented and deal with no service during that period of time because if the PN fragments it will be operating as its own island away from the rest of the PBX and may require another PN restart to agglomerate it back with the rest of the system. Unfortunately, there is no crystal ball that exists which can be queried to determine the outage time in advance. At this point, the recovery option evaluation time becomes a no service time since the PN is out of service longer than it needs to be to prevent fragmentation. The last part of the recovery timeline is the recovery action time portion. After the fault is detected and the decision has been made to take a particular recovery, that recovery takes place. The period of time that the recovery action takes is dependant on what is being done. For example, if port network goes into fall back because of an IPSI fault, only a single EPN warm restart is done which is very fast. However, when a Manual Back-Up Server takes over a system, it requires a complete system reboot (a cold restart) of the entire SPE which takes much longer.

4.2 What are Survivable Remote Processors (SRP) ?


A communication system configured to use a center-stage switch PNC has its port networks interface with the CSS through EI boards. Ideally, a point-to-point fiber connects the PNs EI board directly to a Switch Node Interface (SNI) board (TN571) in the CSS. However, direct fiber runs have distance limitations and geographically disperse port networks maybe desired. Therefore, a port network can utilize the PSTN to interface its EI board with its peer SNI. This type of EPN, called a DS1C remoted port network, has its EI board connected to a dedicated DS1 board which communicates with another DS1 board residing in the CSS. This second DS1 board is connected to the EI boards peer SNI. This DS1C remoted PN communicates with the rest of the system through this connection both for bearer traffic and for tunneled control. If the main processing complex ceases to operate or the connectivity between the sites has a fault, then the DS1C remoted port network has the potential to become a very expensive paper weight. An alternate controlling complex, called an SRP, can be used to control a DS1C remoted port network in these types of situations. The maintenance board in the remoted EPN is continuously monitoring the control connection link through the PSTN. If a fault is detected, it switches the fiber connecting the EI board to the DS1 to connect the EI board with the SRP. The port networks arch-angel is still resident on the remoted EPNs EI board, but the other terminating side of the EAL is now on the SRP as opposed to the IPSI at the main controllers site. On the left side of the figure below, the DS1C remoted EPN is being controlled from the main site during normal operation. The right side of the figure shows a fragmentation fault and the failover of the DS1C remoted EPN to a local SRP.
SPE LAN DS1C REMOTED EPN #2 EI IPSI
EAL #1

SPE LAN DS1C REMOTED EPN #2 EI IPSI PSTN


EAL #2 DS1 DS1 EAL #1

PSTN
EAL #2

EPN #1 EI

EPN #1 EI
DS1 DS1

SNI SNI CSS Normal Operation

SRP

SNI SNI CSS Fragmented Operation

SRP

Figure 8 SRP Control of DS1C Remoted EPN

14

As discussed previously, the SRP will take over the remoted DS1C port network only after all other recovery methods have been exhausted (server interchanges, IPSI interchanges, PNC interchanges, etc). There are two main reasons for ordering the recoveries this way call preservation and port network isolation. Before introducing the possibility of multiple controlling clusters, all control links to port networks came from the main active SPE. If another control pathway is needed, then the control link is transferred to a new path. If the logical EAL link is transferred onto another physical medium, the port network and the SPE become slightly out of synch. A port network warm reset is done to re-synch the EPN and the SPE. This resynchronization consists of the SPE verifying its view of resource allocation with the actual usage in the EPN. For example, if the SPE believes a call is using some resources on the TDM bus within the port network and an audit of those resources shows that they are still in use, then the SPE assumes that the call associated with those resources is still active and continues to allow it to proceed normally. However, if an audit shows that the resources are no longer in use, the SPE derives the conclusion that the call has been terminated (e.g. a user went on hook) and updates its resource usage maps. The What is a Server Interchange section described how a standby SPE can takeover for an active SPE without dropping any stable calls since call state was shadowed between the SPEs within the main cluster. Unfortunately, there is no call state shadowing between SPEs which are in different clusters (the main SPEs are in the main cluster and the SPE being used as an SRP is in another cluster). When an SRP assumes control of an EPN, it needs to synchronize with it in order to assume control. However, since the SRP has no call state information it has nothing with which to audit the EPN against. Therefore, the only way to synchronize the SRP SPE and the EPN is via a cold restart which tears down all existing calls. Once an SRP assumes control of a DS1C remoted port network, that EPN becomes an isolated entity. In other words, an SRP controlling an EPN becomes a stand-alone PBX separated from the rest of the main system. The effects of this isolation can be summarized in the following call flow. Assume that user A is supported by some PN at the main site and that user B is supported by the DS1C remoted port network. In a non-faulted environment, if user A dials user Bs number, the SPE will attempt to route the call. The SPE knows the status of user B, on hook in this example, since it controls the DS1C remoted EPN and therefore can successfully route the call to the end user through the port networks involved and the PNC. However, if the remoted EPN is currently isolated away under the control of an SRP, the call flow changes dramatically. When user A dials user Bs number the SPE determines that it cannot route the call. The SPE unfortunately does not know the status of user B since the support EPN is not within its control and therefore routes the call to an appropriate coverage path. In other words, when a user is not under control of the SPE, it assumes that the users phone is out-of-service even if it is being supported by another controlling entity such as the SRP. In addition, the isolated system consisting of the SRP and the EPN no longer has access to centralized resources at the main location. For example, if the interface to the voicemail system is located on a different PN at the main site, then the isolated fragment will not have access to it. This will cause intercept tone to be played to any end user which is supposed to be routed to voicemail coverage. Also, any calls that are generated or received by the SRP controlled EPN will not be part of the overall systems call accounting. This implies that a call made by a user off an EPN being controlled by an SRP will not appear in the main systems Call Detail Records (CDR). The SPE on the SRP is always operational and is continuously attempting to bring a control link up to the remoted EPN. Under normal operation, there is no pathway from the SRP to the EI, where the control link terminates, since the maintenance board has chosen to have the fiber connection from the EI go to the supporting DS1 board. Determining the recovery time for this port network therefore breaks down into the following time periods maintenance board detection of a control link failure from the main SPE, maintenance board no service time waiting for a possible quick network recovery, and SRP bringing up an EAL after the maintenance board switches over the fiber and cold restarting the port network. The maintenance board can detect an anomaly in the control link very quickly (within 2-3 seconds), but will wait in a no service time for approximately two minutes. Once the fiber switchover takes place, the SRP

15

will create an EAL within 180 seconds since it is a periodic attempt and the pathway now exists. The EPN cold restart will complete within approximately 1-2 minutes. For these reasons, the complete recovery time for a DS1C remoted EPN from a catastrophic fault, either due to a network fault or a main server crash, is approximately 7-10 minutes. As shown in Figure 8, the SRP controls an EPN via a point-to-point fiber which terminates at the EPNs EI board. Since there is no CSS PNC involved with this connection, the SRP can only control one port network. Therefore, it can be said, an SRP provides survivability for a port network, but not survivability for an entire system. In fact, since the SRP can only control one port network, it does not need translation information for the rest of the system. This has the advantage that SRPs can have distinct translations from the main system which gives the administrator the ability to have the EPN operate differently in survivability mode if desired (e.g. completely different route patterns or call vectors). However, since the SRP uses distinct translations, it has the disadvantage of needing double administration in some cases. In order to add a user to the main system, the translations on the main server need to be edited. Since there is no auto-synchronization of translations between the main server and the SRP, the same translation edits have to be done manually on the SRP. Once the connectivity between the main server and the DS1C remoted EPN has been restored, it is up to the system administrator to manually put the system back together. The main server has no knowledge that an SRP has taken over the remoted port network. Therefore, it is continuously attempting to bring that EPN back into service by instantiating an EAL to it for control. During the network fragmentation, it is obvious that every EAL creation attempt will fail (no pathway available). After the network fragmentation has been repaired, the EAL creation attempts will still fail since the fiber connectivity to the EI has been physically shifted. When the system administrator determines it is time to agglomerate the system, they will issue a command on the SRP forcing the maintenance board to swing the fiber back to the EI board from the DS1 board. The next periodic EAL creation attempt will now succeed since there is a viable pathway. It is important to keep in mind that when the main server resumes control over the DS1C remoted port network all current calls on that EPN will be dropped since the main server has no information about them. A cold restart of the EPN is required to bring it back online with the rest of the system.

4.3 What are ATM WAN Spare Pairs (ATM-WSP) ?


A communication system configured to use an ATM PNC has its port networks interface with an ATM backbone switch through ATM EI boards (TN2305/6). ATM WSP is an offering for DEFINITY G3r which provides alternate sources of control for EPNs that are unable to communicate with the main SPE. This communication failure can occur due to two reasons an ATM network fragmentation or a complete main server complex malfunction. In the first case, an ATM network fragmentation, the control links from the main server have no viable pathway to get to the port network. It is important to note however, that if the ATM PNC is duplicated with the critical reliability offer, then both ATM networks need to fragment in order to have PNs become isolated. For example, if the A-side ATM network is fragmented, the main SPE would transfer the controlling EAL to traverse the B-side ATM network. The second case, a complete main server complex failure, also prevents EPNs from receiving service since the controlling entity is unable to perform its task. If the SPEs are duplicated with the high or critical offer, then both SPEs need to have complete failures in order to leave the PN out-of-service. For example, if the A-side SPE has incurred a failure, then the B-side SPE would takeover the system through a server interchange. ATM WAN Spare Processors are added strategically throughout a system to provide alternate sources of control if needed. They consist of an SPE and an interface into the ATM network. All of the ATM WSPs in a system are priority ranked and are continuously heart-beating with each other. The figure below shows an ATM PNC PBX leveraging ATM WSP for higher system availability.

16

SPE PPN PKTINT EI


EAL #2 EAL #1

EPN #1 EI

EPN #3 EI
EAL #4

SPE ATM WSP #2 PKTINT EI

ATM PNC SPE


EAL #3

SPE ATM WSP #1 PKTINT EI

EI EPN #2

EI EPN #4

ATM WSP #3 PKTINT EI

Figure 9 G3r with ATM WAN Spare Processors

If an ATM WSP loses heart-beats with the main server for 15 minutes, it will then assume control of all EPNs it can reach if it is the highest ranked spare processor on its fragment. The figure below shows a catastrophic main server failure and control shift for all EPNs to ATM WSP #1. The other ATM WSPs in the system did not takeover any port networks since there is a higher ranking spare processor with which to communicate. In addition, since all the port networks failover to the same SPE, as it did before, the system is providing 100% equivalent service.
SPE PPN PKTINT EI
EAL #1

EPN #1 EI

EPN #3 EI
EAL #4

SPE ATM WSP #2 PKTINT EI

ATM PNC SPE ATM WSP #1 PKTINT EI


EAL #3

SPE ATM WSP #3 PKTINT EI

EI
EAL #2

EI EPN #4

EPN #2

Figure 10 G3r Failure with ATM WSP Takeover

Figure 11 shows an ATM network fragmentation and the failover of EPNs #3 and #4 to ATM WSP #2. ATM WSP #1 can not takeover those EPNs since no viable pathway exists to communicate with them and ATM WSP #3 does not takeover those EPNs since it can communicate with ATM WSP #2 which is ranked higher. While all port networks are receiving service from SPEs that have equivalent power as the main SPE, the system is running in a handicapped fashion. As discussed in the What are Survivable Remote Processors section above, some centralized resources may not be accessible by everyone and call flows between the fragmentations are disturbed. Each ATM WSP will provide equivalent service to EPNs that the main SPE did, but there is no guarantee that the ATM WSPs will control all system resources.

17

SPE PPN PKTINT EI


EAL #2 EAL #1

EPN #1 EI

EPN #3 EI
EAL #4

SPE ATM WSP #2 PKTINT EI


EAL #3

ATM PNC

SPE ATM WSP #1 PKTINT EI

SPE ATM WSP #3 PKTINT EI

EI EPN #2

EI EPN #4

Figure 11 ATM Network Fragmentation with ATM WSP Takeover

ATM WSPs are not designed to protect the system against all types of failures especially since the ATM WSPs does not attempt to assume control of any port networks until after a lack of communication with the main SPE for 15 minutes. If the communication between the main server and an EPN is severed for less than 60 seconds, the main SPE should continue to control the EPN once the connectivity is restored. If the ATM WSP took over the EPN too quickly, then an unnecessary fragmentation and EPN cold restart occurs. ATM WAN Spare Processors are designed to protect the system against catastrophic failures only. They are not designed to shield against temporary outages or software glitches. With that said, upon a catastrophic failure, an EPN is taken over by an ATM WSP within 15 minutes provided connectivity exists and then it goes through a cold restart which takes approximately 1-2 minutes. Therefore, an EPN which has its control transferred due to a fault from a main server to an ATM WSP will be operational within approximately 17 minutes. At all times the main server is attempting to control all EPNs regardless if the EPNs are being controlled by a spare processor. The control link that the main server is trying to instantiate will be blocked by the ATM EI board. The ATM EI board only allows one EAL link at a time to be terminated to it and when the ATM WSP took over the port network an EAL is created. Therefore, in order to agglomerate the system back to the main server controlling ATM WSPs need to be reset. If an ATM WSP is reset, then all EALs which it is using to control port networks are dropped. Once the EAL is dropped, the main servers cyclical attempt to terminate an EAL to that EPN will succeed and allow the main server to resume control.

4.4 What are Manual Backup Servers (MBS) ?


Communication Manager is the next generation of the DEFINITY PBX whereby the SPE is no longer resident within the system, but rather controls all EPNs either directly or indirectly over IP. Manual Backup Servers (MBS) is an offering that provides an alternate source of control to the PNs if a catastrophic event prevents the primary cluster from providing service by leveraging this IP control paradigm. MBS was created as an interim offer until ESS became generally available. The figure below shows how an MBS cluster can be positioned into a system to provide an alternate control source which is independent of the main cluster. This independence is crucial to increasing system availability because if the MBS is not independent then the event which rendered the main servers useless may also affect the MBS servers. While the example in the figure below shows a CCS PNC, the MBS offering works in conjunction with any of the PNC options available (CSS PNC, ATM PNC, Direct Connect, or IP PNC).

18

SPE

MBS

SPE

MBS

LAN

LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #1

IPSI
EAL #2

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #1 EI

EPN #2 EI

EPN #3 EI

EAL #3

EAL #3

PNC
Normal Operation

PNC
MBS Compensating for Main SPE Failure

Figure 12 CM with MBS Servers

Upon a failure scenario which prevents the main servers from controlling the port networks, the end users of the Communication System shall be out-of-service. There are many types of failures, described in the Reliability - Single Cluster Environments section, which can be resolved by the main servers without any intervention. However, if the failure is such that the main servers are not going to be operational for an extended period of time and all other recovery methods fail, then the MBS servers can be manually activated to take over the system. Under normal situations, the MBS servers are in a dormant state with translation sets synchronized to it on a demand basis. The activation of an MBS cluster is the process of taking the dormant SPE into an active state, or, in other words, doing a complete system reboot without any call state (no memory shadowing exists between the main cluster and the MBS cluster). This implies that when an MBS takes over control of the system, all EPNs go through a cold restart and any calls supported by these port networks are dropped. While the concept of the MBS servers was the basis for the ESS offering, there are many pitfalls with it most notably a non-deterministic recovery time. While the actual reboot time of the system (the method the MBS servers use to recover the system) can be assumed to be on the small order of minutes (see Table 4 in the What are SPE Restarts section), the other parts of the overall recovery time, detection time and no service time, are based on non-automated procedures and therefore have a tremendous amount of variance and no upper bound due to the human factor. There are some situations where the communication system is being tightly monitored at all times and, in this case, the complete system recovery time can hover around 10 minutes. However, if the system is not being closely monitored, it could takes tens of minutes to just detect that there is a problem and then a time consuming scramble to manually activate the MBS. Another pitfall is that the MBS servers are not continually proving their viability since MBS servers remain in a completely dormant state during normal operation. Ideally, the MBS would always be running diagnostics on its hardware and resident software and checking its connectivity to all of the port networks IPSIs. Without this information, there is no way of knowing ahead of time if the MBS servers will be able to provide service to the port networks if needed and no way of alarming the issues so they can be addressed before the MBS is ever activated. Furthermore, the MBS offering does not scale well and, in fact, there can only be one MBS for a main server. This restriction limits the complexity of disaster recovery plans and the number of different catastrophic scenarios which can be protected against.

19

MBS servers also have operational pitfalls. The key to the MBS servers successfully taking over the system is that the main cluster is completely eliminated from the equation. The MBS servers are exact duplicates, from a software point of view, of the main servers and are unaware of the main servers presence in the system. Therefore it cannot operate in conjunction with the main servers for any length of time. For example, if a situation arose whereby the MBS servers were activated while the main servers were still operational, then the port networks would consistently thrash back and forth between getting control from the main cluster and the MBS cluster (this would render the port network inoperable). Furthermore, suppose the MBS is activated at an appropriate time while the main cluster is inactive, but at a later time the main cluster is fixed and comes back on-line. This would cause the same thrashing problem. This leads to the largest downfall of the MBS offering it does not, although it appears to, provide protection against network fragmentation. If the network fragments as shown in the diagram below, the port networks unable to communicate with the main server will be out-of-service (fallback is unavailable since there is no CSS or ATM PNC in this example). If the MBS is activated, it will assume control of the port networks it can reach. During the time that the network is fragmented, both the main and MBS servers are attempting to control all port networks in the system but are blocked from doing so due to a lack of connectivity. However, this implies that the system is only stable while the fragmentation continues to exist because when the network is healed, both clusters will be able to connect to all port networks and therefore fight over control of all them. It cannot be stated strongly enough that MBS servers should never be used to protect against network fragmentation faults. The ESS offering provides all the advantages of MBS without any of the drawbacks.
Stable only while network is fragmented. The fragmentation is preventing contention.

SPE

MBS

LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

EPN #1

EPN #2

EPN #3

EPN #4

Figure 13 MBS Attempting to Assist in Network Fragmentation Fault

Section 5 - ESS Overview


5.1 What are Enterprise Survivable Servers (ESS) ?
ESS is an offering which simultaneously protects Avaya Communication Systems against both catastrophic main server failures and network fragmentations. This is achieved by having ESS servers placed strategically within an enterprise as an alternate source of control for port network gateways when control from the primary source is disrupted for an extended period of time. Unlike the MBS offering, the control transfer between the primary source and the alternate source is automatic in the ESS offering and multiple ESS servers may be operational concurrently.

20

ESS servers consist of media servers executing the SPE software. While the SPE software running on ESS servers is identical to the SPE software running on the main server, it is licensed differently which drastically affects its operational behavior. Some of these behavioral variations include different alarming rules, capability to register with a main server, and ability to receive translation synchronization updates. Currently, ESS licensed software can only execute on a subset of the media servers available S87XX and S8500 Media Servers.

5.2 What is ESS designed to protect against ?


Upon a main cluster failure, port networks lose their control source. If this failure is catastrophic in nature, whereby the main servers will not be operational again for an extended period of time, port networks can search out and find ESS servers that will provide them service. Therefore, ESS protects port networks against extended failures of the main controlling cluster. For most scenarios, the desired failover behavior caused by a catastrophic main server failure is to have all surviving port networks find service from the same ESS server in order to keep the system in a non-fragmented state. The figure below shows a catastrophic main cluster failure with all of the port networks failing over to a single ESS cluster.

Main SPE IP Network

ESS #1

ESS #2

EPN #1

EPN #2

EPN #3

EPN #4

Figure 14 ESS Protecting Against Catastrophic Server Failure

Upon a connectivity failure to the main cluster, port networks lose their control source. If this fragmentation continues to exist over an extended period of time, port networks can search out and find ESS servers on their side of the fragmentation that will provide them service. Therefore, ESS protects port networks against significantly long fragmentations away from the main controlling cluster. For most scenarios, the desired failover behavior caused by a network fragmentation is to have all port networks which can still communicate with each other to agglomerate together forming a single stand-alone system. The figure below shows a network fragmentation with all of the port networks on the right side of the fragment grouping together to form a single autonomous system.

Main SPE IP Network

ESS #1

ESS #2

EPN #1

EPN #2

EPN #3

EPN #4

Figure 15 ESS Protecting Against Network Fragmentation

It cannot be emphasized enough that ESS increases the availability of a communication system when faced with catastrophic server failures or extended network outages. ESS builds upon all of the existing fault recovery mechanisms which are already integrated into the system. For example, when a network outage is

21

detected, the PBX attempts to re-establish PN control by interchanging IPSIs (if the port network has duplicated IPSIs), fall-back control through another PN (if the port network is leveraging CSS or ATM PNC), and reconnecting to the PN and warm restarting it (if the outage is persistent for less than 60 seconds) before ESS servers ever get involved. ESS is the last line of defense to provide service to port networks while in these severely handicapped situations.

5.3 What is ESS NOT designed to protect against ?


The ESS offering is designed to protect systems against major faults which exist over a prolonged period of time. ESS builds upon all of the existing fault recovery mechanisms which are already integrated into the communication system. This section will present a number of common faults which can occur and explain how they are resolved without ESS. Attempting to have ESS address these types of faults causes overall system reliability to actually be decreased since race conditions between ESS and existing recoveries are introduced. Fault Example: Main Active Server Hardware Failure Resolution: Server Interchange Description: When the active server encounters a hardware fault, either the local Arbiter will be informed of the degraded hardware state of health (SOH) or the standby servers Arbiter will lose contact with its peer. Regardless of the detection mechanism, the decision will be made for the standby server to take over control of the system from the currently active server. In addition, resolving this problem by interchanging servers will preserve all calls within the system. ESS will not get involved with the recovery unless the main server is a simplex server or the standby server is in an inoperable state. Fault Example: Extended Control Network A Outage Resolution: IPSI Interchange or Control Fall-back (depending on system configuration) Description: When the control network A outage is detected, the system will attempt to transition its PN control link from the active IPSI, which is connected through control network A, to the standby IPSI, which is connected though control network B provided the port network has duplicated IPSIs. If the IPSI interchange fails or the standby IPSI does not exist, the system attempts to transition its PN control link from an IPSI local in the PN to an IPSI which it can communicate with in another PN provided the PNC supports control link tunneling (CSS or ATM PNC). If the problem is resolved by either of these methods, all calls within affected port networks will be preserved. Only if both IPSIs experience extended network faults simultaneously (control network A and B) and fall-back control is not supported through the PNC (IP), would the system need ESS to take over control for that port network. Fault Example: Short Control Network Outage Resolution: EPN Warm Restart Description: When a control network outage occurs and no other recovery mechanism can work around the problem (discussed in the fault example above), the controlling server continuously attempts to establish a new connection to the port networks. If the network outage is less than 60 seconds, the SPE will reconnect to the PNs IPSI and resynchronize resource allocation with the port network. This resynchronization process will preserve all of the calls (both call state and bearer connection) currently being supported by the PN. If the outage is over an extended period of time, then the ESS would takeover control of the port network. It is important to ensure that the call preserving recovery has a chance to complete before allowing ESS servers to assume control. Fault Example: Software Fault Resolution: System Restart or Server Interchange Description: If a software fault occurs on the main active server, the CM application will attempt to initiate a system restart it deems necessary to resolve the software fault and, at the same time, inform the local Arbiter. If the system restart fails to resolve the issue, the Arbiter may choose to interchange servers or to initiate another system restart at a higher level. Depending on the restart level chosen to resolve the issue, calls may (warm restart) or may not (cold restart) be preserved. ESS would enter the recovery solution only if the software faults are severe enough to completely freeze both main servers preventing them from communicating with IPSIs and issuing self reboot actions.

22

Fault Example: Complete IPSI Failure Resolution: IPSI Interchange Description: An IPSI is made up of a number of components including a PKTINT, an arch-angel, and a tone clock. If a complete IPSI failure occurs causing everything on it to cease operation, then the system will attempt to transition the control link to the standby IPSI. However, since the tone clock for the PN also has failed, the IPSI interchange will not preserve calls. If the port network did not have duplicated IPSIs then it would become inoperable. The IPSI is in charge of seeking out alternate ESS clusters when needed and if the IPSI is completely down, then ESS clusters will not be requested to take over. Also, even if an ESS took over the ISPI, the tone clock is a critical component of a PN and without it the port networks cannot operate. Fault Example: Complete Port Network Power Outage Resolution: None Description: Without power to the PN, it cannot provide any services to its endpoints. Until power is restored, the resources provided to the system through the powerless port network are out of service. In situations like this, ESS does not help. In addition, since no calls could exist within a powered off cabinet, no calls are preserved when the power is restored to the cabinet. One of the greatest assets of an Avaya Communication System is the ability it has to resolve faults that occur. When the DEFINITY G3r system evolves into the Avaya Communication Manager additional recovery mechanisms were invented, but they work as add-ons to, not replacements for, the already existing ones. ESS provides a new layer of system availability without compromising the tried-and-true mechanisms that are already built into the system. If ESS is used to protect against the faults mentioned above, then unnecessary resets will occur along with the possibility of unwanted system fragmentation.

Section 6 - How does ESS Work ?


Except for some basic initial configuration of the ESS servers, all system administration is done on the main cluster. Once an ESS server informs the main server that it is present (via registration), the main server synchronizes the entire system translation set to it (via file synch). This synchronization process continues periodically through maintenance and/or on a demand basis to ensure that the ESS clusters have updated translations. With these full system translations, the ESS servers have the information required to connect to all IPSIs such as IP addresses, ports, QOS parameters, and encryption algorithms. Once the ESS servers connect and authenticate with the IPSIs, they advertise administered factors which the IPSIs use to calculate a priority score for the ESS cluster. The IPSI is always maintaining an ordered failover preference list based on these priority scores. This priority list is dynamic based on network and server conditions and only contains currently connected viable servers. Upon an IPSI losing connectivity to its controlling cluster, it starts a no service timer. If a re-connection does not occur before the timer expires, the IPSI will request service from the highest ranking viable ESS clusters in its preference list. This algorithm prevents knee jerk and non-fault control shifts, but the IPSI service requesting process can be overridden by manual or scheduled agglomeration procedures. The rest of this section examines in detail each of the operational steps of the ESS offering.

6.1 What is ESS Registration ?


When an ESS is installed into a system, it is configured with an IP address (local IP interface), a remote registration IP address (CLAN gatekeeper), and a unique server ID (SVID). In addition, a license file will be loaded which provides the ESS cluster with a unique module ID (otherwise known as a cluster ID or CLID) and a system ID (SID). Concurrently, the ESS is added to the system translations on the main server by creating a record with all of this information (IP address, CLID, SVID, and SID) along with associated factors which will be described later.

23

Upon an initial start-up or reset, an ESS will send a registration packet (an RRQ) containing its configured data to a remote registration IP address (port 1719). The initial time an ESS attempts to register with the main server, it only tries at the originally configured registration IP address. However, after the initial translation synchronization (discussed in the next section), the ESS first tries to register at the configured registration IP address, but upon a failure, tries all possible registration points, regardless of network region, administered within the system. Upon the main server receiving the registration request, it attempts to validate the ESS server by ensuring that the IP address, CLID, and SVID match the administered data for that ESS and that the SID provided by the ESS matches its own system ID. If it authenticates, the main server will respond to the ESS server with a confirmation (an RCF) and continue handshaking with it. The handshaking takes the form of heartbeats (KA-RRQs, a slight modified version of RRQs) and heartbeat responses (KA-RCFs, a slightly modified version of RCFs). Keep alive handshaking is performed by every ESS server on a periodic basis of once per minute. Since the keep-alive packets are very small, occur only once per minute, and there are relatively few ESS clusters within a system (maximum of 63), the bandwidth required for this is negligible (approximately 2.5 Kbps per ESS cluster). Figure #16 below shows the registration link between the ESS clusters and the main cluster. If the ESS cluster consists of a duplicated server, an S87XX media server, then only the active server within the cluster will register.
IPSI Main SPE EPN #1 IP Network CLAN IP Network

IPSI EPN #2

CLAN

ESS Registration Pathway

Figure 16 ESS Registration Pathway

Currently, as the figure shows, CLANs are utilized as the registration entry point to the main servers. It is important to note that CLANs in a port network controlled by an ESS cannot serve as a registration access point for another ESS cluster within the system (ESS servers do not register with each other).

6.2 How are ESS Translations Updated ?


Other than for alarming reasons and providing a medium for issuing some remote commands (discussed later), the main purpose of ESS registration is to allow the main server to know the translation status of each ESS. Administration changes made on the main server are not updated in real time to the ESS servers. Rather, the translation changes are synched to the ESS server on either an automated periodic basis once per day or on demand with a manually entered command. Since the ESS offering is designed to only protect against catastrophic failures, taking over port networks with slightly out-of-date translations is completely valid. For example, if an end user has added a new feature button to his/her phone and a failure occurs before periodic maintenance has updated the ESS server, then the phone operates as if no administration actions took place (e.g. no new feature button) when it is taken over by the ESS server. Whenever synchronization between the main server and the ESS server is requested, the main server checks to see if it is necessary based on the current translation state of the ESS (which was provided in its registration heartbeats). If the ESS server has translations that are identical to the main servers, then the actual synchronization process is skipped. However, if the ESS translations are not up to date with the main server, the main server synchronizes the ESS server by supplying the complete system translation file or by providing the changes since the last synchronization (incremental file synch). Incremental file synchs require the ESS cluster to be at most one translation synchronization behind since the changes need to be applied to identical images. This implies that full translation files need to be synched whenever initial startup occurs or if the ESS server has been un-registered for an extended period of time. Obviously the preferred method is to only transmit the changes since the amount of data needed to be sent is relatively

24

small (very dependant on the number of changes, but usually in the 100s of kilobyte range) as opposed to transmitting the entire system translation file which is relatively large (very dependant on the size of the system, but usually ranges from 2 to 30 megabytes and can be compressed to approximately a 5:1 ratio). The bandwidth required for this process is dictated by the rule that the synch process must be completed within 5 minutes. Therefore, the bandwidth required in the worst cases (largest systems synching full translation sets which are compressed) is approximately 200 Kbps over that 5 minute period. ESS servers are operational at all times proving their viability by running the CM software, monitoring system state-of-health, and handshaking with the IPSIs. After translation synchronization completes, the ESS server must stop running on the old translation set and begin executing on the new set. In order to achieve this, the ESS is required to perform a reboot (as described previously in the What are SPE Restarts section). Therefore, the entire synch process, under normal conditions, can be summarized by the following steps: 1. 2. 3. 4. The main server creates a compressed file consisting of the admin changes since the last synch. This file is transferred from the main server to the ESS server. The ESS server applies the changes given to it onto its existing translation file. The ESS initiates a self reboot in order to begin running on the new translations.

However, based on the current state of the system, some of these steps may be altered. The first step may require an entire system file transfer as opposed to just the changes which causes the third step to swap out the files rather than applying changes. Rebooting an ESS server which is not currently controlling any port network gateways does not affect end-users what-so-ever except for not being available as an alternative source of control if needed during the reboot cycle (approximately 2 3 minutes). On the other hand, if the ESS is actively controlling port networks, then a reboot of an ESS server would cause an outage to all of the supported end-users. Therefore, the fourth step of the process will be delayed until the ESS server is no longer controlling any end-users or gateways. That being said, administration changes made on the main server will not be made available to an ESS which is actively controlling resources even if translation synchronization occurs. Another consequence of the fourth step is that an excessive number of demand synchronization requests will cause an ESS to go through many resets. While the resets themselves do not affect system operation or performance, the more time ESS servers spend resetting, the less time they are available as an alternate control source. Therefore, demand synchronizations, via the save translations ess command, should only be performed after critical administration has been completed which is absolutely needed in catastrophic failure modes and cannot wait for the automated synch process. The license file installed on the ESS server gives the CM software running on it the ability to receive file synchronizations from the main server. However, the ability to make administration changes locally on the ESS is not prohibited although there are three important caveats to take into account. The first caveat is that any administration changes made on the ESS are lost whenever translation synchronization takes place followed by the subsequent reboot. Therefore, administration changes can be made to an ESS server while it is not in control of any resources and have them be effective if the ESS needs to take over any port networks, but all the changes are lost when periodic or demand translation synchronization takes places. The second caveat is that translation changes made on an ESS cannot be re-synchronized back to the main server. Hence, administration changes made locally to an ESS server which is controlling end-points will not be reflected in the overall system translations and will only be effective until the main server resumes control of the end-points again. The last caveat is that translation changes made locally to an ESS cannot be saved to disk. Since an ESS server is just as powerful as a main server and has identical RTUs, a customer could conceivably purchase an ESS server and illegally make it into its own stand-alone system at a substantial discount. To prevent this from happening two major blockades are introduced; ESS servers enter no license mode whenever it controls resources which gives a limited operational lifespan of the server and elimination of the ability to save translations to disk on ESS servers which puts all administration changes done at risk if a system reboot occurs (which is what happens if the no license mode timer expires). The figure below shows the network interconnectivity required to have ESS servers get synchronizations from the main server. It is important to note that the secure transfer of the translation file or translation

25

changes file takes a drastically different path than the registration link. File synchs are done directly between media servers (utilizing port 21874) without going through CLANs. Also, the diagram shows two separate, distinct networks, the control LAN and the corporate LAN, but they can actually be implemented as one and the same.
Translation Synchronization Pathway Corporate LAN

IPSI Main SPE EPN #1 IP Network

CLAN IP Network

Control LAN

IPSI EPN #2

CLAN

ESS Registration Pathway

Figure 17 ESS Registration Pathway

Moreover, the diagram is leveraging a simplex S8500 Media Server as the main cluster and a simplex S8500 Media Server as the ESS cluster. If either of the clusters consisted of duplex S87XX Media Servers, then the translation synchronization will only happen between the active media servers. In a duplex media server cluster, it is the responsibility of the active SPE to update the standby SPE translations. Unfortunately, there is a negative implication of synchronizing translations on a periodic basis as opposed to being done in real time some phone state information will be lost. There are a class of features that end-users can execute which change the state of their phone and the associated phones translation information. Two primary examples of these types of features are send-all-calls and EC500 activation. If an end-user changes the state of these features (either enabling or disabling) and a failure occurs causing an ESS to takeover before the translation changes have been sent to the ESS server, the feature states will revert to their previous state. For example, imagine a scenario whereby a user hits their send-all-calls feature button in the morning and then goes on vacation. If a catastrophic failure occurs that afternoon before a translation synch occurs and the end-user is transferred to an ESS, then the send-all-calls feature will be deactivated (its previous state). Any calls that come to that user while being controlled by the ESS are not immediately sent to coverage.

6.3 How do Media Servers and IPSIs Communicate ?


Once an ESS server receives a translation set from the main server, it has all the information needed in order to connect to all the IPSIs within a system. In other words, once the ESS servers have each IPSIs IP address & listen port and each IPSIs encryption algorithm settings & QOS parameters, it will initiate a TCP socket connection to all IPSIs (utilizing port 5010). If the connection is unsuccessful, the servers will periodically attempt a new connection every 30 seconds, otherwise the authentication process commences. The authentication process consists of the validation of key messages based on an encryption method commonly known between the servers and the IPSIs. Once the server and IPSI can communicate securely over this established TCP link, the ESS server uniquely identifies itself (via cluster ID and server ID). Following that, the IPSI will request priority factors (discussed in the next section) from the connecting servers in order to calculate a priority score (discussed in the next section) for that server which will be used to create the IPSIs self-generated priority list (discussed below). At a high level, with the details to follow, the IPSI receives pre-programmed factors from each ESS server that it will use to calculate a score for that ESS cluster. The higher the score an IPSI assigns for an ESS cluster, the higher the ESS cluster

26

resides in the failover preference list. The following flow chart shows the initial message sequence that the server and the IPSI go through in order to establish a session. Media Server Open TCP Socket TCP Socket Up Confirmation Authentication & Encryption Setup Authentication & Encryption Completion Notification update message gets sent to all media servers currently connected to the IPSI IPSI

ESS Indication Message Factors Request Message Factors Message Notification Update Message

Figure 18 Message Flow upon Server-IPSI Connection

After the socket has been brought up between the IPSI and a non-controlling ESS server, there are three tasks that occur. First, the ESS servers need to be consistently monitoring the health of the IP connection and this is achieved by doing an application level keep alive once per second. These heartbeat messages can easily be seen by monitoring traffic on the control network and have an average size of 10 bytes (8 bytes for server to IPSI heartbeats and 12 bytes for IPSI to server response heartbeats). This implies, since these messages are sent as minimally sized TCP packets, that there is a static bandwidth of approximately 1 Kbits per second required between the servers and IPSIs for these operations. Secondly, the IPSI is responsible for keeping every connected server up to date with respect to its own selfgenerated priority preference list. This is done by sending up priority list notification messages whenever a change occurs (new server connecting, existing server disconnecting, or change in controlling cluster). An ESS server gets the current priority list upon its initial connection to the IPSI and then gets any subsequent updates. For example, if an ESS server disconnects from an IPSI for any reason, the IPSI sends updated priority lists, with that ESS server removed, to all servers which are still connected. If the disconnected ESS reconnects to the IPSI the same procedure is followed whereby the reconnecting ESS is inserted appropriately back into the priority list and all connected servers receive this update. The notification message size varies based on the number of ESS clusters and the ESS cluster types (simplex or duplex), from a minimum of 8 bytes to a maximum of 68 bytes. Since priority preference lists do not change on a periodic basis, only on network failures, ESS server failures, or certain ESS maintenance operations and the notifications are relatively small, virtually no static bandwidth is required for this operation. The third responsibility is to ensure there is always a socket up between servers and IPSIs. This implies that upon a connectivity failure, the server attempts to restore a TCP connection. If there is a socket failure the server will periodically try to bring up a new socket every 30 seconds. As the figure #18 above shows, there are initial handshakes that occur when a socket is established. This exchange comprises of

27

approximately 512 bytes. However, since socket connections go down during failure scenarios and this can not be occurring with any type of frequency, there is no static bandwidth required for this operation. To summarize, the bandwidth required between ESS servers and IPSIs during normal operation (main server in control of the entire system) is only 1 Kbits per second. During failure scenarios there is a slight spike in traffic between servers and IPSIs. It is important to realize however, that the 1 Kbits / second bandwidth requirement is for non-controlling servers only. The bandwidth required between a controlling server and an IPSI is approximately 64 Kbits / second per 1000 busy hour call completions supported by that IPSI. When designing a system, enough bandwidth must be available between ESS servers and port networks in case the ESS is required to take over port network control. In other words, even though only 1 Kbits per second are needed during normal operation between ESS servers and IPSI, the network bandwidth requirements must be established assuming the ESS will be controlling port networks.

6.4 What is a Priority Score ?


Any disaster recovery plan worth its weight provides survivability for critical components in a deterministic manner. ESS addresses this by having critical components (the port network gateways) generate, via its resident IPSI, a ranked ordering of alternate control sources (ESS clusters). The IPSI ranks connecting ESS servers using a priority score calculated from factors provided by the ESS clusters. The most basic way to have a system react to catastrophic faults is to have all IPSIs share the same failover preference list. To achieve this, each ESS cluster can be assigned a base score value, between 1 and 100, which it will advertise to the IPSIs upon a connection. Suppose there are six port network gateways within a system distributed in three separate locations as shown in the figure below.
Main SPE
IP LAN Main Location

ESS #40 ESS #20


IP LAN Location #1

ESS #30
IP LAN Location #2 IP LAN Location #3

EPN #1

EPN #2

EPN #3

EPN #4

EPN #5

EPN #6

Figure 19 Basic System Layout If the disaster recovery plan states that all the port networks should attempt to get service from ESS #20, then from ESS #30, and then finally from ESS #40 in failure situations, the ESS clusters can be given the following base scores: ESS CLID Base Score 20 75 30 50 40 25 Table 5 ESS Cluster Admin

28

The IPSIs will rank the ESS clusters based on their base scores (higher the better) and in this example each IPSI will have identical priority lists as follows: Port Network #: 1st Alternative: 2nd Alternative: 3rd Alternative: PN #1 PN #2 PN #3 PN #4 #20 #20 #20 #20 #30 #30 #30 #30 #40 #40 #40 #40 Table 6 IPSI Priority Lists PN #5 #20 #30 #40 PN #6 #20 #30 #40

However, the network design may not allow this to be feasible. For example, suppose the WAN links interconnecting the locations is limited and attempts should be made to avoid sending control traffic over them unless absolutely necessary. This requirement can be realized if the IPSIs generated the following lists: Port Network #: 1st Alternative: 2nd Alternative: 3rd Alternative: PN #1 PN #2 PN #3 PN #4 #20 #20 #20 #30 #30 #30 #30 #20 #40 #40 #40 #40 Table 7 IPSI Priority Lists PN #5 #30 #20 #40 PN #6 #40 #20 #30

To achieve this ESS, introduces two new concepts communities and local preference. Every ESS server in the system is assigned to a community (with a default of 1) and every IPSI is assigned to a community (with a default of 1). By definition, if an ESS server is assigned to the same community as an IPSI, then the ESS server and IPSI are in the same community or local to each other. The local preference attribute can be assigned to an ESS server which can boost the priority score calculated for it by an IPSI if it is local to it. The priority score (PS) calculation is written generically as:

PS = wi vi
i =1

Calculation 1 Generic Priority Score Calculation with Index (i) 1 2 Factor Type base score local preference boost Factor Value (v) Administered base score (1-100) 0 if not within same community or boost is not enabled 1 if in the same community and boost is enabled Table 8 Factor Definitions Factor Weight (w) 1 250

The 250 weight for the local preference boost guarantees that an ESS with this enabled will rank higher with respect to the IPSI if it is local than the ESS with the highest base score. If each ESS cluster in this example has this local preference boost enabled, shown below in table 9 below, then the IPSIs will all generate preference lists as shown above in table 8. ESS CLID 20 30 40 Base Score Locally Preferred 75 Yes 50 Yes 25 Yes Table 9 ESS Cluster Admin Community 1 2 3

With this administration, it is important to realize that upon a catastrophic main server fault three autonomous systems will be created. All of the IPSIs in community #1 will failover to ESS #20 which is in community #1, all of the IPSIs in community #2 will failover to ESS #30 which is in community #2, and

29

the IPSI in community #3 will failover to ESS #40. If an additional disaster recovery plan requirement is added stating that the system should attempt to remain as one system as long as possible, then ESS #10 can be added to the system layout.
Main SPE
IP LAN Main Location

ESS #10 ESS #40

ESS #20
IP LAN Location #1

ESS #30

IP LAN Location #2

IP LAN Location #3

EPN #1

EPN #2

EPN #3

EPN #4

EPN #5

EPN #6

Figure 20 System Layout with System Preferred ESS However, even if ESS #10 is given a base score of 100, the IPSIs priority lists will unfortunately be as follows: Port Network #: PN #1 PN #2 PN #3 PN #4 PN #5 #20 #20 #20 #30 #30 1st Alternative: 2nd Alternative: #10 #10 #10 #10 #10 3rd Alternative: #30 #30 #30 #20 #20 4th Alternative: #40 #40 #40 #40 #40 Table 10 Undesired IPSI Priority Lists Even though the desired IPSI priority lists should be: Port Network #: 1st Alternative: 2nd Alternative: 3rd Alternative: 4th Alternative: PN #1 PN #2 PN #3 PN #4 #10 #10 #10 #10 #20 #20 #20 #30 #30 #30 #30 #20 #40 #40 #40 #40 Table 11 Desired IPSI Priority Lists PN #5 #10 #30 #20 #40 PN #6 #10 #40 #20 #30 PN #6 #40 #10 #20 #30

The desired IPSI priority lists causes the IPSIs to attempt to transfer over to ESS #10 in the case of a main cluster failure and then failover to local service if ESS #10 is unable to provide adequate control. To achieve this, ESS#10 must be given the system preferred attribute. This attribute will boost the priority score calculated of a system preferred ESS above any local locally preferred ESS clusters. This factor fits into the factor definitions as: Index (i) 1 2 Factor Value (v) Administered base score (1-100) 0 if not within same community or boost is not enabled 1 if in the same community and boost is enabled system preference boost 500 0 if boost is not enabled 1 if boost is enabled Table 12 Factor Definitions Factor Type base score local preference boost Factor Weight (w) 1 250

30

The 500 weight obviously allows system preferred ESS clusters to rank higher than all other clusters (even one that has a 100 base score and is locally preferred, 350 < 500). It is important to understand that the only need for assigning the system preferred attribute is to compensate for local preference attribute usage. In other words, if no ESS servers are given the local preferred setting, then there is no need to use the system preferred attribute. For simplicity, the IPSI was designed to treat all servers, main and ESS, consistently when calculating preference scores. The leads to the last factor that needs to be discussed main server attribute. Since priority preference lists include the main cluster and it should always be ranked the highest (under normal conditions, the IPSIs should be receiving their control from the main cluster). Index (i) 1 2 Factor Value (v) Administered base score (1-100) 0 if not within same community or boost is not enabled 1 if in the same community and boost is enabled system preference boost 500 0 if boost is not enabled 1 if boost is enabled main cluster boost 1000 0 if an ESS cluster 1 if the main cluster Table 13 Factor Definitions Factor Type base score local preference boost Factor Weight (w) 1 250

3 4

Therefore, the disaster requirements are achieved by the following administration (the main cluster is always assigned a CLID of 1): ESS CLID 1 10 20 30 40 Base Score x 100 75 50 25 Main System Preferred Locally Preferred Yes x x No Yes x No No Yes No No Yes No No Yes Table 14 ESS Cluster Admin Community 1 1 1 2 3

Which results in the following lists generated independently by each IPSI: Port Network #: 1st Alternative: 2nd Alternative: 3rd Alternative: 4th Alternative: 5th Alternative: PN #1 PN #2 PN #3 PN #4 #1 #1 #1 #1 #10 #10 #10 #10 #20 #20 #20 #30 #30 #30 #30 #20 #40 #40 #40 #40 Table 15 IPSI Priority Lists PN #5 #1 #10 #30 #20 #40 PN #6 #1 #10 #40 #20 #30

To throw a final wrench into the picture, suppose the WAN link between locations 2 and 3 in Figure #20 is extremely limited and there is only enough bandwidth for one control link to traverse it. This implies ESS #40 cannot be an alternate control source for any port network gateway except for port network #6. This requirement manifests itself in the following preference lists: Port Network #: 1st Alternative: 2nd Alternative: 3rd Alternative: 4th Alternative: 5th Alternative: PN #1 PN #2 PN #3 PN #4 #1 #1 #1 #1 #10 #10 #10 #10 #20 #20 #20 #30 #30 #30 #30 #20 x x x x Table 16 IPSI Priority Lists PN #5 #1 #10 #30 #20 x PN #6 #1 #10 #40 #20 #30

31

ESS #40 must be administered to only offer its services to IPSIs within the same community or, stated differently, provide service locally only, which is realized by giving ESS #40 the local only attribute. This attribute is unlike the other factors whereby there is not a boost in score, but rather if this attribute is enabled for an ESS, that ESS server does not even attempt to contact IPSIs which are not within its own community and thereby never get onto non-local IPSI priority lists. The administration to achieve the priority lists in table X above is: ESS CLID 1 10 20 30 40 Base Score x 100 75 50 25 Main Yes 10 20 30 40 Sys Preferred Locally Preferred x x Yes x No Yes No Yes No Yes Table 17 ESS Cluster Admin Community 1 1 1 2 3 Local Only x No No No Yes

It is the combination of these basic factors which give the ESS offering the ability to achieve very complicated deterministic failovers for IPSIs, but at the same time is easily administered. While every IPSI independently generates its own priority list some commonalities will exist among all of them. The IPSIs priority list can be broken down generically as follows: Main Server Common Among All IPSIs Possibly Different Among All IPSIs Figure 21 Generic IPSI Priority Lists Where the break line between common among all IPSIs and possibly different among all IPSIs is based on the number of system preferred ESS clusters administered and the number of ESS which have the local preference attribute enabled. If the local preference attribute is not utilized, then the last section of the priority lists, possibly different among all IPSIs, would always be empty.

6.5 How do IPSIs Manage Priority Lists ?


The previous section discussed how IPSIs generate their priority failover lists in non-faulted environments (every ESS can connect to every IPSI). Whenever a new cluster connects to an IPSI, a priority score is calculated for that cluster based on the advertised factors. The IPSI then dynamically inserts the cluster into its priority list based on the priority score and informs all currently connected ESS servers about the updates. Whenever an IPSI cannot communicate with a media server due to either a server fault or network fragmentation, it is dynamically removed from the priority list and all ESS servers still connected will receive updates.

32

ESS #10 Main (CLID #1)


IP LAN

ESS #20 ESS #30

EPN #1

Figure 22 IPSI Interfacing with Multiple ESS Clusters Using the diagram above, under non-faulted environments, the IPSIs priority failover list will be: PN #1 1st Alternative: #1 2nd Alternative: #10 3rd Alternative: #20 4th Alternative: #30 Table 18 IPSI Priority Lists If a communication failure is detected (discussed in the next section) between the IPSI and, for this example, ESS #20, then the IPSI would remove ESS #20 from its priority list. PN #1 1st Alternative: #1 2nd Alternative: #10 3rd Alternative: #30 4th Alternative: none Table 19 IPSI Priority Lists Once the communication between the IPSI and the ESS is restored, the IPSI will insert it back into the list appropriately. PN #1 1st Alternative: #1 2nd Alternative: #10 3rd Alternative: #20 4th Alternative: #30 Table 20 IPSI Priority Lists It should be obvious that these concepts are easily extended to the addition/removal and enabling/disabling of ESS servers. For example, if an ESS server is added to the system (ESS #40 with a base score of 60), it will be inserted into the IPSIs list appropriately. PN #1 1st Alternative: #1 2nd Alternative: #10 3rd Alternative: #40 4th Alternative: #20 5th Alternative: #30 Table 21 IPSI Priority Lists

33

Since the IPSIs do not keep any type of historical records of previously connected ESS clusters, the system survivability plans can be altered at any time (e.g. move ESS clusters around strategically) or implemented over time (e.g. no flash cuts are required). While this list manipulation is very simple, the complexity occurs when port networks have duplicated IPSIs. It is absolutely critical that a pair of IPSIs in the same PN share identical priority lists. This is achieved by communication between the IPSI pairs over the carriers backplane. In a stable state shown in Figure 23, the IPSIs will each have priority lists of: A-side IPSI B-side IPSI 1st Alternative: #1 #1 2nd Alternative: #10 #10 3rd Alternative: #20 #20 Table 22 IPSI Priority Lists
Main (CLID #1) ESS #10 ESS #20

Control Network A

Control Network B

IPSI A

IPSI B

EPN #1

Figure 23 Port Network with Duplicated IPSI and Multiple ESS Clusters If a failure occurs which causes ESS #10 to be unable to keep a session up with the B-side IPSI, the following steps take place: 1. 2. 3. Socket failure is detected by B-side IPSI. The B-side IPSI informs the A-side IPSI and checks if ESS #10 is still connected to the A-side IPSI (which it is). The B-side IPSI keeps ESS #10 in its list because the A-side IPSI is still connected to ESS #10. A-side IPSI B-side IPSI 1st Alternative: #1 #1 2nd Alternative: #10 #10 3rd Alternative: #20 #20 Table 23 IPSI Priority Lists If the A-side IPSI then looses its connection to ESS #10, then the following procedural steps take place: 1. Socket failure is detected by A-side IPSI

34

2. 3. 4.

The A-side IPSI informs the B-side IPSI and checks if ESS #10 is connected to the B-side IPSI (which it is not). The A-side IPSI removes ESS #10 from its priority list because it is not connected to itself or its peer. The B-side IPSI removes ESS #10 from its priority list for the same reasons. A-side IPSI B-side IPSI 1st Alternative: #1 #1 2nd Alternative: #20 #20 3rd Alternative: none none Table 24 IPSI Priority Lists

The conclusion of this should be that an ESS server does not get eliminated from either IPSIs list in a PN unless it cannot communicate to either of the IPSIs. This concept is further extended to cover the case where the ESS cluster has duplicated servers. As long as at least one server from an ESS cluster can communicate with at least one IPSI in a port network, it will remain on the priority lists of both IPSIs in the PN. The final caveat surrounding the management of the IPSIs priority lists deals with the maximum size of these lists. If there was no cost associated with an ESS being on an IPSIs lists, then it would make sense to have maximum list sizes of 64 clusters (maximum of 63 ESS clusters along with a main cluster). However, there are a number of costs, albeit small, of having an ESS in an IPSIs priority list. The first cost is the number of heartbeat messages that are continuously being exchanged between the ESS cluster and the IPSI. While each message is small and only occurs once per second, in a mesh connectivity configuration (every ESS server connected to every IPSI) unnecessary traffic is introduced onto the network. Secondly, the IPSI needs to spend precious resource cycles and memory managing the ESS clusters on its list. And finally, since every connected IPSI gets notified whenever any changes occur, a very large number of ESS clusters on a list increase the probability of changes occurring and increase the number of notification messages that are sent out when these changes happen. Therefore, the maximum list size of an IPSI for the ESS offering is 8 clusters. In other words, the IPSIs will maintain a priority list, under normal conditions, which contains the main cluster and the 7 best (based on calculated priority scores) ESS alternatives. This leads to the question of what happens to all the other ESS clusters in the system with respect to that IPSI. The answer is that if an ESSs calculated priority score does not rank it high enough to be within the top 8, then it will be rejected. If an ESS is rejected from an IPSI it will disconnect from it and start a 15 minute timer. Once the timer expires, the ESS will re-attempt to get onto the IPSIs priority list.
ESS #30 ESS #20 ESS #10 Main (CLID #1) EPN #1
IP LAN

ESS #40

ESS #50

ESS #60 ESS #70 ESS #80 ESS #90

Figure 24 IPSI Interfacing with More Than 8 ESS Clusters The figure above shows a system with 9 ESS servers. While there is no correlation between CLID and priority ranking (except for the main cluster which is always CLID #1), this example will assume

35

administration which causes lower CLIDs to have higher priority scores. This gives the IPSI the following priority list: PN #1 1st Alternative: #1 2nd Alternative: #10 3rd Alternative: #20 4th Alternative: #30 5th Alternative: #40 6th Alternative: #50 7th Alternative: #60 8th Alternative: #70 Table 25 IPSI Priority Lists Notice that ESS #80 and ESS #90 are not on the IPSIs list even though network connectivity exists and are in a rejected state with respect to this IPSI. After the reject timer (15 minutes) expires, ESS #80 and #90 will attempt to re-connect to the IPSI only to reach the same result get rejected again. However, suppose that some network fragmentation fault takes place which prevents ESS #30, #40, and #50 from communicating with the IPSI. This will leave the IPSIs list looking as follows: PN #1 1st Alternative: #1 2nd Alternative: #10 3rd Alternative: #20 4th Alternative: #60 5th Alternative: #70 6th Alternative: none 7th Alternative: none 8th Alternative: none Table 26 IPSI Priority Lists If the network connectivity is not restored within 15 minutes, ESS #80 and #90 reject timers will expire and attempt to reconnect to the IPSI. Unlike the previous time however, the IPSI will insert them on the list as the 6th and 7th alternatives respectively. PN #1 1st Alternative: #1 2nd Alternative: #10 3rd Alternative: #20 4th Alternative: #60 5th Alternative: #70 6th Alternative: #80 7th Alternative: #90 8th Alternative: none Table 27 IPSI Priority Lists When the network faults are resolved, the isolated ESS clusters will reconnect with the IPSI and the IPSI will insert them onto its lists. When the first ESS returns, it will be inserted as the 4th alternative and all the lower ranked alternatives will be shifted down. However, when the next ESS server reconnects, the IPSI will already have a full list. This is resolved by having the IPSI reject the lowest ranked alternative, in this case ESS #90, and then insert the returning ESS cluster appropriately. This procedure will continue until the priority list goes back to how it was originally.

36

6.6 How are Communication Faults Detected ?


After a media server connects to an IPSI and is inserted onto the priority list, heartbeat handshaking commences. These heartbeats are generated from the media server once per second and sent to the IPSI. Upon reception of a media servers heartbeat, the IPSI will generate a heartbeat response and send it back to the media server. An application keep-alive is required because quick communication failure detection is needed. Therefore, a communication failure can be detected if there is an explicit closure of the TCP socket or there is a period of time that goes by without receiving a heartbeat. From the servers perspective, a heartbeat response is expected from the IPSI in response to the periodically generated heartbeats. Under normal conditions, this response is received within a few milliseconds of sending the heartbeat message. This time delay is a function of the network latency and IPSI processing time (which is extremely minimal). However, the media server only requires that a response was received before it needs to send out the next heartbeat message. If the response is not received within one second (the time interval between heartbeats), a sanity fault is encountered. If three consecutive sanity failures occur and no other data has been transferred from the IPSI to the server within that time, the server closes the socket signifying a communication failure and socket recovery (as described previously) begins. If there are connectivity problems which caused this communication failure, then the IPSI will probably not receive the TCP socket closure. In this case, the IPSI will have to detect the connectivity problem on its own. The IPSIs perspective is very similar to the servers perspective. The IPSI expects to receive a heartbeat from the server once per second and if it does not get one, a sanity failure occurs. If six consecutive sanity failures occur and no other data has been received from the server during that time, a communication failure is detected and this causes the IPSI to close the existing TCP socket. Like the previous perspective above, if there is a connectivity problem, then it is up to the server to self-detect the issue since it probably will not receive the TCP socket closure from the IPSI.

6.7 Under What Conditions do IPSIs Request Service from ESS Clusters ?
Up to this point in the discussion, how the IPSIs create and manage their priority lists has been covered, but not for which it is used. The discussion now shifts gears to examine how IPSIs use these ESS ranked lists to request service when needed. In order to eliminate any ESS contention over controlling an IPSI, the power of selecting an IPSIs master is done by the IPSI itself. The IPSIs follow the basic rule that if no service is being supplied, request an ESS cluster to take control. In the Background section, it was shown that there are two different ways that a port network can be controlled either directly though a co-resident IPSI or indirectly through an EI board when leveraging an ATM or a CSS PNC. When a port network is being controlled, an arch-angel has been designated which is the master of all other angels within the PN. By definition, if an angel has an arch-angel as a master, the angel is referred to as being scanned. Using this definition, an IPSI is said to be getting service if it has a master cluster or is being scanned. There are a few implications to this. First and foremost, if an IPSI has a media server which has declared itself the IPSIs master and remains connected, it will not request service from another ESS cluster. This allows the current master server to attempt all other recoveries, as described in the Reliability - Single Cluster Environments section, without having the port network failover to another cluster. This is critical as failing over to an ESS server should only occur after all other recovery methods have been exhausted. For example, the recovery plan to address a failure may consist of IPSI interchanges or server interchanges and if the IPSI requests service from another cluster it would prevent these operations. The second implication is that if the port network is being controlled indirectly through another PN, then the IPSI should not attempt to find its port network another master. One of the recoveries previously discussed is to have a port network go into fallback mode upon a connectivity failure if possible. In Figure 25 below, the IPSI in port network #2 cannot communicate with the main cluster and therefore shifts the ESS to the first alternative. However, the main cluster gained control of the port network through the PNC. Even thought the IPSI in PN #2 does not have a master, it should not request service since the port network is up and functioning.

37

Main (CLID #1)

ESS #10

Main (CLID #1)

ESS #10

LAN

LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #1

IPSI EPN #2 EI
EAL #2

EPN #1 EI

EPN #2 EI

EPN #1 EI

IPSI #1 Priority List #1 #10

PNC Normal Operation

IPSI #2 Priority List #1 #10

IPSI #1 Priority List #1 empty

PNC Fallback Recovery Preferred over ESS Failover

IPSI #2 Priority List #10 empty

Figure 25 Fallback Recovery Occurs is Preferred to ESS Takeover Another topic covered in the Reliability Single Cluster Environments section is the recovery methods used if an IPSI and server become disconnected. It is stated that if the connectivity is restored within 60 seconds, then the server shall regain control of the port network via a warm restart (which does not affect stable calls). Therefore, an IPSI gives its previous master some time, referred to as the no service time, to re-connect after a connectivity failure before requesting service from another ESS cluster. If an IPSI loses its connection to its master cluster, then it starts a no service timer. If the cluster reconnects within the no service timer, the IPSI remains under its control. If the no service timer expires and the IPSI is not getting scanned, then the IPSI requests service from the highest ranked ESS cluster in its list. Since IPSIs all act independently with respect to requesting service, there are times that could cause a single system to fragment into smaller autonomous systems. In Figure 26 below, the IPSI in port network #3 requests service from the ESS which it can communicate with after the no service timer expires. If the network then heals itself the main cluster reconnects to PN #3s IPSI. In this case, the IPSI will place the main cluster as the 1st alternative and shift the ESS, which is currently controlling the IPSI, to the 2nd alternative. It is very important to realize, that a reconnection of a higher ranked ESS does not imply that the IPSI will shift control over to it. The IPSI in PN #3 will remain under the control of the ESS until another fault occurs or control is overridden (discussed in the next section). In this situation, every port network provides service to all of its end users, but does not get all the advantages of being one system such as availability of centralized resources, simple inter-port network dialing, and efficient resource sharing. Therefore, there is a trade-off between waiting for a previous controlling cluster to return (during that time no service is being provided to the port network) and requesting service from another ESS cluster (which causes system fragmentation). If an outage is only going to last a few minutes, it may be worth while to be out-of-service during that time, but remain as one large system when the outage is rectified. Since no crystal balls exist which can be queried when an outage occurs to determine how long the outage will be, the IPSI relies on the no service timer. This no-service timer is configurable by the customer with a range of 3 to 15 minutes. The minimum of three minutes was derived from waiting at least one minute to allow pre-ESS recoveries and then two additional minutes to avoid fragmentation.

38

Main

ESS

Main

ESS

IP LAN

IP LAN

EPN #1

EPN #2

EPN #3

EPN #1

EPN #2

EPN #3

Single System

Fragmented Network Two Autonomous Systems

Figure 26 Autonomous Systems Created Upon ESS Takeover The final piece of the requesting service discussion involves the duplication of IPSIs within a port network. The previous section covered how the priority lists between the peer IPSIs are guaranteed to be identical, but the service requests must also be synchronized. The IPSI is a conglomeration of a number of different components with one being the tone clock. When two IPSIs are within the same PN, one of the IPSI tone clocks drives the port network and the other IPSIs tone clock is in a standby mode. The IPSI which has the active tone clock, otherwise known as the selected IPSI, owns the responsibility of the no service timer and when the request for service will be sent. Upon a connectivity failure from the master cluster to both IPSIs in the PN, the selected IPSI starts the no service timer. When the no service timer expires, the selected IPSI sends out a request for service and also instruct its peer IPSI to do the same via the backplane communication channel. It is important to see however, if the master cluster only loses connectivity to the selected IPSI and not the standby IPSI, the controlling server will attempt an IPSI interchange. This IPSI interchange, driven completely by the server, causes the previously standby IPSI to become the port networks active, or selected, IPSI. The following figure shows the handshaking that occurs after an IPSI decides to send a request for service to its highest ranked alternative ESS once the IPSIs no service timer expires. If the handshaking is not successful and the requested ESS does not takeover the IPSI, the IPSI will attempt to get service from the next alternative in its priority list. Media Server Service Request Control Takeover Message Notification Update Message (Takeover Confirmation Included) IPSI Notification update message gets sent to all media servers currently connected to the IPSI

Figure 27 Request For Service Message Flow

39

6.8 How Does Overriding IPSI Service Requests Work ?


When catastrophic failure situations occur, ESS servers will assume control of all port network gateways which cannot be given service by the main servers. As shown, this can occur for two reasons main server goes out-of-service or the network prevents communication from the IPSI to the server. Eventually, the failure, of either type, will be fixed and the conditions whereby the IPSI shifts control over to the ESS server no longer exist. However, since shifting control between clusters is not connection preserving for all types of calls, the IPSI does not automatically go back to the main server. As covered in the last section, the IPSI does not request service unless it does not have a current master. Hence, unless there is an override to the master selection process, the port networks which are being controlled by ESS servers will never return to be part of the main system again until another fault occurs. The ESS offering introduced a new command into the already large suite supported by CM to address this situation. The command get ipserver-interface forced-takeover <PN # / all> can be issued from the main cluster in order to have the IPSI revert control over to it. This command must be used with caution because it causes a service outage on the port network that it is forcing over under its control. The IPSI, upon reception of the unsolicited takeover message (all other takeovers are in response to a service request) from the main cluster, informs its current master cluster it is shifting control back to the main cluster. Along with the manual agglomeration command introduced here, there are methods to put the system back together on a scheduled basis. If a fault occurs that requires ESS servers to assume control of some or all of the port networks, the system can be administered to converge back to the main cluster at a specified time. However, unlike the manual command, all the port networks under control of ESS servers are shifted over, and therefore, being reset at the same time. There is never a good time to have an outage, but controlling when it occurs is much better than the alternative unplanned service outages.

Section 7 - ESS in Control


This section is broken into two topics how existing calls are affected by failing over to ESS control and what the operational consequences are if a system becomes fragmented. ESS servers offer equivalent service (no feature debt and no performance degradation) to resources they assume control of as the main servers provided. However, there are many types of conditions when ESS servers only take control of some of the port networks in the system and that has a number of caveats. These topics include how dialing patterns may change in fragmented situations and what happens to centralized resources such as voicemail and other adjuncts such as CMS and CDR.

7.1 What Happens to Non-IP Phone Calls During Failovers ?


If a failure occurs which requires an ESS to assume control of a port network, calls supported by that PN are affected drastically. As previously discussed, when the control of a PN shifts between entities that do not share call state data, the control transfer process requires a cold restart of the PN. This cold restart causes all TDM resource allocation to be reset and therefore any calls using those resources to be torn down. Since all analog phones, digital phones, and PRI trunks always use their associated port networks TDM bus for a call, the calls they are members of are reset during this failover. IP telephony does not follow the same paradigm and is discussed in the next section. The following timeline shows the effects on the calls when recoveries take place.

40

60 seconds

No Service Timer

If previous master returns, EPN warm restart recovery. All calls within the EPN are preserved.

If previous master returns, EPN cold restart recovery. All calls within the EPN are dropped.

ESS takes over which causes an EPN cold restart recovery. All calls within the EPN are dropped.

Figure 28 Call Effects Based on Different Recovery Mechanisms It is very important to understand what calls are preserved means. If a call is preserved, that means the bearer connection and the call control are both unaffected when the recovery is complete. The next section, which deals with IP telephony, relies on the concept of connection preservation. If only the connection is preserved, then the bearer connection is unaffected, but the call control is lost. For example, if a user is on a call and a call preserving recovery takes place, there is no effect on the end user. However, if a user is on a call and a connection preserving recovery occurs, the communication pathway is undisturbed, but all call control is lost (e.g. the user will be unable to transfer or conference the call in the future). Another question that needs to be answered is what happens to the calls while waiting for a recovery. Stated differently, if a port network fails over to an ESS server after five minutes, what happened to the existing calls during that time? The answer is that the bearer connection continues to persist until the port network is restarted, but with zero call control (cannot put the call on hold, retrieve another call appearance, or even hang-up). This comes from the fact that if a PN loses its control link it preserves the current resource allocations. The following figure shows five phone calls which are currently active. The phones in this example are non-IP endpoints such as digital or analog.

PSTN EPN #1 EPN #2 EPN #3 EPN #4 5 PNC

PSTN

Figure 29 Effects on Non-IP Phone Calls during Failover Suppose the two port networks on the right, PN #3 and PN #4, encounter some condition which causes them to failover to an ESS server. Calls #1 and #2 are not affected by this failover since their supporting port networks does not have a control shift and they are not using any resources on PN #3 or PN #4. Call #3 drops because it is not completely independent of PN #3 (the controlling server of PN #2 tears down call #3 once it loses control of all the resources it was managing involved with the call). Calls #4 and #5 also drop because the ESS taking over performs an EPN cold restart on port networks #3 and #4.

41

7.2 What Happens to IP Phone Calls During Failovers ?


A traditional TDM phone (e.g. analog, digital, or BRI) is hardwired to a circuit pack which supports it. Through this wire, the phone receives signaling (e.g. when to ring or what to display). In addition, the wire is also used as the medium for bearer traffic. An H.323 IP phone does not have a single dedicated pathway for both of these functions.

Digital Line Card


Signaling & Bearer

MEDPRO

CLAN

Bearer Signaling

IP Network
Digital / Analog Phone Bearer

2
H.323 IP Phone H.323 IP Phone

Figure 30 Control and Bearer Paths for IP and TDM Phones The H.323 IP phone and the CLAN establish a logical connection to be used for call control signaling. The MEDPRO and the IP phone exchange RTP streams when certain types of bearer paths are required. For instance, when an IP phone dials the digital phones extension (call #1 in Figure 30), all of the signaling goes through the CLAN which is the gatekeeper for the SPE controlling the port network. The actual bearer communication pathway goes from the phone over IP to the MEDPRO, which is a conduit onto the TDM bus and then to the digital phone. If the IP phone called another IP phone (call #2 in Figure 30), the signaling would still go through the CLAN, but the bearer path will be directed at the terminating IP phone rather than the MEDPRO. Unlike digital phones whereby a failure of the circuit pack supporting it means it becomes inoperable, IP phones have the ability to shift from where they are getting control. In other words, if an IP phone fails to communicate with its current gatekeeper, the CLAN in this case, it attempts to get service from another one within the system. While the IP phone seeks out an alternate gatekeeper it keeps up its current bearer path connection. If the IP phone attempts to register with a CLAN that is being controlled by the same entity which is controlling its previous CLAN gatekeeper (shown on the left side of Figure 31), a control link is immediately established and the call is preserved. The IP phone continues to search gatekeepers until this condition is met. If it fails to do this, the IP phone will keep the existing connection up until the end user decides to terminate the call (e.g. hang-up). After the end user hangs up, the IP phone will search out a gatekeeper without the conditions just discussed.

42

Main IP

ESS

Main IP

ESS

controlled by same server

IPSI EPN #1 CLAN

IPSI EPN #2 CLAN failed control link new control link, established during call

IPSI EPN #1 CLAN

IPSI EPN #2 CLAN

controlled by different servers

IP failed control link

IP

new control link, established after call is ended

Call Preserving

Connection Preserving

Figure 31 H.323 IP Phone Link Bounce That being said, if an IP phone is not using any port network resources for the bearer portion of its call, the connection is preserved through a failover. If the port network supporting these three calls in Figure 32 below fails over to an ESS, only calls #1 and #2 are dropped since PN resources are being used from the bearer path. Call #3 has two IP phones communicating directly with each other (a shuffled call) and is connection preserving through the failover.

Digital Line Card 1

MEDPRO

DS-1 Card

PSTN

2 IP Network

Digital / Analog Phone

H.323 IP Phone

3
H.323 IP Phone H.323 IP Phone

Figure 32 Effects on IP Phone Calls during Failover

7.3 What Happens to H.248 Gateway Calls During Failovers ?


Up to this point in the paper the only gateways that have been discussed were port networks. There is another series of gateways that the Avaya Communication Server supports H.248 gateways (G700, G350, and G250). These gateways, which support digital and analog phones, PRI trunks, and VoIP resources, get controlled by the SPE indirectly through CLAN gatekeepers as H.323 IP phones do and also share the ability to seek out alternate gatekeepers. Therefore, since ESS servers provide survivability to the CLAN gatekeepers, they also provide survivability for H.248 gateways. This section covers the effects of calls

43

supported by H.248 gateways as they failover to the control of an ESS. The ability for H.248 gateways to also utilize Local Spare Processors (LSP) as alternate sources of control is discussed in the LSP/ESS Interaction section of this paper.
SPE

IP
Digital / Analog Phone

IPSI EPN #1 Digital Line Card

CLAN

MEDPRO

1 PSTN

Signaling

IP

Bearer

2 H.248 Gateway 3
Digital / Analog Phone

Digital / Analog Phone

Digital / Analog Phone

Digital / Analog Phone

Figure 33 H.248 Gateway Integrated into the System The figure above shows how H.248 gateways integrate into the communication system. As with H.323 IP phones, the H.248 gateway receives its control from the SPE via the CLAN gatekeeper. If the gateway loses its control link to the CLAN, it seeks out other possible gatekeepers that could provide it service. The gateway has a pre-programmed fixed failover list of alternate gatekeepers if a failure occurs. The gateway registers and brings up a control link to the first alternate gatekeeper that it can contact. If the gateway registers with a CLAN that is controlled by the same SPE which controlled its previous CLAN, then all the calls on the gateway are preserved (left side of Figure 34 below). If the gateway registers with a CLAN that has a master which is unaware of its calls, then the failover is connection preserving only (right side of Figure 34 below).

44

Main

ESS

Main IP

ESS

IP

controlled by same server

IPSI EPN #1 CLAN

IPSI EPN #2 CLAN failed control link new control link, established immediately

IPSI EPN #1 CLAN

IPSI EPN #2 CLAN

controlled by different servers

IP failed control link H.248 Gateway

IP H.248 Gateway

new control link, established immediately

Call Preserving

Connection Preserving

Figure 34 H.248 Gateway Control Link Bounce Figure 33 shows how H.248 gateways integrate in the system and some example calls. If the port network containing the CLAN gatekeeper fails over to an ESS server, the existing calls are affected. Call #1 would be dropped because the bearer path of the call traverses the TDM bus of the port network. However, calls #2 and #3 only use the H.248 gateway resources for the bearer path and therefore do not lose their connections over the failover to an ESS.

7.4 How are Call Flows altered when the System is Fragmented ?
Phone #2 Phone #1 Phone #3 Phone #4

4 PSTN

5 PSTN

1 EPN #1

2 EPN #2

PNC

Figure 35 Phone Calls on Non-Faulted System The system laid out above has four digital phones with extensions 2001, 2002, 2003, and 2004 respectively and two PSTN trunks. In addition, suppose there is a route pattern which attempts trunk #1 and then trunk #2. The following table shows the example calls and the steps involved to establish them.

45

Phone Call #1: Phone #1 ! Phone #2

#2: #3: #4:

#5:

Establishment Steps Phone #1 dials 2002. SPE identifies 2002 as phone #2s extension. SPE is in control of phone #2 and knows it is in service. 4. SPE rings phone #2 and the user answers. 5. SPE connects a bearer path between the phones. Phone #3 ! Phone #4 Same as previous call. Phone #2 ! Phone #3 Same as previous call. Phone #1 ! External Number 1. Phone #1 dials external number. 2. SPE determines route pattern for dialed number. 3. SPE gets 1st option of route pattern (trunk #1). 4. SPE is in controls trunk #1 and knows it is in service. 5. SPE sends call out to the PSTN through trunk #1. Phone #4 ! External Number Same as previous call. Table 28 Phone Call Establishment Steps (Non-Faulted) 1. 2. 3.

Some of the five call flows just reviewed change drastically if the system becomes fragmented. The following table examines the steps taken during the same call examples when port network #2 has failed over to an ESS server. It is very important to note that these flow changes only occur because the system is fragmented. If the entire system (PN #1 and PN #2) failed over to ESS control, then the call flows would remain unchanged.
Phone #2 Phone #1 Phone #3 Phone #4

4 PSTN

5 PSTN

1 EPN #1

2 EPN #2

PNC

Call

does not succeed.

Figure 36 Phone Calls on Fragmented System Phone Call #1: Phone #1 ! Phone #2 #2: Phone #3 ! Phone #4 #3: Phone #2 ! Phone #3 Establishment Steps Identical to non-faulted call flow. Identical to non-faulted call flow. Phone #2 dials 2003. SPE identifies 2003 as phone #3s extension. SPE does not control phone #3 and therefore assumes it is out-of-service. SPE send the call to phone #3s defined coverage path. If an in service coverage path termination point is found, the call is routed there; otherwise, re-order tone is given to the originator (phone #2). (see below how phone #1 can successfully call phone #3 in a fragmented system)

1. 2. 3. 4. 5.

46

#4: Phone #1 ! External Number #5: Phone #4 ! External Number

Identical to non-faulted call flow. Phone #4 dials external number. SPE determines route pattern for dialed number. SPE gets 1st option of route pattern (trunk #1). SPE does not control trunk #1 and therefore assumes it is out-of-service. 5. SPE then gets 2nd option of route pattern (trunk #2). 6. SPE is in control of trunk #2 and knows it is in service. 7. SPE sends call out to the PSTN through trunk #2. Table 29 Phone Call Establishment Steps (Fragmented System) 1. 2. 3. 4.

If a user desired to call a phone which was under control of a portion of the system which has been fragmented away, then the user will have to dial differently. Assuming that DID numbers map to trunks local to the user, if the originating phone can dial the terminating phones extension as an external number, the SPE processes it as such and routes it out through a PSTN trunk under its control. The PSTN would then route the incoming call to the port network which is fragmented away. The ESS SPE would then route the incoming PSTN call to the appropriate extension. Using the previous example, call #3 with phone #1 dialing phone #3s extension as an external number would succeed as follows:
Phone #2 Phone #3 Phone #4

outgoing PSTN call

Phone #1

outgoing PSTN call

PSTN EPN #1 EPN #2

PSTN

call routed over the PSTN

PNC

Figure 37 Extension to Extension Call Routed over PSTN In addition, when a system enters into a fragmented situation, different call routing may be desired. In the How does ESS Works section, it was discussed that changes can be made to ESS servers translations, but any changes will be lost when the ESS reset. This behavior can be leveraged to have ESS systems utilize resources differently than the main server does without effecting the permanent systems translation set. For example, if the disaster plan requires that all incoming calls to the part of the system which is fragmented away get rerouted to a recorded announcement, administration changes can be made on the ESS system to accomplish this. However, these changes are not reflected on the main server and therefore once the port network returns to the main server, the calls are handled as normal. This manual technique can be used if a system is going to be staying in a fragmented state for an extended period of time.

7.5 What Happens to Centralized Resources when a System is Fragmented ?


A big advantage to collapsing autonomous systems together is the more efficient usage of resources such as trunks, tone detectors, and announcements. Another advantage is the centralization of adjuncts. For example, rather than having two distinct CDR systems serving two distinct systems, one centralized CDR can serve one larger PBX switch. However, the cost benefit of doing this must be weighed against the risk of being isolated away from the resource in failure situations. Suppose two separate systems (shown on the

47

left side of Figure 38) are collapsed together and cost savings can be realized by the elimination of two PSTN trunks (shown on the right side of Figure 38). The danger of eliminating the trunks in this way (two from site #2 rather than one from each) is upon a network fragmentation failure which causes the ESS server to takeover EPN #2 and becomes its own stand-alone system, the users under the ESS servers control are functional, but have no trunk access.
Main Site #1 Main Site #2 Main ESS

IP

IP

IP

IPSI EPN #1

IPSI EPN #1

IPSI EPN #1

IPSI EPN #2

TR #1

TR #2

TR #3

TR #4

TR #1

TR #2 PSTN Collapsed Systems

PSTN Separate Systems

Figure 38 Two Systems Collapsed Together Many adjuncts interface with the communication system over IP through a CLAN gatekeeper such as CMS and CDR. These adjuncts, unlike H.323 IP phones and H.248 gateways, do not have the ability to find an alternate gatekeeper if a problem arises. For that reason, the adjuncts are tightly associated with a CLAN, or in other words, these adjuncts are bound to the port network. The following diagram shows an adjunct interfacing with the system. As long as the link is up, the SPE continuously transmits pertinent data over the adjunct link. The service state of the link is inherently tied to the state of the port network and the state of the CLAN. With these short-comings, the communication between adjuncts and the SPE is increased by having multiple communication links a primary link and an optional secondary link. The SPE does not treat them as an active/standby pair, but rather as both simultaneously functioning in an active mode (the relevant data is transmitted on both links concurrently). In some cases, both links may terminate to the same physical adjunct (shown below) and in others, the links may terminate to different physical adjuncts.

48

Main

ESS

Main

ESS

IP

IP

IPSI EPN #1 CLAN

IPSI EPN #2

IPSI EPN #3

IPSI EPN #1 CLAN

IPSI EPN #2 CLAN

IPSI EPN #3

IP

primary link

IP

IP

primary link secondary link

Adjunct

Adjunct

Primary Adjunct Link

Primary & Secondary Adjunct Links

Figure 39 Adjuncts Interfacing over IP with PBX If a catastrophic server failure takes place and the entire system is taken over by an ESS cluster, the ESSs SPE re-establishes the adjunct links and sends the appropriate data. The ESS SPE gained control of the adjunct links once it gains control of the port network with which it is associated. The interesting cases are when the system fragments. Assume a network failure which causes PN #2 and PN #3 to failover to the ESS server (left side of Figure 40 below). The main server continues to supply service to PN #1 and therefore still communicates over the adjunct link. The ESS server, on the other hand, controls PN #2 which has the other adjunct link. Therefore it transmits appropriate data over that link. In this situation, both fragmentations are still interfacing with the adjunct.
Main ESS Main ESS

IP

IP

IPSI EPN #1 CLAN

IPSI EPN #2 CLAN

IPSI EPN #2

IPSI EPN #1 CLAN

IPSI EPN #2 CLAN

IPSI EPN #2

IP

IP under ESS control

IP

IP under main SPE control

Adjunct

Adjunct

ESS Fragment Interfaces with Adjunct

ESS Fragment Does Not Interface with Adjunct

Figure 40 Adjuncts Working in Fragmented Situations

49

The right side of the figure above shows a different network fragmentation whereby EPN #3 is isolated away from the rest of the system. In this case, the main server still owns both the primary and secondary links and continues to pump relevant data to the adjuncts over them. The autonomous system that was created made up of EPN #3 and the ESS server is unable to interface or utilize the adjuncts to which it no longer has access. For example, suppose the adjunct links in the example were CDR links. During this fragmented operation, all calls on the main servers portion of the switch would be sent over the primary and secondary CDR links. Unfortunately, the fragmented portion of the system would not have any medium to output the call data on and after the SPEs internal CDR buffers filled up, the data would be lost.

7.6 What Happens to Voicemail Access when a System is Fragmented ?


Phone #1

call rerouted to voicemail

Voicemail System

EPN #1

PNC

EPN #2

PSTN incoming call to phone #1

Figure 41 Call Rerouted to Voicemail The figure above shows a voicemail system interfaced to the PBX over trunks that connected with PN #1. Any calls that are attempting to terminate to phone #1, but go unanswered, are rerouted to phone #1s coverage point which is usually voicemail. The SPE sends the call to the voicemail system and informs it who the original call is for, who is calling, and the reason the call is being rerouted to the voicemail system (e.g. busy or ring/no answer). With this information the voicemail system can place the call into phone #1s mailbox and plays the appropriate greeting. After a message is left for the phone, the voicemail instructs the SPE to turn on phone#1s message waiting indicator (MWI). If phone #1 dials the voicemail system to get its messages, the voicemail system receives the appropriate information to identify phone #1 and place it in its mailbox. If this information about the calling and called parties is not supplied, the voicemail system provides a generic greeting and allows the calling party to manually identify itself for message retrieval or whom they are calling in order to leave a message. Suppose a failure occurs which causes the system to fragment in a way that PN #2 falls under the control of an ESS server. In this situation, a call needing to go to phone #1s coverage point will need an alternate route because the voicemail system is no longer directly accessible from the fragmented portion of the switch. The SPE on the ESS server is intelligent enough to reroute the call over the PSTN to get to the voicemail system. In this rerouted call through the PSTN the ESS SPE provides all the information needed to allow the call to enter the correct mailbox. The same procedure is followed when phone #1 desires to check its messages and the voicemail system puts phone #1 into the appropriate mailbox. The only draw backs to operating in this environment is that all calls going to voicemail coverage on the fragmented site traverse the PSTN and the message waiting indicators on the phones not on the same part of the system as the voicemail are non-functional (no indication of new messages).

50

call rerouted to voicemail

Phone #1

Voicemail System

EPN #1

PNC

EPN #2

PSTN incoming call to phone #1

Figure 41 Call Rerouted to Voicemail Over PSTN

Section 8 - ESS Variants Based on System Configuration


The ESS offering ensures port networks are able to receive control from alternate sources upon a catastrophic failure of the main servers or lack of network connectivity to them for an extended period of time. Up to this point in the paper, recovery descriptions and explanations have been done independently of the systems configuration. Unfortunately, there are subtle behavioral differences in port network operation depending on the type of PNC to which the PN is connected. Therefore, this section simulates both types of failure scenarios for each type of PNC configuration available and discusses the slight variations in operational behavior that occur.

8.1 Basic IP PNC Configured System


A port network can be controlled in two different manners either directly through a resident IPSI or indirectly from a non-resident IPSI tunneled through the PNC. As the Background section covered, an IP PNC does not support tunneling of control links, so every port network using an IP PNC must contain an IPSI in order to be operational. In addition, the point of an IP PNC is to transmit bearer communication between port networks over an IP network. The figure below shows a system which is levering an IP PNC.
Main
IPSI Control Links

ESS

Control LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

IPSI
EAL #6

EPN #1 MEDPRO

EPN #2 MEDPRO

EPN #3 MEDPRO

EPN #4 MEDPRO

EPN #5 MEDPRO

EPN #6 MEDPRO

Inter-PN Bearer Traffic

IP PNC

Figure 42 Non-faulted System with IP PNC

51

In a non-faulted situation, every PN in the system is receiving control from the main server directly over IP (no tunneled control). In addition, every IPSI has created a priority list consisting of the main cluster followed by the ESS cluster.

8.2 Catastrophic Server Failure in an IP PNC Environment


The simplest scenario to describe is the failover of an entire system (every port network) to an ESS cluster that is configured to have its port networks use an IP PNC. The figure below shows a catastrophic failure leaving the main cluster inoperable and all the IPSIs failing over to the ESS cluster.

Main

IPSI Control Links

ESS

Control LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

IPSI
EAL #6

EPN #1 MEDPRO

EPN #2 MEDPRO

EPN #3 MEDPRO

EPN #4 MEDPRO

EPN #5 MEDPRO

EPN #6 MEDPRO

Inter-PN Bearer Traffic

IP PNC

Figure 43 Catastrophic Server Failure in an IP PNC Environment The following list discusses the steps that occur when the non-faulted system (Figure 42) encounters a server failure causing all the systems port networks to transition over to an ESS (Figure 43). 1. 2. 3. 4. 5. 6. 7. 8. An event renders the main cluster inoperable. The socket connections between the main cluster and the IPSIs all fail. Each IPSI independently detects the loss of its control link. Each IPSI removes the main cluster from its priority list and moves the ESS cluster into the 1st alternative slot. Each IPSI starts its own no service timer. When each IPSIs no service timer expires, it requests service from its 1st alternative (the ESS). The ESS acknowledges each IPSIs service request by taking control of it. Every port network is then brought back into service via a cold restart.

After the failover is completed, the system operates exactly the same as before the failure since one autonomous system image is preserved (a non-fragmented system), the IP PNC is still used as the connectivity medium for inter-port network bearer traffic, and the ESS servers have no feature debt, identical RTUs, and no performance compromises as compared to the main server. If the main server comes back into service, it will inform all the IPSIs that it is available to supply them service if needed, but

52

the IPSIs will remain under the control of the ESS until the system administrator decides to transition them back.

8.3 Network Fragmentation Failure in an IP PNC Environment


The next scenario examined is the effect on the system if an event occurs that fragments the control network, as shown in figure below, causing part of the system to failover to an ESS server.

Main

IPSI Control Links

ESS

Control LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

IPSI
EAL #6

EPN #1 MEDPRO

EPN #2 MEDPRO

EPN #3 MEDPRO

EPN #4 MEDPRO

EPN #5 MEDPRO

EPN #6 MEDPRO

Inter-PN Bearer Traffic

IP PNC

Figure 44 Network Fragmentation in an IP PNC Environment The following list discusses the steps that occur during the transition of a non-faulted system (Figure 42) into a fragmented system (Figure 44). An event occurs that fragments the control network preventing communication between the main server and some port networks. 2. The socket connections between the main cluster and port networks #4, #5, and #6 all fail. 3. The socket connections between the ESS cluster and port networks #1, #2, and #3 all fail. 4. Each IPSI in PN #4, #5, and #6 independently detects the loss of the main servers control link. 5. Each IPSI in PN #1, #2, and #3 independently detects the loss of the ESSs keepalive link. 6. Each IPSI in PN #4, #5, and #6 removes the main cluster from its priority lists and moves the ESS cluster into the 1st alternative slot. 7. Each IPSI in PN #1, #2, and #3 removes the ESS cluster from their priority lists leaving them with no alternatives to the main cluster. 8. Each IPSI in PN #4, #5, and #6 each starts its own no service timers. 9. When each IPSIs no service timer expires, it requests service from its 1st alternative (the ESS). 10. The ESS acknowledges each IPSIs service request by taking control of it. 11. Port networks #4, #5, and #6 are brought back into service via a cold restart. 1.

53

Even though the same ESS server is being used to provide survivability as the previous example, the end users do not receive 100% equivalent service. The drawbacks to running in a fragmented mode were discussed in the ESS in Control section. The important things to note are that every port network is providing service to its end users and that both fragments still use the IP network for bearer traffic between the PNs. In the following scenarios, this is not always the case. If the network fragmentation is mended, the main server will inform port networks #4, #5, and #6 that it is available to supply them service if needed, but the IPSIs will remain under the control of the ESS until the system administrator decides to transition them back. If the network experiences the same problem again (e.g. a flapping network), the ISPI lose contact with the main server, but since it is not being controlled by it, there are no adverse effects on the end users.

8.4 Basic ATM PNC Configured System


Unlike the IP PNC discussed in the previous section, the ATM PNC supports tunneled control links and therefore not every port network is required to have an IPSI. In order for the SPE to control a non-IPSI connected PN, it must select an IPSI in another port network to indirectly support it. The SPE preferences, but with absolutely no guarantees, control for non-IPSI connected PNs to be controlled by an IPSI in its same community. The following diagram, showing a non-faulted ATM PNC configured system, has PN #1 getting service through PN #2 and PN #6 getting service through PN #5. The ATM PNC in this case is being used as the inter-port network bearer medium and also as a conduit for control links to non-IPSI connected PNs. In addition, every IPSI has created a priority list consisting of the main cluster followed by the ESS cluster.
Main
IPSI Control Links

ESS

Control LAN

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #4 EI

EPN #5 EI

EPN #6 EI

EAL #1

ATM PNC
Inter-PN Bearer Traffic

EAL #6

Figure 45 Non-faulted System with ATM PNC Be aware however, that the stability of PN #6 is inherently tied to PN #5 and the health of the ATM PNC. Therefore a reset of PN #5 leads to a reset of PN #6. In addition, the SPE must get port network #5 into service before attempting to tunnel control through it for PN #6. The implication of this is that upon a system cold restart, PN #5 becomes operational before PN #6 does. Also, if there is some malfunction of the ATM PNC whereby the communication between PN #6 is severed from the rest of the port networks, PN #6 goes out of service since, without an IPSI, there is no way to get a control link to it from the SPE. In summary, IPSI connected port networks stability does not rely on the stability of other PNs, comes into service faster than non-IPSI connected PNs after restarts, and are not completely reliant on the ATM PNC for control.

54

8.5 Catastrophic Server Failure in an ATM PNC Environment


The next scenario described is the failover of an entire system (every port network) to an ESS cluster that is configured to have its port networks use an ATM PNC. The figure below shows a catastrophic failure leaving the main cluster inoperable and all the IPSIs failing over to the ESS cluster.

Main

IPSI Control Links

ESS

Control LAN

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #4 EI

EPN #5 EI

EPN #6 EI

EAL #1

ATM PNC
Inter-PN Bearer Traffic

EAL #6

Figure 46 Catastrophic Server Failure in an ATM PNC Environment The following list discusses the steps that occur when the non-faulted system (Figure 45) encounters a server failure causing all the systems port networks to transition over to an ESS (Figure 46). 1. 2. 3. 4. 5. 6. 7. 8. 9. An event renders the main cluster inoperable. The socket connections between the main cluster and the IPSIs all fail. Each IPSI independently detects the loss of its control link. Each IPSI removes the main cluster from its priority list and moves the ESS cluster into the 1st alternative slot. Each IPSI starts its own no service timer. When each IPSIs no service timer expires, it requests service from its 1st alternative (the ESS). The ESS acknowledges each IPSIs service request by taking control of it. Every IPSI connected port network is then brought back into service via a cold restart. The SPE tunnels control links to every non-IPSI connected port network and brings them into service via a cold restart.

After the failover is completed, the system operates exactly the same as before the failure since one autonomous system image is preserved (a non-fragmented system), the ATM PNC is still used as the connectivity medium for inter-port network bearer traffic, and the ESS servers have no feature debt, identical RTUs, and no performance compromises as compared to the main server. If the main server comes back into service, it will inform all the IPSIs that it is available to supply them service if needed, but

55

the IPSIs will remain under the control of the ESS until the system administrator decides to transition them back.

8.6 Network Fragmentation Failure in an ATM PNC Environment


This next scenario, network fragmentation failure, has two variations that need to be addressed. The first and much more common variation is a failure condition that only causes the control network to fragment while leaving the ATM PNC intact. The second variation is a failure condition that causes both the control network and the ATM PNC to fragment simultaneously. This can occur if the control network and the ATM PNC network are one and the same or using the infamous backhoe explanation. The following figure shows the first variation whereby only the control network fragmented and how the system restores control to the effected port networks.
Main
IPSI Control Links

ESS

Control LAN

IPSI
EAL #2

IPSI
EAL #3

IPSI EPN #4 EI
EAL #4

IPSI EPN #5 EI
EAL #5 EAL #6

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #6 EI

EAL #1

ATM PNC
Inter-PN Bearer Traffic

Figure 47 Control Network Only Fragmentation in an ATM PNC Environment The following list discusses the steps that occur when the non-faulted system (Figure 45) encounters a control network fragmentation causing some of the systems port networks to go into fallback control (Figure 47). It is important to note that this type of failure is resolved with system capabilities which existed prior to the ESS offering. This scenario is being reviewed to show how ESS allows other recovery mechanisms, which usually have a less dramatic effect on end users, to attempt to resolve issues before the ESS jumps in to provide service. 1. 2. 3. 4. 5. 6. An event occurs that fragments the control network preventing communication between the main server and some port networks. The socket connections between the main cluster and port networks #4 and #5 both fail. The socket connections between the ESS cluster and port networks #2 and #3 both fail. The main SPE detects the loss of connectivity to the IPSIs in PN #4 and #5. The IPSIs in PN #4 and #5 independently detect the loss of the main servers control link. The IPSIs in PN #2 and #3 independently detect the loss of the ESSs keep-alive link.

56

7.

The IPSIs in PN #4 and #5 remove the main cluster from its priority lists and moves the ESS cluster into the 1st alternative slot. 8. The IPSIs in PN #2 and #3 remove the ESS cluster from their priority lists leaving them with no alternatives to the main cluster. 9. The IPSIs in PN #4 and #5 each starts its own no service timers. 10. The main SPE takes control of PN #4, #5, and #6 indirectly over the ATM PNC through the IPSIs in PN #2 and #3. 11. The IPSIs in PN #4 and #5 detect that someone is controlling their associated port networks and cancel their no service timers.

Since the port networks on the right side of the fragmentation are taken over by the main SPE through a fallback recovery (discussed in the Reliability - Single Cluster Environments section) the port networks are brought back into service via a warm restart implying no stable calls are affected. In addition, the IPSIs in PN #4 and #5 do not request service from the ESS server while the main server is controlling their port networks through the ATM PNC. If the tunneled link has a failure, then the IPSI will begin its no service timer once again and the process starts over at step #9 above. Once the network fragmentation is fixed, the main SPE will contact the IPSIs in PN #4 and #5 and immediately take control of them since the SPE knows it is already controlling (albeit indirectly) their associated port networks. After the SPE deems the connectivity between itself and the IPSIs is stable, it transitions control links from being tunneled through the PNC to the IPSI directly (control link fall-up). Up to this point in the paper, the ATM PNC has been abstractly represented as a simple ATM switched network cloud. In reality, however, the cloud may represent a single ATM switch or a number of ATM switches integrated together to form the ATM PNC network. The second fragmentation variation, whereby the control network and the ATM PNC both fragment concurrently, needs to be looked at in both frameworks. Figure #48 below shows a system that has a fragmented control network and a fragmented ATM PNC. Also shown in the figure are the ATM EI boards interfacing with a local ATM switch. The combination of both of these interconnected ATM switches creates the ATM PNC infrastructure.
IPSI Control Links

Main

ESS

Control LAN

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #4 EI

EPN #5 EI

EPN #6 EI

EAL #1 EAL #6 Inter-PN Bearer Traffic ATM Switch ATM Switch Inter-PN Bearer Traffic

ATM PNC

Figure 48 Control Network and ATM PNC Fragmentation (Multi-switched ATM Network)

57

The following list discusses the steps that occur when the non-faulted system (Figure 45) encounters a control network fragmentation and ATM PNC fragmentation causing the some of the system to be taken over by an ESS cluster (Figure 48). An event occurs that fragments the control network preventing communication between the main server and some port networks and that fragments the ATM PNC preventing fallback recovery. 2. The socket connections between the main cluster and port networks #4 and #5 both fail. 3. The socket connections between the ESS cluster and port networks #2 and #3 both fail. 4. The IPSIs in PN #4 and #5 independently detect the loss of the main servers control link. 5. The IPSIs in PN #2 and #3 independently detect the loss of the ESSs keep-alive link. 6. The IPSIs in PN #4 and #5 remove the main cluster from its priority lists and moves the ESS cluster into the 1st alternative slot. 7. The IPSIs in PN #2 and #3 remove the ESS cluster from their priority lists leaving them with no alternatives to the main cluster. 8. The IPSIs in PN #4 and #5 each starts its own no service timers. 9. When each IPSIs no service timer expires, it requests service from its 1st alternative (the ESS) since no fallback recovery took place. 10. The ESS acknowledges each IPSIs service request by taking control of it. 11. Port networks #4 and #5 are brought back into service via a cold restart. 12. The ESS tunnels a control link through the IPSI in PN #5, for this example, over the ATM PNC to port network #6 and brings PN #6 back into service via a cold restart. The left side of the system (PN #1, #2, and #3) remains under the control of the main servers and is operating as a stand-alone system. Since the ATM switch is still interconnecting port networks #1, #2, and #3, bearer connectivity between them is possible and the tunneled control link to PN #1 is still viable. Other than being in a fragmented state, the left side of the system is operating equivalently as it was before the failure. The right side of the system (PN #4, #5, and #6) experiences a cold restart when it is taken over by the ESS server, but then operates as it did before the failure occurred but in a fragmented environment (see the ESS in Control section). Since the port networks (PN #4, #5, and #6) are still interconnected via an ATM switch, inter-port network bearer traffic is still possible along with tunneled control for PN #6. When the control network is healed, the main server informs the IPSIs in PN #4 and #5 that it is available to provide service and the ESS server informs the IPSIs in PN #2 and #3 that it available to provide service. All the IPSIs make appropriate changes to their priority lists based on the recovered connectivity, but do not switch the cluster from which they are currently getting service. Immediately after the fragmentation fault occurs, the main server continuously attempts to establish tunneled control links to port networks #4, #5, and #6 through the ATM network, but fails to do so due to the lack of connectivity. After the ATM PNC is healed, the connectivity is restored, but the tunneled control links still fails because the EI boards, where the control link terminates, rejects it. The EI boards in PN #4 and #5 disallow the control link because the port network is already being controlled through the IPSIs. This is accomplished by a method discussed in the last section of this paper. The EI board in PN #6 also disallows the control link because the EI board is designed to only allow one control link at any one time and it already has a control link up to it through PN #5. Also, the ATM PNC is self-managed (see the Background section) allowing two intelligent entities (the main SPE and the ESS SPE) to both use its resources at the same time without conflicting with each other. If this was not the case, it would prevent the two systems from acting independently and still allow inter-port network bearer communication over the ATM PNC. Figure 49 below shows the same fragmentation of the control network and the ATM network as discussed above and, in this case, the ATM network consists of a single ATM switch which happens to be on the left side of the fragmentation. 1.

58

Main

IPSI Control Links

ESS

Control LAN

PN #6 is Out-of-Service

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #4 EI

EPN #5 EI

EPN #6 EI

EAL #1 ATM Switch

Inter-PN Bearer Traffic

No Inter-PN Bearer Traffic

ATM PNC

Figure 49 Control Network and ATM PNC Fragmentation (Single-switched ATM Network) The following list discusses the steps that occur when the non-faulted system (Figure 45) encounters a control network fragmentation and ATM PNC fragmentation causing the some of the system to be taken over by an ESS cluster (Figure 49). An event occurs that fragments the control network preventing communication between the main server and some port networks and that fragments the ATM PNC preventing fallback recovery. 2. The socket connections between the main cluster and port networks #4 and #5 both fail. 3. The socket connections between the ESS cluster and port networks #2 and #3 both fail. 4. The IPSIs in PN #4 and #5 independently detect the loss of the main servers control link. 5. The IPSIs in PN #2 and #3 independently detect the loss of the ESSs keep-alive link. 6. The IPSIs in PN #4 and #5 remove the main cluster from its priority lists and moves the ESS cluster into the 1st alternative slot. 7. The IPSIs in PN #2 and #3 remove the ESS cluster from their priority lists leaving them with no alternatives to the main cluster. 8. The IPSIs in PN #4 and #5 each starts its own no service timers. 9. When each IPSIs no service timer expires, it requests service from its 1st alternative (the ESS) since no fallback recovery took place. 10. The ESS acknowledges each IPSIs service request by taking control of it. 11. Port networks #4 and #5 are brought back into service via a cold restart. The left side of the system (PN #1, #2, and #3) operates in exactly the same manner as the previous failure. Since the single ATM switch resides on the left side of the fragmentation, the port networks #1, #2, and #3 1.

59

remain interconnected. The main SPE is controlling port networks #2 and #3 through the resident IPSI and port network #1 via the ATM PNC. However, the right side of the fragmentation is tremendously different than the previous example since there is no ATM switch interconnecting port networks #4, #5, and #6 which has two major implications. The first issue is that no tunneled control links are possible (no viable pathway) and therefore PN #6 cannot receive control from the SPE. Port networks #4 and #5 have resident IPSIs and therefore their service state is independent of the interconnection ATM PNC health. However, without an ATM switch interconnecting port networks #4 and #5 there will be no inter-port network bearer communication. For example, a call entering through a trunk in PN #4 destined for a user off of PN #5 would not be able to be completed and is instead routed to the called partys coverage. In other words, the port networks #4 and #5 are in service and controlled by the same SPE (the ESS server), but function as stand-alone islands without the ATM PNC connectivity. When the faulted ATM network is fixed, two major events take place. First of all, port networks #4 and #5 will be able to communicate over the ATM PNC again. Once the SPE recognizes this, it begins to allow inter-port network bearer traffic again since a viable pathway exists. Furthermore, once PN #6 can communicate with the ATM PNC, it can receive a tunneled control link to provide it service. The problem, however, is that it is non-deterministic race which SPE, the main or the ESS, gets a control link up to it first. As talked about in the previous section, when an SPE is not in control of a PN, it continuously attempts to bring up a control link through the ATM PNC. This operation usually fails because either there is no pathway for the link to be established on or the terminating EI rejects the establishment if it is already being controlled. In this case however, once the link establishments are no longer blocked by the ATM PNC fragmentation and since the main SPE and the ESS SPE do not coordinate recovery actions, it is a race condition between them in respect to who is going to get control of PN #6 first. Said differently, once the ATM PNC is healed all the port networks receive service, but there is no administrator control in the decision of which SPE controls it. The only way to ensure that non-IPSI connected port networks receive their control from a certain SPE is to have that SPE control all the IPSIs in the system. If other SPEs in the system do not control any IPSIs, they have no entry point into the ATM PNC and therefore never attempt to establish control links to the non-IPSI connected PNs. An undesirable scenario can occur if not every port network has a resident IPSI is port network called hijacking. With the example above, the main SPE brought up a tunneled control link through PN #3 to PN #6 before the ESS server did. Port network #6 would be brought into service via a cold restart and then be part of the partial system controlled by the main SPE. Since the ESS SPE does not control PN #6 it is continuously attempting to bring up a control link to it, but is blocked by the EI board since another control link is already established from the main SPE. However, if another event occurs that causes a short communication fault between the main SPE and PN #6, then PN #6 would temporarily be un-controlled (control link goes down). Typically, the main SPE re-establishes a control link very quickly and resumes controlling the SPE by bring it into service via a warm restart (no effect on stable calls). Unfortunately, there is a chance that the ESS SPE brings up its control link to port network #6 during this time and if this happens, PN #6 would be brought into service under the control of the ESS server via a cold restart (dropping all active calls) and the main SPEs attempts to restore service would be blocked. This type of event is referred to as port network hijacking. While a system configured to use an ATM PNC does not require an IPSI in every port network to be functional and be protected by the ESS offering, there are a number of drawbacks from doing so. First of all, the port networks service state is inherently tied to the availability of the ATM PNC and conditions can arise, as shown in the previous example, whereby non-IPSI port networks do not receive control from ESS clusters in failure situations. Second of all, there are no mechanisms built that allow the administrator to dictate where a non-IPSI connected port network gets control from and it is non-deterministic where a nonIPSI connected port work gets service from during recovery. And finally, recovery of non-IPSI connected port networks take longer than IPSI connected port networks do since they rely on IPSI connected port networks to be in service before tunneling control links. It can be concluded, due to the reasons mentioned, a system that has IPSIs in every port network has higher availability than systems which do not.

60

8.7 Basic CSS PNC Configured System


Another configuration covered is a system configured to use a CSS PNC. As in the ATM PNC case, the CSS PNC supports not only inter-port network bearer connections, but also a medium for tunneling control links. Therefore, a system leveraging a CSS PNC is not required to have an IPSI in every port network. The following diagram, showing a non-faulted CSS PNC configured system, has PN #1 getting service through PN #2 and PN #6 getting service through PN #5. The CSS PNC in this case is being used as the inter-port network bearer medium and also a conduit for control links to non-IPSI connected PNs. In addition, every IPSI has created a priority list consisting of the main cluster followed by the ESS cluster.
Main
IPSI Control Links

ESS

Control LAN

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #4 EI

EPN #5 EI

EPN #6 EI

EAL #1

CSS PNC
Inter-PN Bearer Traffic over CSS PNC

EAL #6

Figure 50 Non-faulted System with CSS PNC In the How are Port Networks Interconnected section the CSS PNC is described to be SPE managed. The ramifications of a PNC being SPE managed is that only one intelligent entity may utilize its resources and if two or more intelligent entities attempted to simultaneously use the PNC, many resource conflicts will be encountered causing complete CSS PNC failure. The IP PNC and ATM PNC are self managed and therefore do not have the same limitations (see Figure 44 and Figure 48 respectively). With many intelligent entities within the system (main SPE and ESS SPEs) the only way to guarantee only one SPE is utilizing the CSS PNC at one time is to block all other SPEs from using it. The main SPE and the ESS SPEs do not communicate about call state or resource allocation, so the blocking cannot be on a dynamic basis. Instead, a requirement exists in the ESS offering stating that an ESS SPE never attempts to utilize the CSS PNC under any circumstances. As the fault scenario unfold below, this has a major effect on port network recovery and the method PNs communicate with each other in a failure mode.

8.8 Catastrophic Server Failure in a CSS PNC Environment


The next scenario described is the failover of IPSI connected port networks to an ESS cluster that is configured to have its port networks use a CSS PNC. The figure below shows a catastrophic failure leaving the main cluster inoperable and all the IPSIs failing over to the ESS cluster.

61

Main

IPSI Control Links

ESS

PN #1 is Out-of-Service

Control LAN

PN #6 is Out-of-Service

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

EPN #1 EI

EPN #2
MEDPRO EI

EPN #3
MEDPRO EI

EPN #4
EI MEDPRO

EPN #5
EI MEDPRO

EPN #6 EI

EAL #1

CSS PNC

EAL #6

Inter-PN Bearer Traffic over IP PNC

IP PNC

Figure 51 Catastrophic Server Failure in a CSS PNC Environment The following list discusses the steps that occur when the non-faulted system (Figure 50) encounters a server failure causing some of the systems port networks to transition over to an ESS (Figure 51). 1. 2. 3. 4. 5. 6. 7. 8. An event renders the main cluster inoperable. The socket connections between the main cluster and the IPSIs all fail. Each IPSI independently detects the loss of its control link. Each IPSI removes the main cluster from its priority list and moves the ESS cluster into the 1st alternative slot. Each IPSI starts its own no service timer. When each IPSIs no service timer expires, it requests service from its 1st alternative (the ESS). The ESS acknowledges each IPSIs service request by taking control of it. Every IPSI connected port network is then brought back into service via a cold restart.

Abiding by the rule that ESS servers can never utilize the CSS PNC prevents a system configured with a CSS PNC from operating in the same manner or with equivalent service when being controlled by the main cluster versus being controlled by an ESS cluster. As Figure 51 shows, port networks without IPSIs (PN #1 and #6) are not provided service from the ESS server and therefore are not supplying service to their end users. This is due to the fact that without an IPSI, the only method to control a port network is with a tunneled control link through the CSS PNC and since the ESS SPE cannot use any CSS PNC resources, no pathway for the control link to traverse exists. This leads to a very basic design principle a port network in a CSS PNC environment must have a resident IPSI if it to be provided additional reliability by the ESS offering. Another side-effect of the ESS SPEs inability to use the CSS PNC is a different communication medium between the port networks is needed. If an ESS server controls multiple CSS PNC connected port

62

networks, it routes inter-port network bearer traffic over an IP PNC. In other words, an ESS SPE ignores the existence of EI boards in port networks it controls and treats the PN as if it were configured to use an IP PNC. This implies that for a port network to communicate with other PNs in survivability mode, it needs to have a MEDPRO allowing it to interface with an IP network for VoIP traffic. If the port network does not have a MEDPRO, the ESS still provides it service, but the PN becomes its own island unable to communicate with other PNs controlled by the ESS. When the main cluster is repaired, it informs all the IPSIs that it is available to provide it service if they require it. The IPSIs place the main cluster into their priority lists appropriately, but do not shift under the main clusters control automatically. The main SPE has the ability to utilize the CSS PNC to provide service to PN #1 and #6, but it does not have to an access point to do this. Once the control of one IPSI is transferred to the main SPE, it immediately tunnels control links through that IPSI to port networks #1 and #6.

8.9 Network Fragmentation Failure in a CSS PNC Environment


This next scenario, network fragmentation failure, has two variations that need to be addressed. The first and much more common variation is a failure condition that only causes the control network to fragment while leaving the CSS PNC intact. The second variation is a failure condition that causes both the control network and the CSS PNC to fragment simultaneously. This variation is very unlikely due to the fact that the control network and the CSS PNC are always completely separate networks implying the only way for this to statistically occur is if there are single points of failure that they share such as a conduit. If the wiring of the networks is done within the same physical conduit and that conduit was cut, both networks would fragment at the same time. The following figure shows the first variation whereby only the control network fragmented and how the system restores control to the effected port networks.
Main
IPSI Control Links

ESS

Control LAN

IPSI
EAL #2

IPSI
EAL #3

IPSI EPN #4 EI
EAL #4

IPSI EPN #5 EI
EAL #5 EAL #6

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #6 EI

EAL #1

CSS PNC
Inter-PN Bearer Traffic over CSS PNC

Figure 52 Control Network Only Fragmentation in a CSS PNC Environment The following list discusses the steps that occur when the non-faulted system (Figure 50) encounters a control network fragmentation causing some of the systems port networks to go into fallback control (Figure 52). It is important to note that this type of failure is resolved with system capabilities which existed prior to the ESS offering. This scenario is being reviewed to show how ESS allows other recovery

63

mechanisms, which usually have a less dramatic effect on end users, to attempt to resolve issues before the ESS jumps in to provide control. An event occurs that fragments the control network preventing communication between the main server and some port networks. 2. The socket connections between the main cluster and port networks #4 and #5 both fail. 3. The socket connections between the ESS cluster and port networks #2 and #3 both fail. 4. The main SPE detects the loss of connectivity to the IPSIs in PN #4 and #5. 5. The IPSIs in PN #4 and #5 independently detect the loss of the main servers control link. 6. The IPSIs in PN #2 and #3 independently detect the loss of the ESSs keep-alive link. 7. The IPSIs in PN #4 and #5 remove the main cluster from its priority lists and moves the ESS cluster into the 1st alternative slot. 8. The IPSIs in PN #2 and #3 remove the ESS cluster from their priority lists leaving them with no alternatives to the main cluster. 9. The IPSIs in PN #4 and #5 each starts its own no service timers. 10. The main SPE takes control of PN #4, #5, and #6 indirectly over the ATM PNC through the IPSIs in PN #2 and #3. 11. The IPSIs in PN #4 and #5 detect that someone is controlling their associated port networks and cancel their no service timers. Since the port networks on the right side of the fragmentation were taken over by the main SPE through a fallback recovery (discussed in the Reliability - Single Cluster Environments section) the port networks were brought back into service via a warm restart implying no stable calls were affected. In addition, the IPSIs in PN #4 and #5 do not request service from the ESS server while the main server is controlling their port networks through the CSS PNC. If the tunneled link has a failure, then the IPSI will begin its no service timer once again and the process starts over at step #9 above. Once the network fragmentation is fixed, the main SPE will contact the IPSIs in PN #4 and #5 and immediately take control of them since the SPE knows it is already controlling (albeit indirectly) their associated port networks. After the SPE deems the connectivity between itself and the IPSIs is stable, it transitions control links from being tunneled through the PNC to the IPSI directly (control link fall-up). It should be apparent that in this failure scenario, systems configured with an ATM PNC or with a CSS PNC behave identically. 1.

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #4 EI

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #4 EI

Carrier #1 CSS PNC


Single Carrier CSS

Carrier #1 CSS PNC

Carrier #2

Multi-Carrier CSS (Split CSS)

Figure 53 CSS PNC Layouts The CSS PNC, as discussed in the How are Port Networks Interconnected section, can be physically laid out in two manners a single carrier format or over multiple carriers (referred to as split CSS) as displayed above in Figure #53. Since an ESS server never attempts to use the CSS PNC, it is inconsequential for this discussion which layout is being used. In the case of a single carrier, the port networks on the right side of the fragment lose physical connectivity between each other that the CSS carrier provided. In the case of

64

split CSS, the port networks on the right side of the fragment may still have physical connectivity through on of the carriers making up the CSS, but the ESS server will not leverage it. Figure #54 shows a system that has a fragmented control network and a fragmented CSS PNC.
Main
IPSI Control Links

ESS

Control LAN

PN #6 is Out-of-Service

IPSI
EAL #2

IPSI
EAL #3

IPSI
EAL #4

IPSI
EAL #5

EPN #1 EI

EPN #2 EI

EPN #3 EI

EPN #4
EI MEDPRO

EPN #5
EI MEDPRO

EPN #6 EI

EAL #1

CSS PNC
Inter-PN Bearer Traffic over CSS PNC Inter-PN Bearer Traffic over IP PNC

IP PNC

Figure 54 Control Network and CSS PNC Fragmentation The following list discusses the steps that occur when the non-faulted system (Figure 50) encounters a control network fragmentation and CSS PNC fragmentation causing the some of the system to be taken over by an ESS cluster (Figure 54). An event occurs that fragments the control network preventing communication between the main server and some port networks and that fragments the ATM PNC preventing fallback recovery. 2. The socket connections between the main cluster and port networks #4 and #5 both fail. 3. The socket connections between the ESS cluster and port networks #2 and #3 both fail. 4. The IPSIs in PN #4 and #5 independently detect the loss of the main servers control link. 5. The IPSIs in PN #2 and #3 independently detect the loss of the ESSs keep-alive link. 6. The IPSIs in PN #4 and #5 remove the main cluster from its priority lists and moves the ESS cluster into the 1st alternative slot. 7. The IPSIs in PN #2 and #3 remove the ESS cluster from their priority lists leaving them with no alternatives to the main cluster. 8. The IPSIs in PN #4 and #5 each starts its own no service timers. 9. When each IPSIs no service timer expires, it requests service from its 1st alternative (the ESS) since no fallback recovery took place. 10. The ESS acknowledges each IPSIs service request by taking control of it. 11. Port networks #4 and #5 are brought back into service via a cold restart. 1.

65

The left side of the system (PNs #1, #2, and #3) remains under the control of the main servers and is operating as a stand-alone system. Since the CSS PNC is still interconnecting port networks #1, #2, and #3, bearer connectivity between them is possible and the tunneled control link to PN #1 is still viable. Other than being in a fragmented state, the left side of the system is operating equivalently as it was before the failure. The right side of the system does not recover as smoothly as in the ATM PNC environment. First of all, PNs #4 and #5 are being supplied service from the ESS server, but the inter-port network traffic is now being routed over an IP network. In addition, port network #6 is not supplied service from either the main SPE or the ESS SPE and therefore is inoperable. This implies that a CSS PNC connected port network is required to have a resident IPSI (for control) and a MEDPRO (for inter-port network bearer communication) if it is to be supplied service by an ESS in failure modes. When the control network is healed, the main server informs the IPSIs in PN #4 and #5 that it is available to provide service and the ESS server informs the IPSIs in PN #2 and #3 that it available to provide service. All the IPSIs make appropriate changes to their priority lists based on the recovered connectivity, but do not switch the cluster that they are currently getting service from. Immediately after the fragmentation fault occurs, the main server continuously attempts to establish tunneled control links to port networks #4, #5, and #6 through the CSS PNC, but fails to do so due to the lack of connectivity. After the CSS PNC is healed, the connectivity is restored, but the tunneled control links to PN #4 and #5 still fails because the EI boards, where the control link terminates, rejects it. The EI boards in PN #4 and #5 disallow the control link because the port network is already being controlled through the IPSIs. This is accomplished by a method discussed in the last section of this paper. However, the EI board in PN #6 accepts the control link because no other control links were terminating to it and the main server takes control of the port network. Unlike the ATM PNC scenario whereby the main SPE and the ESS SPEs race to control non-IPSI connected PNs in some cases, there is no race condition since only the main SPE is ever attempting to tunnel control links through the CSS PNC. After the administrator agglomerates the system by transferring control of PN #4 and #5 from the ESS server to the main server, the bearer traffic between the port networks is sent back over the CSS PNC rather than the IP PNC.

8.10 Mixed PNC Configured System


The final configuration covered in this section is a system designed to use a mixed PNC environment. In the three previous sections, the PNC configurations discussed were all inclusive meaning that all port networks shared the same PNC. In an IP PNC configured system, all of the port networks are utilizing an IP network for inter-port network bearer communication. In an ATM PNC configured system, all of the port networks are utilizing an ATM network for inter-port network bearer communication. In a CSS PNC configured system, all of the port networks were utilizing a center-stage switch for inter-port network bearer communication. The mixed PNC configuration allows some of the port networks to utilize one instance of either a CSS PNC or ATM PNC and the other port networks in the system to utilize IP PNC. Figure 55, in the ESS in Action A Virtual Demo section, shows a system using the mixed PNC configuration. A port network in a system configured to use mixed PNC operation in a failure scenario is based on what type of PNC it is interfacing with. For example, the port networks connected to either an CSS PNC or an ATM PNC operate in survivability mode as if the entire system was using CSS PNC or ATM PNC respectively and port networks connected to an IP PNC operate in survivability mode as if the entire system was configured to use IP PNC.

Section 9 - ESS in Action - A Virtual Demo 9.1 Demo - Setup


To summarize the material presented thus far and to show how it is applied to a real system, this section takes you through a virtual demo of the ESS feature. The example system, shown in Figure 55 below, is supplying phone service to three geographically disperse campuses that are interconnected through an IP WAN. The main site, on the left, is an upgraded DEFINITY switch consisting of traditional MCC cabinets interconnected with a CSS PNC. The other two locations are newer sites using G650 cabinets that are interconnected to each other and back to the main site via an IP network. This layout is another example of

66

a system configured to use a mixed PNC. Except for PN #4, all the other port networks have a resident IPSI through which the main server is using to control them by means of a direct EAL. Port network #4 is a non-IPSI connected port network and gets its control from the main server indirectly via a tunneled EAL through port network #3 in this case.
Main ESS #10 ESS #20 ESS #30

LAN IPSI EPN #4 EI EPN #5

LAN IPSI EPN #6

LAN IPSI EPN #7

IPSI EPN #1 EI

IPSI EPN #2 EI

IPSI EPN #3 EI

CSS PNC

Figure 55 Communication System Layout for Virtual Demo There are a number of steps that can be taken to increase a systems reliability including the duplication of IPSIs with in each PN, the duplication of the CSS PNC, and the addition of ESS servers. Since the focus of this virtual demo is the ESS feature and to keep the failure cases as simple as possible, standard reliability (simplex IPSIs and simplex PNC) is used in this example. Given the following list of objectives the strategic placement of ESS servers can be determined and prioritizations of them can be derived. 1. 2. 3. 4. All sites must be able to survive in isolation from the rest of the system. All sites must be able to survive a catastrophic main server failure. There is more than adequate WAN bandwidth available for port network control links and inter-port network bearer communication. Upon any failure, the number of autonomous systems created should be minimized.

Objective #1 requires an ESS server to exist at every remote location since each site needs to be operational if it gets fragmented away from the rest of the system. Objective #2 subtly requires that an ESS server also be placed at the main location because it must be able to survive a catastrophic main server failure. It is unable to achieve this if it does not have a local control source alternative and becomes isolated from the other sites. Objective #3 and objective #4 dictate how the priority scores and attributes for the three ESS clusters should be administered. Since bandwidth is not an issue and the failover objective is to minimize fragmentation, none of the ESS clusters should utilize the local preference or the local only attributes which implies they also do not need to override local preference boosts with the system preferred option (see the What is a Priority Score section for more details). In addition, the objectives do not specify a desired failover order, so there are no restrictions for the priority score rankings of the ESS clusters. The administration of the ESS clusters for this demonstration is shown in the SAT screen shot below by executing the display system-parameters ess command.

67

display system-parameters ess ENTERPRISE SURVIVABLE SERVER INFORMATION

Page

1 of

Cl Plat Server A Server B Pri Com Sys Loc Loc ID Type ID Node Name ID Node Name Scr Prf Prf Only -----------------------------------------------------------------------------MAIN SERVERS 1 Duplex 1 172.16 .192.1 2 172.16 .192.2

ENTERPRISE SURVIVABLE SERVERS 10 20 30 Simplex 10 Duplex 20 Simplex 30 ESS10-10 ESS20-20 ESS30-30 21 ESS20-21 75 100 50 1 1 1 1 1 1 1 1 n n n n n n n n n n n n n n n n n n n n n n n n

Screen Shot 1 ESS Cluster Administration The first observation from the ESS cluster administration is that ESS #10 and ESS #30 have platform form types (Plat Type) of simplex implying they are from the S8500 family of media servers and therefore have no Server B information. ESS #20 has a platform type of duplex meaning it is from the S87XX family of media servers. The second observation is that the ranked order of the ESS clusters is ESS #20, then ESS #10, and finally ESS #30 since they have priority scores of 100, 75 and 50 respectively. It is important to remember that the cluster IDs assigned come directly from the license file on each server and the main cluster always has an ID of #1. Once the ESS clusters are added to the system and installed they need to register with the main server to avoid alarming and to get updated translations. The status of the registration links and the ESS clusters can be seen in the SAT screen shot below using the command status ess clusters command. This command shows a quick view of the status of every ESS cluster in the system. It includes basic information such as if the ESS cluster is enabled (Enabled?), if the ESS cluster is registered (Registered?), and which server is active within the each cluster (Active Server ID). In addition, it also shows the administrator if the ESS clusters translation set is up-to-date (if the time stamp under Translations Updated for the ESS cluster matches the main clusters time stamp) and what software load each cluster is currently running (Software Version).

68

status ess clusters Cluster ID Cluster ID 1 10 20 30 1 ESS CLUSTER INFORMATION Active Server ID Registered? 1 10 20 30 y y y y Translations Updated 19:30 19:30 19:30 19:30 11/14/2005 11/14/2005 11/14/2005 11/14/2005 Software Version R013x.01.0.626.0 R013x.01.0.626.0 R013x.01.0.626.0 R013x.01.0.626.0

Enabled? y y y y

Command successfully completed Command:

Screen Shot 2 ESS Cluster Registration Status As described in the How do IPSIs Manage Priority Lists section, IPSIs maintain independent priority failover lists of ESS clusters. For this reason, the command status ess port-networks queries each IPSI in real-time for its priority list and then displays them for the administrator. This command is critical in verifying the actual failover behavior of the IPSIs match the desired failover behavior as administered and in determining which clusters are currently in control of which IPSIs.

status ess port-networks Cluster ID Com PN Num 1 2 3 4 5 6 7 1 1 1 1 1 1 1 1 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State 1A01 actv-aa 2A01 actv-aa 3A01 actv-aa Cntl Connected Clus Clus(ter) ID IDs 1 1 1 1 1 1 20 20 20 10 10 10 30 30 30

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc 1A01 IPSI up 2A01 IPSI up 3A01 IPSI up 4A01 EI up 1A01 2A01 3A01 3A01 5A01 6A01 7A01

5A01 IPSI up 6A01 IPSI up 7A01 IPSI up

5A01 actv-aa 6A01 actv-aa 7A01 actv-aa

1 1 1

1 1 1

20 20 20

10 10 10

30 30 30

Command successfully completed Command:

Screen Shot 3 Non-Faulted Environment IPSI Priority Lists

69

The first column (PN) on the left shows the port network number for which all the data in that row corresponds to. The next column (Com Num) shows what community the port network and its resident IPSI are assigned to. The fifth column, Port Ntwk Ste, presents the port network state, either up (in-service) or down (out-of-service), with respect to the sever executing the command. In this case, the command was run on the main server and the main cluster is currently controlling all of these port networks in a nonfaulted environment. The third and fourth columns, Intf Loc and Intf Type, along with the sixth column, IPSI Gtway Loc, describe how the port network is currently being given service by the controlling server. The interface location is the board where the EAL link is terminating to and the interface type is the type of board (EI or IPSI) where the arch-angel is residing. The IPSI gateway location field displays to the administrator which IPSI is supporting the control link for this port network. As described in the How are Port Networks Controlled section, there are two methods to control a port network -- either directly through a resident IPSI (left side of Figure 56) or indirectly though an IPSI residing in another PN (right side of Figure 56). If a port networks IPSI gateway location is the same as the port networks interface location, then it can be concluded that the port network is being controlled directly from a server through its own IPSI. Otherwise, if they do not match, the port network is getting controlled indirectly through another port networks IPSI. To determine which port network is supporting another port network in these indirect control situations, figure out what port network the IPSI gateway board resides in via the list cabinet command.
SPE LAN IPSI
EAL #1

SPE IPSI Gateway IPSI Gateway IPSI EPN #1 Interface Location (Type ! IPSI) EI EPN #2 EI LAN

EPN #1

Interface Location (Type ! EI)

PNC Direct Port Network Control

EAL #2

Indirect Port Network Control

Figure 56 Direct PN Control versus Indirect PN Control The next two columns, Pri/Sec Loc and Pri/Sec State, describe the port networks IPSI(s), if they exist. This screen shot shows that PN #1 has its primary IPSI in board location 1A01 and it is currently active supporting the arch-angel (actv-aa). If the port network had duplicated IPSIs, a second row would appear immediately below describing the secondary IPSI. Since this is a stable non-faulted environment, the primary IPSI location should also be the IPSI gateway location (since this IPSI is supporting the PN) and the interface location (since this IPSI is supporting the arch-angel). There are failure scenarios for which these three values may not be equal. For example, if the port network experienced an IPSI outage, control could be tunneled through the CSS PNC. In this case, the interface location would be the EI board (where the arch-angel resides) and the IPSI gateway location would be another port networks IPSI location (where the hybrid EAL terminates). The primary location of the IPSI would remain unchanged since that is an administered location and the primary IPSI state would be active (because it is the active IPSI regardless if it is being used or not). The rest of the row contains the results of a real-time query to the IPSIs concerning whom their master cluster is and what their priority lists currently are. The Cntl Clus ID column is the cluster ID returned by the IPSI referring to its currently controlling cluster. In this example, cluster ID #1 is returned which is the

70

main clusters default ID. The next columns, Connected Clus(ter) IDs, show the IPSIs priority list. In this example based on the administration previously done, each IPSI has a preference list headed by the main cluster (ID #1) followed by ESS clusters #20, #10, and #30. It is important to note that the entry for port network #4, the non-IPSI connected PN, has significant differences from the others. First of all, the interface type is always an EI board since there are only two types of boards which can support arch-angels, EIs and IPSIs, and no IPSIs reside in the port network. The next difference is that the interface location and the IPSI gateway location are never the same since the port network is always receiving control indirectly through another port network. And finally, there are no entries for primary / secondary IPSI locations since no IPSIs exist in the port network. The last item to analyze before introducing failure into the system is the port network view from the ESS clusters themselves. The execution of the status ess port-networks command from ESS #20 is shown below.

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 20 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State 1A01 active 2A01 active 3A01 active 5A01 active 6A01 active 7A01 active Cntl Connected Clus Clus(ter) ID IDs 1 1 1 1 1 1 1 1 1 1 1 1 20 20 20 20 20 20 10 10 10 10 10 10 30 30 30 30 30 30

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc down down down down down down

Command successfully completed Command:

Screen Shot 4 status ess port-networks from ESS #20 in Non-faulted Environment The most glaring difference from the execution of the command from ESS #20 as compared to when executed from the main server is the port network state and fields associated with it. Since the ESS server does not control the IPSI, and therefore not control the port network, from its perspective the port network does is not in service, or down. Also, the ESS server does not attempt to establish EAL links to the port network which implies there is no information to report in the interface location, interface type, and IPSI gateway fields. Another difference is that port network #4 has been omitted from the list of port networks. Abiding by the rule that ESS servers cannot tunnel control links through the CSS PNC and given that PN #4 is a non-IPSI connected port network, it is impossible for the ESS server, under any conditions, to take control of port network #4 or provide any relevant status of it. This same viewpoint is shared by all of the ESS clusters in this non-faulted situation.

9.2 Demo Catastrophic Main Server Failure


Now that the setup of the demo has been reviewed, it is time to introduce the first major fault into the system - a catastrophic main server failure. Suppose a cataclysmic event occurs that renders the entire

71

main cluster inoperable. Within a few seconds of the failure, the IPSIs detect that the connection to the main cluster is down which causes them to adjust their priority lists by removing the mains cluster ID from it and starting their no-service timers. Screen Shot #4 shows the results of the status ess port-networks executed from ESS #20 again after the failure, but before the IPSIs no service timers have expired.

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 20 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State 1A01 active 2A01 active 3A01 active 5A01 active 6A01 active 7A01 active Cntl Connected Clus Clus(ter) ID IDs * * * * * * 20 20 20 20 20 20 10 10 10 10 10 10 30 30 30 30 30 30

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc down down down down down down

Command successfully completed Command:

Screen Shot 5 Viewpoint from ESS #20 Immediately after Main Cluster Failure There are two very important pieces of information that the results of the command give the administrator. For every IPSI in the system, the controlling cluster ID transitioned from #1 (the main clusters ID) to *. The * implies to the administrator that the IPSIs are reporting they do not currently have a controlling cluster. Also, each IPSIs connected cluster IDs list has been adjusted by removing the main clusters ID and shifting all the ESS clusters up one position. Since the main cluster no longer appears on any list, the administrator can determine a catastrophic event has taken the main servers out-of-service. All of the other ESS clusters in the system share the same viewpoint as ESS #20. After the no service timer for each IPSI expires, the IPSI sends a request for service to its highest ranking cluster in the priority list ESS #20. ESS #20 receives the request and takes over all the port networks, establishes a control link, and brings the IPSIs PN back into service via a cold restart. PN #4 does not have a resident IPSI and since the ESS servers do not support tunneled control links, port network #4 remains non-functional. From this point forward in the demo, it is important to understand the viewpoints of the system from both ESS #20 and ESS #30.

72

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 20 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State 1A01 actv-aa 2A01 actv-aa 3A01 actv-aa 5A01 actv-aa 6A01 actv-aa 7A01 actv-aa Cntl Connected Clus Clus(ter) ID IDs 20 20 20 20 20 20 20 20 20 20 20 20 10 10 10 10 10 10 30 30 30 30 30 30

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc 1A01 IPSI up 2A01 IPSI up 3A01 IPSI up 5A01 IPSI up 6A01 IPSI up 7A01 IPSI up 1A01 2A01 3A01 5A01 6A01 7A01

Command successfully completed Command:

Screen Shot 6 Viewpoint from ESS #20 after Failover due to Main Cluster Failure

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 30 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State active active active active active active Cntl Connected Clus Clus(ter) ID IDs 20 20 20 20 20 20 20 20 20 20 20 20 10 10 10 10 10 10 30 30 30 30 30 30

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc down down down down down down

Command successfully completed Command:

Screen Shot 7 - Viewpoint from ESS #30 after Failover due to Main Cluster Failure Screen shots #6 and #7 show the current control state of the system. Since the status ess port-networks command queries the IPSIs in real time from their priority lists, both ESS cluster should report the same information. They both show that ESS #20 is the controlling cluster for every IPSI and they both show the same priority lists for each IPSI. The viewpoint of ESS #20, since it is the controlling cluster of every port network, is that all port networks are up and their EAL links are described appropriately. ESS #30s viewpoint is different since it is does not have any control links established and reports that from its

73

perspective, all the port networks are out-of-service. Figure 57 shows the control links for every IPSI connected port network after the system recovers from the main cluster outage.

Main

ESS #10

ESS #20

ESS #30

LAN IPSI EPN #4 EI EPN #5

LAN IPSI EPN #6

LAN IPSI EPN #7

IPSI EPN #1 EI

IPSI EPN #2 EI

IPSI EPN #3 EI

CSS PNC

EPN #4 is out-of-service

Figure 57 Communication System In-service after Main Cluster Failure

9.3 Demo Extended Network Fragmentation


The next phase of the demo is the introduction of a severe network fragmentation fault on the WAN link between sites #2 and #3 on top of the already failed main cluster. Once the network fragmentation occurs, as shown in Figure 58, the IPSI in port network #7 detects the connectivity loss to its controlling cluster and starts its no service timer. The IPSIs on the other side of the fragmentation detect the connectivity loss to ESS #30 and make the appropriate changes to their priority lists, but service to them is uninterrupted. Unfortunately, unlike the previous situations, the network fragmentation prevents all ESS servers from being able to contact every IPSI and therefore they are unable to provide the administrator with a complete system view. Executing status ess port-networks from ESS #20 provides a system view of the left side of the fragmentation. As Screen Shot #8 shows, port networks #1, #2, #3, #5, and #6 are in service and being controlled by ESS #20. In addition, the all of these port networks IPSIs have removed ESS #30 as an alternate source of control because of the lack of network connectivity. And finally, ESS #20 does not report any status about PN #7 because it cannot communicate with it.

74

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 20 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State 1A01 actv-aa 2A01 actv-aa 3A01 actv-aa 5A01 actv-aa 6A01 actv-aa active Cntl Connected Clus Clus(ter) ID IDs 20 20 20 20 20 20 20 20 20 20 10 10 10 10 10

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc 1A01 IPSI up 2A01 IPSI up 3A01 IPSI up 5A01 IPSI up 6A01 IPSI up down 1A01 2A01 3A01 5A01 6A01

Command successfully completed Command:

Screen Shot 8 Viewpoint from ESS #20 Immediately after Network Fragmentation On the other side of the fragmentation, executing the status command from ESS #30 gives a much different view of system control. Immediately after the network failure occurred, ESS #30 lost communication with every IPSI except the one residing in port network #7, hence, ESS #30 does not report any control information or current priority lists for those IPSIs. Additionally, port network #7s IPSI detects the connectivity failure to the main server and begins recovery actions by removing the main server from its priority list and starts its no service timer. The screen shot below is a snapshot of ESS #30s viewpoint during the no service time interval.

75

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 30 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State active active active active active active * 30 Cntl Connected Clus Clus(ter) ID IDs

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc down down down down down down

Command successfully completed Command:

Screen Shot 9 - Viewpoint from ESS #30 Immediately after Network Fragmentation Once the no service timer expires, PN #7s IPSI requests service from its best option which happens to be the only ESS cluster it can communicate with (ESS #30). ESS #30, after receiving the service request, takes over PN #7 by establishing a control link and then brings it into service by cold restarting it. Figure 58 below shows the new control link from ESS #30 to PN #7 along with the unaffected control links from ESS #20 to the other port networks.
Main ESS #10 ESS #20 ESS #30

LAN IPSI EPN #4 EI EPN #5

LAN IPSI EPN #6

LAN IPSI EPN #7

IPSI EPN #1 EI

IPSI EPN #2 EI

IPSI EPN #3 EI

CSS PNC

EPN #4 is out-of-service

Figure 58 Communication System In-service after Network Fragmentation Executing the status command from ESS #20 does not reveal any no information that it did before ESS #30 took over PN #7. Since the network is fragmented, no pathway exists for ESS #20 to get status from PN #7s IPSI and report it to the administrator. On the other hand, status ess port-networks executed from ESS #30 shows that it is controlling port network #7 and that port network #7 is in-service.

76

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 30 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State active active active active active 7A01 7A01 actv-aa 30 30 Cntl Connected Clus Clus(ter) ID IDs

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc down down down down down 7A01 IPSI up

Command successfully completed Command:

Screen Shot 10 - Viewpoint from ESS #30 after Failover due to Network Fragmentation

9.4 Demo Network Fragmentation Repaired


At this point, the strategic placement of ESS servers within the system has allowed the system to continue to provide service to all its end-users through both a catastrophic main server failure and an extended network fragmentation. The rest of the demonstration concentrates on restoring the fragmented system back into one autonomous system controlled by the main servers. While the network is fragmented, all the servers are attempting to bring up TCP sockets to all of the IPSIs which they are not communicating with, but the attempts are always unsuccessful since network connectivity is faulted. However, once the network is repaired, all ESS servers will be able to communicate with all IPSIs and all of the socket establishment attempts will be successful. This has two major effects on the system, but does not cause any adverse effects to any end-users. Every ESS cluster is going to be able to provide a complete system view and all of the IPSIs are going to be able to include all of the ESS clusters in their priority failover lists. As Figure 59 shows, ESS #30 continues to provide service to PN #7 even though ESS #10 and ESS #20 have informed the IPSI that they are available. The results of the status commands from both ESS #20 and ESS #30 in Screen Shots #11 and #12 reflect this situation.

77

Main

ESS #10

ESS #20

ESS #30

LAN IPSI EPN #4 EI EPN #5

LAN IPSI EPN #6

LAN IPSI EPN #7

IPSI EPN #1 EI

IPSI EPN #2 EI

IPSI EPN #3 EI

CSS PNC

EPN #4 is out-of-service

Figure 59 Network Fragmentation Fixed

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 20 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State 1A01 actv-aa 2A01 actv-aa 3A01 actv-aa 5A01 actv-aa 6A01 actv-aa active Cntl Connected Clus Clus(ter) ID IDs 20 20 20 20 20 30 20 20 20 20 20 20 10 10 10 10 10 10 30 30 30 30 30 30

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc 1A01 IPSI up 2A01 IPSI up 3A01 IPSI up 5A01 IPSI up 6A01 IPSI up down 1A01 2A01 3A01 5A01 6A01

Command successfully completed Command:

Screen Shot 11 - Viewpoint from ESS #20 after Network is Repaired

78

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 30 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State active active active active active 7A01 7A01 actv-aa Cntl Connected Clus Clus(ter) ID IDs 20 20 20 20 20 30 20 20 20 20 20 20 10 10 10 10 10 10 30 30 30 30 30 30

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc down down down down down 7A01 IPSI up

Command successfully completed Command:

Screen Shot 12 Viewpoint from ESS #30 after Network is Repaired

9.5 Demo Main Cluster is Restored


At this point a number of things can be done. The administrator can leave the system running as is only suffering from the effects of operating in a fragmented mode (see the How are Call Flows Altered when the System is Fragmented section). Another approach the administrator could take is to shift port network #7 under the control of ESS #20 and then operate as one homogenous switch with every IPSI connected port network receiving its control from one source, but the non-IPSI connected port network (PN #7) would still be out of service. This approach can and should be used if the main servers are not going to be operational for an extremely long period of time. This demo takes a another approach to returning the system to normal operation fix the main cluster and after it becomes stable again, transition all of the port networks in a controlled manner back under its control. The transition of the port networks back to the main servers can be achieved by forcing one port network at a time or all of them at once. Once the main cluster is operational, it contacts all of the IPSIs and informs them that it is available to provide service and also gathers the IPSIs current state information. Upon the main servers reconnection, the IPSIs insert the main cluster back into its priority list, but continue receive service from their current master cluster. The following screen shot shows that the main server knows the control status of the IPSI for each port network and is an alternative for each one, but is not in control of any of them.

79

status ess port-networks Cluster ID Com PN Num 1 2 3 4 5 6 7 1 1 1 1 1 1 1 1 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State active active active Cntl Connected Clus Clus(ter) ID IDs 20 20 20 1 1 1 20 20 20 10 10 10 30 30 30

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc down down down down down down down

active active active

20 20 30

1 1 1

20 20 20

10 10 10

30 30 30

Command successfully completed Command:

Screen Shot 13 Viewpoint from Main Cluster after the Main Cluster has been Restored The next screen shot shows ESS #20s viewpoint of the system after the main cluster becomes operational once again. ESS #20 is still in control of port networks #1, #2, #3, #5, and #6 even though it is no longer the best alternative according to the dynamically adjusted priority lists. The viewpoint from ESS #30 (not shown) is in complete agreement with the main clusters and ESS #20s viewpoints and is in control of port network #7.

status ess port-networks Cluster ID Com PN Num 1 2 3 5 6 7 1 1 1 1 1 1 20 ESS PORT NETWORK INFORMATION Pri/ Pri/ Sec Sec Loc State 1A01 actv-aa 2A01 actv-aa 3A01 actv-aa 5A01 actv-aa 6A01 actv-aa active Cntl Connected Clus Clus(ter) ID IDs 20 20 20 20 20 30 1 1 1 1 1 1 20 20 20 20 20 20 10 10 10 10 10 10 30 30 30 30 30 30

Port IPSI Intf Inft Ntwk Gtway Loc Type Ste Loc 1A01 IPSI up 2A01 IPSI up 3A01 IPSI up 5A01 IPSI up 6A01 IPSI up down 1A01 2A01 3A01 5A01 6A01

Command successfully completed Command:

Screen Shot 14 - Viewpoint from ESS #20 after the Main Cluster has been Restored

80

To force all of the IPSI to leave their current controlling cluster to return to the main server, a call disruptive command get forced-takeover ipserver-interface all has been introduced. This command causes the SPE that it was executed on to send special messages to the IPSIs informing shut down their current control links and allowing this new SPE to be its master. After this command is executed, the viewpoints of all the servers in the system will return to their original non-faulted viewpoints as shown a while back in Screen Shots #3 and #4 with control links as shown below.
Main ESS #10 ESS #20 ESS #30

LAN IPSI EPN #4 EI EPN #5

LAN IPSI EPN #6

LAN IPSI EPN #7

IPSI EPN #1 EI

IPSI EPN #2 EI

IPSI EPN #3 EI

CSS PNC

Figure 60 Control Return to Restored Main Cluster

Section 10 - More Thoughts and Frequently Asked Questions


10.1 How is ESS Feature Enabled ?
The license file is extremely important to the ESS feature. In addition to providing a server with a unique identifier (module ID, otherwise known as cluster ID) which is required for proper registration, the license file also includes the necessary values that tell a media server what type it is either the main server or an ESS server (which fundamentally changes its behavior). To this end, there are two primary customer options to check before loading a license file onto a server. FEAT_ESS is the customer option that determines whether or not the ESS feature itself is enabled. Without this option, the main server rejects all ESS cluster registrations and therefore ESS clusters cannot be deployed into the system. The other ESS relevant customer option is FEAT_ESS_SRV. This option informs a given server whether or not it is an ESS server or one of the main servers. This particular value should be set to no for the main server and yes for every ESS server. These values in the license file can be checked by issuing the following commands from the CLI interface: statuslicense v f FEAT_ESS statuslicense v f FEAT_ESS_SRV The matrix below summarizes the customer option settings that are required. Customer Option Main Server ESS Server FEAT_ESS_SRV No Yes FEAT_ESS Yes Yes Table 30. License File Settings License files can be loaded onto a media server at any time. For a number of features, the SPE can activate or deactivate a feature based on the newly uploaded license file without performing a reboot. For example,

81

if a license file is loaded which increases the number of IP endpoints allowed in the system, the SPE will allow more IP phones to register immediately. However, the ESS customer option, FEAT_ESS, does not work that way. If the current license file loaded onto a machine has FEAT_ESS set to no and the new license file has FEAT_ESS set to yes, the main SPE needs to be rebooted in order to turn on the ESS feature after the new license file is loaded. To this end, it is wise to have FEAT_ESS enabled in the initial installation or upgrade even if no ESS servers are currently being deployed. Doing this prevents a system reboot in the future when, and if, the ESS feature is set up in the system.

10.2 What are the Minimum EI Vintages Required for the ESS Feature ?
As mentioned previously, a port network is controlled by an enhanced angel or an arch-angel. The archangel is the master of the port networks TDM bus, scanning the other angels in the carrier for information and feeding that information back to the SPE for processing. If a port network does not have an active arch-angel, then that PN is completely out of service because no information is successfully being exchanged between the SPE and the angels residing on the port network. Various recovery methods have been discussed in the single cluster reliability section which attempt to activate one of the enhanced angels residing on the port networks by bringing up an EAL link. The EAL link can terminate at possibly four different locations in a port network (a-side IPSI, b-side IPSI, a-side EI, or b-side EI). Before the introduction of ESS, there was only one SPE in the system that could instantiate the EAL links and therefore it was ensured that only one EAL would be brought up to a port network at once. If a software fault or race condition occurs that causes two active EAL links to be brought up to the same port network concurrently, then two arch-angels will be activated. This leads to a deadlocked PN and is referred to as dueling arch-angels. Due to the possibility, albeit a very low probability, of software faults and race conditions, hardware level protection was introduced to avoid this scenario of dueling arch-angels. This hardware level protection is justified because nothing can be done remotely from the servers to bring the PN back online. The only way to recover a port network that is suffering from this dueling arch-angel condition is to physically remove one of the boards supporting one of the activated arch-angels. The hardware level protection took the form of an arch-angel token. The concept is that only one entity, and one entity only, has the ability to become the arch-angel if, and only if, it possesses the arch-angel token. This arch-angel token capability does not exist in all vintages of the EI boards as shown in the table below. EI Board TN570 TN2305 TN2306 Vintage A or B CSS C or D A ATM B A ATM B Table 31. Arch-Angel Token Support Type of PNC Arch-angel Token Aware No Yes No Yes No Yes

While dueling arch-angels in a single cluster environment is very rare (as per the reason it is only suggested to have arch-angel token support on all boards rather than required), it is almost guaranteed to occur in ESS deployments. The following figure shows port network #2 failed over to an ESS server. As mentioned previously, if an IPSI loses its connection to its controlling cluster, it begins a no-service timer before requesting service from an alternate source. During this time, the previously controlling cluster attempts traditional recovery methods. If, for whatever reason, the fall-back recovery fails, the IPSI would request service from the alternate ESS server. At this point everything is stable with PN #1 under the control of the main server and PN #2 under the control of the ESS server. However, as far as the main server is concerned, it believes PN #2 is out-of-service and is therefore continuously attempting to restore service to it by re-connecting to the IPSI or establishing an EAL to the EI board (fall-back). If the connection to the IPSI finally succeeds, the IPSI will allow the main server to return to its priority list, but not instantly switch back control (see How ESS Works section). On the other hand, if the failure which was preventing the tunneled EAL from being established was rectified, the main server would attempt to activate the arch-

82

angel on the EI board. Since there is no real-time communication between the main server and the ESS servers, both servers continually attempt to control the port network as shown below. This classic case of dueling arch-angels is avoided by having the IPSI and EI boards token aware. If this was the case, the main SPE would never succeed in bringing up an EAL to the EI board while the ESS server has an established EAL to the IPSI.
Main ESS Main ESS

LAN

LAN

IPSI
EAL #1

IPSI
EAL #2

IPSI
EAL #1

IPSI
EAL #2

EPN #1 EI
EAL #2

EPN #2 EI

EPN #1 EI
EAL #2

EPN #2 EI

PNC

PNC

Fragmented PNC prevents EAL from getting established.

Dueling arch-angels if EI board in PN #2 is not token aware.

Figure 61. Dueling Arch-angels in ESS Environment In summary, arch-angel token aware boards are required in any port network which contains them both EI and IPSI boards. There are three different configurations for a port network no EI boards (PN using an IP PNC), only EI boards (PN using a CSS or ATM PNC with no controlling IPSIs), and both EI boards and IPSIs (PN using a CSS or ATM PNC with controlling IPSIs). Only the final configuration mentioned requires EI boards be token aware.

10.3 What IPSI Vintages Support the ESS Feature ?


IPSI boards were the main new component to the PBX system which allowed the SPE to control port networks via IP as opposed to only through a private CSS or ATM PNC. The original release of the IPSI, referred to as IPSI-1 or IPSI-AP, was able to reside in the tone clock slots supported by either traditional MCC/SCC cabinets or the G600 cabinetry. From the initial release of ACM up to release of the ESS offering, there were changes made to the IPSIs firmware and hardware for both new feature support and bug fixes. In order for the IPSI to be able to also reside in the new G650 cabinet, it needed to be able to supply maintenance board functionality and be able to interface with a new carriers backplane. The new IPSI is referred to as the IPSI-2 or IPSI-BP. The new functionality needed for the ESS feature was first supported in firmware release 20 (anything previous to this release is called pre-ESS firmware) and could run on either IPSI-AP boards or IPSI-BP boards. In other words, the ESS feature does not dictate, nor require, the need for IPSI hardware versions (IPSI-AP or IPSI-BP), only the carrier type does. The table below shows which IPSI versions are compatible with which software versions.

83

Pre-ESS IPSI Firmware ESS IPSI Firmware

Pre-ESS Software Supported Supported

Main Server Supported Supported

ESS Servers NOT SUPPORTED Supported

Table 32 IPSI Firmware Compatibility Table

It should be concluded that new firmware will be backwards compatible with pre-ESS release software and also completely supported by all types of servers. In addition, the new software on main servers is backwards compatible with pre-ESS firmware, but ESS servers cannot interface with it.

10.4 What is the Effect of Non-Controlling IPSIs on ESS ?


Throughout this document, IPSIs have only been discussed as an IP control interface into the port networks for the SPE. In addition, the IPSI also was described as a conglomeration of different components a tone clock, a PKTINT, an arch-angel, and (for IPSI-BP boards only) a cabinet maintenance board. On a typical upgrade the IPSI boards replace the tone clock board and replicate its functionality. In fact, if an IPSI was inserted into a port network supported by a G3r SPE in place of a tone clock, it would report itself as a tone clock and the G3r SPE would be none the wiser. When the G650 cabinet was released it needed to have a maintenance board and a tone clock, but only one slot was available for it. Therefore a new board could have been released to support these features or the IPSI could be used instead. For an IP connected PN, the decision was simple because an IPSI was needed for control anyway. However, if the PN was interconnected via a CSS or ATM PNC an IPSI is not necessarily required for control (it could be tunneled through the EI boards). Therefore the concept of a non-controlling IPSI was created which essentially says, The port network needs a tone clock and a maintenance board which the IPSI can provide, but do not control the PN through it. The reason a system might be configured like this is to avoid having to run an IP network from the SPE servers to the PN itself. However, the ramification of this is that the system treats non-controlling IPSIs as purely tone clock boards which prevents any server (main or ESS) from connecting to it in order to control it. As discussed previously, the IPSI creates a failover priority list based on the ESS servers that connect to it. Since a noncontrolling IPSI never receives any connection, it does not have the ability to failover to ESS servers. An IPSI cannot be administered as a non-controlling IPSI with respect to the main server and a controlling IPSI with respect to an ESS server since the translation sets the servers are running are identical.

84

Section 11 - Index
Section 1 - Introduction 1.1 What is ESS ? 1.2 How is this paper organized ? Section 2 - Background 2.1 What is an Avaya Media Server ? 2.2 What are Port Networks ? 2.3 How are Port Networks Interconnected ? 2.4 How are Port Networks Controlled ? 2.5 What are SPE Restarts ? 2.6 What are EPN restarts ? Section 3 - Reliability Single Cluster Environments 3.1 What is a Cluster ? 3.2 How do single cluster environments achieve high availability ? 3.3 What is a Server Interchange ? 3.4 What is Server Arbitration ? 3.5 What is an IPSI Interchange ? 3.6 What is EI Control Fallback ? Section 4 - Reliability Multiple Cluster Environments 4.1 How long does it take to be operational ? 4.2 What are Survivable Remote Processors (SRP) ? 4.3 What are ATM WAN Spare Pairs (ATM-WSP) ? 4.4 What are Manual Backup Servers (MBS) ? Section 5 - ESS Overview 5.1 What are Enterprise Survivable Servers (ESS) ? 5.2 What is ESS designed to protect against ? 5.3 What is ESS NOT designed to protect against ? Section 6 - How does ESS Work ? 6.1 What is ESS Registration ? 6.2 How are ESS Translations Updated ? 6.3 How do Media Servers and IPSIs Communicate ? 6.4 What is a Priority Score ? 6.5 How do IPSIs Manage Priority Lists ? 6.6 How are Communication Faults Detected ? 6.7 Under What Conditions do IPSIs Request Service from ESS Clusters ? 6.8 How Does Overriding IPSI Service Requests Work ? Section 7 - ESS in Control 7.1 What Happens to Non-IP Phone Calls During Failovers ? 7.2 What Happens to IP Phone Calls During Failovers ? 7.3 What Happens to H.248 Gateway Calls During Failovers ? 7.4 How are Call Flows altered when the System is Fragmented ? 7.5 What Happens to Centralized Resources when a System is Fragmented ? 7.6 What Happens to Voicemail Access when a System is Fragmented ?

85

Section 8 - ESS Variants Based on System Configuration 8.1 Basic IP PNC Configured System 8.2 Catastrophic Server Failure in an IP PNC Environment 8.3 Network Fragmentation Failure in an IP PNC Environment 8.4 Basic ATM PNC Configured System 8.5 Catastrophic Server Failure in an ATM PNC Environment 8.6 Network Fragmentation Failure in an ATM PNC Environment 8.7 Basic CSS PNC Configured System 8.8 Catastrophic Server Failure in a CSS PNC Environment 8.9 Network Fragmentation Failure in a CSS PNC Environment 8.10 Mixed PNC Configured System Section 9 - ESS in Action - A Virtual Demo 9.1 Demo Setup 9.2 Demo Catastrophic Main Server Failure 9.3 Demo Extended Network Fragmentation 9.4 Demo Network Fragmentation Repaired 9.5 Demo Main Cluster is Restored Section 10 - More Thoughts and Frequently Asked Questions 10.1 How is ESS Feature Enabled ? 10.2 What are the Minimum EI Vintages Required for the ESS Feature ? 10.3 What IPSI Vintages Support the ESS Feature ? 10.4 What is the Effect of Non-Controlling IPSIs on ESS ?

86