Business Continuity and

Disaster Recovery

Trends, Considerations,
& Lessons Learned

April 2014

Michael Porier
Protiviti – Managing Director
 Agenda

Prevalence of Disasters – Outages Happen

Business Continuity Management – An Overview

Trends, Considerations & Lessons Learned

Resources & Services

2 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Impact of Natural Disasters
Philippines Typhoon - Nov 2013
• Strongest typhoon ever recorded at landfall
• Sustained winds of 195 mph with gusts up to 235
• Estimated 10,000 people killed
• 600,000 people displaced
• 9.5 million people affected

3 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Impact of Natural Disasters
Hurricane Sandy - 2012

• 5th worst hurricane in the history of United States

and unofficially terming it “Superstorm Sandy”
• $20 billion in damages and loss and only half
insured
• 2.2 million customers without power
• 111 structures damaged due to fire
• Around 100,000 homes and office building
structures were damaged
• All modes of transportation affected: rental cars,
trains, planes, subways, etc.
• Financial markets closed in advance of storm
(Source :http://www.reuters.com/article/2012/10/30/us-storm-sandy-impact-idUSBRE89T1IV20121030)

4 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Impact of Technological Disasters (since Jan 2013)
Sl. No Date Headlines Location Hours Reason for Outage Affected by
1 January 25 Government website down New Jersey, USA 6 hours Power outage at Datacenter 800,000 residents
Major Canadian banks,
2 January 28 VISA datacenter power outage Canada 20 hours Power outage at Datacenter including CIBC, Royal Bank of
Canada and TD Canada Trust.
Newly installed electrical relay
3 February 4 Super Bowl Power Loss New Orleans, USA 35 min 108 million viewer
device
4 February 25 Windows Azure Cloud Crashed USA 24 hours Expired SSL Certificate 250,000 customer
5 March 14 Hotmail, Outlook and Skydrive outage USA 16 hours Servers over heating 360 million customers
6 March 20 DreamHost power outage California, USA 48 hours Power system failure 350,000 customers
Inability to access to its
7 April 16 American Airlines Flights Ground Texas, USA 8 hours 2038 flights cancelled
reservations system, Sabre.
4 months Fire damaged the Datacenter
8 April 19 Michigan County IT service offline Michigan, USA 850,000 residents
or more building
9 May 7 Internet Cut off Syria War 22 million residents
Toronto’s leading telecom hub goes Cooling system failure due to
10 July 9 Toronto, Canada ~ 30 min 300,000 users
off flooding
Storage Failure slows
11 July 16 Oregon Jobless Checks delayed Oregon, USA 12 hours 105,000 users
performance
Major Outage for BlueHost,
12 August 2 Utah, USA 20 hours Networking outage 5 million users
HostGator, HostMonster
Outages for New York Times,
13 August 14 California, USA ISS outage 500 million customer
Microsoft Cloud Services
Amazon Recovers from Brief Outages Increased error rates for APIs for
14 August 19 Oregon, USA 45 minutes 30 million customers
for Retail Site, Some AWS Services its Elastic Compute Cloud
School Districts Hit by Data Center Fire suppression system
15 September 11 Oregon, USA 6 hours
Failures damaging hard drives & servers
State Government Slowed by Data
16 September 12 New Jersey, USA 4 hours Power outage at Datacenter 100,000 residents
Center Outage
Facebook Not Working, So Twitter Fills
17 October 21 California, USA 2.5 hours Application performance error 1.15 billion users
With Facebook Humor
Terremark Data Center Outage Knocks
18 October 27 Florida, USA 6 hours Networking issue USA residents
HealthCare.gov Offline

5 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Outages in the last several months…
When would you like to schedule your disaster…?

Earthquake in Pakistan
Hurricanes Manuel and
Typhoon Haiyan
Ingrid – Mexico
– Philippines
Colorado, U.S., floods
(September)
Earthquake in China

July August September October November December

Earthquake in Philippines New England

Earthquake in the Aleutian Tornado
Islands of Alaska Typhoon Phailin – India
Mexico Volcano
California Rim Fire Earthquake – Central
Visayas, Philippines China Flood
Wildfires – Portugal
Southern Asia floods West Africa drought
(October) (ongoing)

Polar Vortex - 2014

Business weather forecaster Planalytics estimates that the U.S. economy took a roughly $5 billion hit because of the frigid weather
stemming from lost productivity, higher heating bills and a drop in consumer spending. The cost of flight cancellations is expected to
reach $1.4 billion. Those losses reflect the enormous area affected by the polar vortex.
Source: http://www.worldvision.org/news-stories-videos/2013-top-natural-disasters, http://www.cbsnews.com/videos/cost-of-flight-cancellations-14b/

6 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Most Expensive Natural Disasters In 2013
Global Billion-Dollar Economic Loss Events by Region

Flooding In Central An Earthquake In Super Typhoon Typhoon Fitow In

Europe Cost $22 Lushan, China Cost Haiyan Cost $13 China And Japan
billion (May) $14 billion (April) billion Cost $10 billion
• $22 billion in • Economic loss was • $13 billion in • $10 billion in
economic losses $14 billion economic losses economic losses
• $5.3 billion in • $250 million in • $1.5 billion in • $1 billion in insured
insured losses insured losses insured losses losses

http://www.ibtimes.com/report-ten-most-expensive-natural-disasters-2013-1540058

7 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
Business Continuity Management

- An Overview
 A Standard Approach?
A Google search on ‘Business Continuity Management’ returns 22,000,000 results

9 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 A Standard Approach?
Another standard approach would be regulation-based and relying on the response to the
different regulations for addressing the issue of business continuity

ISO 22301 – Business continuity

management systems ISO ISO
CobiT FFIEC FRB HIPAA FERC DRII
22301 27031
ISO 27031 – Guidelines for information
Institute a process that
and communication technology
includes crisis
readiness for business continuity
management, business
resumption planning P P P P P
CobiT v5 – Control Objectives for Info &
and IT disaster
Tech
recovery
FFIEC – Federal Financial Institutions Assess current
Examination Council mitigating controls P P P P P
FRB – Federal Reserve Board Review service level
agreements between
the organization and its P P P P P P P
HIPAA – Health Ins. Portability and Acct.
Act external partners
Define standard
FERC – Federal Energy Regulatory methods for
Comm. documenting response,
recovery and P P P P P P
DRII – Disaster Recovery Inst. restoration procedures,
International communication plans.

FEMA – Federal Emergency Mgmt Utilize numerous types

Assoc. of testing approaches P P P P P
NIST – Nat’l Inst. of Standards & Tech Audit the BCM process
on a periodic basis
P P P P P P P P
NFPA – Nat’l Fire Protection Agency

10 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Business Continuity is…
…the development of strategies, plans and actions which provide protection or alternative modes
of operation for those activities or business processes which, if they were to be interrupted, might
otherwise bring about a seriously damaging or potentially fatal loss to the enterprise.

1 Crisis Management Plan

The crisis management plan provides the key
communication mechanisms necessary to
ensure employee safety, provide initial
information and direction, and organize ongoing
actions.

2 Disaster Recovery Plan (IT)

The disaster recovery plan typically refers to
the plans in place to restore essential
Information Technology (IT) systems and
applications that enable critical business
processes.

3 Business Resumption Plan

The business resumption plans are specific to
each critical business function and articulate
the specific steps necessary to enable the
respective process (e.g. payroll).

11 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Business Continuity Methodology

Determine what
Derive recovery
systems and Train
options (price Implement and
business personnel; test
vs. document
processes need and maintain
risk) and solutions.
to be recovered the plans.
identify gaps.
and why.

Executive Support and Ownership; Tactical Ownership; Governance Structure

BCM Crisis Implement Crisis

Program Management Crisis Mgmt Mgmt
Governance Strategy Plan Training/Testing

BCM Risk Business Implement Business BCM

Diagnostic Assessment Recovery Business Recovery Quality
Strategy Rec. Plan Training / Testing Assurance

Business IT Disaster Implement IT Disaster

Impact Recovery IT Disaster Recovery
Analysis Strategy Rec. Plan Training / Testing

12 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Business Impact Analysis (BIA)
Supporting Critical Business Existing
• Spending for business Technologies Processes Recovery Plans

continuity planning is finite;

therefore plans need to focus
on recovery mechanisms for the
high risk elements of the
organization BIA

• The business impact analysis

is the single most important Information Necessary to Develop
component as it provides the Risk-Based Recovery Solutions

guidance, metrics, and purpose

for the ongoing development of • Recovery time objectives by
process
the business continuity plan
• Financial impact of an outage

• customer Impact of an outage

• Prioritization of recovery steps

13 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Recovery Options Analysis
Once the impact of potential outages has been determined, recovery solutions must be evaluated and
selected. Numerous options may exist, ranging from obtaining an external recovery (or “hotsite”) provider,
utilizing the resources of an existing owned data center, or modifying the current equipment to ensure
necessary redundancy.

The ultimate solution should be based on the potential risk of an outage, the level of
risk that management is willing to accept, and the cost constraints faced by the
organization.
A good strategy development approach Business Continuity Plan
includes logical options for each business
function/location, together with the
pros/cons of each and their
Business RISK GAP IT Recovery
implementation implications. Requirement Ability
Management can then weigh the cost of
the recovery strategy (both
implementation and maintenance costs)
against the potential cost of the business
interruption.

Recovery Options
Analysis

14 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Strategies for Data Recovery

• Recovery time objective (RTO) drives selection of alternative

strategies that enable data restoration anywhere from point
of failure (e.g. synchronous mirroring) to multiple days (e.g. Mirroring
$$$
traditional “tape” backup) Synchronous
Mirroring

• Implementation cost for data recovery strategies will

increase as data loss exposure is reduced. As data Semi-

Cost of Solution
loss exposure is minimized, recovery time may be Synchronous
Mirroring
reduced
Stand-By
Database

Electronic
Journaling
Electronic
Traditional Vaulting
Backup

Chronological Time of Data Recovery

Days Hours Minutes Zero

15 © 2014 Protiviti Inc.
CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Plan Development & Documentation

• Once an organization understands their strategic business continuity direction,

the need to organize and document those solutions into a “living” plan becomes
critical
• This helps to ensure that the plan can be maintained as the organization
changes through time
• The end result should be business continuity documentation that has been
properly distributed and that contains the necessary detail to quickly recover
from an interruption
BCM Documentation Components
• Also, during this phase,
emergency procedures,
recovery teams, and incident
plans will need to be
assembled and documented

16 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 BCP Trends, Considerations & Lessons Learned

A) Ownership of BCP
B) “Right-sized” level of documentation
C) Plan Integration
D) Technology Considerations
E) Vendor Management

17 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 A) Ownership of BCP

Trends and Considerations

• Define an owner; Lack of an owner results in

outdated and uncoordinated plans
• Visibility and knowledge of the entire business –
approached and managed from a business risk
perspective
• Too often, BCP lies within the IT department
• Owned by someone with responsibilities for
business operations with direct alignment to
enterprise risk management

18 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 A) Ownership of BCP

• Not an “IT only” venture. Distinct teams are needed for

system recovery and business resumption:
– The recovery of the network or telecommunications
requires:
o Key members of the IT department
o Any vendors servicing the company
– To resume operation of processes requires:
o Thoroughly examined business processes
o Effective prioritization of what needs to be restored
first and which customers need service first

19 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 A) Ownership of BCP

– Communication is critical to effective disaster response

o Review the communication plan
o Determine how employees will be provided with
information when land lines and cell towers are
down or overloaded
o Include an employee call-in line to a location remote
from the disaster and test the operation of the call-in
process
o Pre-determine employee meeting points
o Define how you will communicate with stakeholders
when all communication systems are down
– A good plan includes frequent and scheduled status
updates to the media and employees
o Designated media spokespeople should be
adequately trained

Source: www.disastersrus.org/katrina/ACP_Hurricane_Katrina_Observations.pdf

20 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 B) “Right-sized” Level of Documentation

Trends & Considerations

• Determine the level of knowledge you want to

assume is in place when you document your
business recovery procedures
• For more technical processes, documentation
should enable the activity to be executed by
resources not normally engaged in the activity
• Action-oriented and flexible
• Document decision “triggers” for executive
personnel
• Provide for viable manual operation of critical
functions

21 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 B) “Right-sized” Level of Documentation

• Avoid delay in critical decisions with a defined

decision hierarchy
– A well defined command and control structure is
essential to effective decision making
– Define command and control and give them the
authority to act
– Emphasize that businesses, families, and individuals
must be decisive and personally responsible when
responding to emergencies

Source: www.disastersrus.org/katrina/ACP_Hurricane_Katrina_Observations.pdf

22 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 B) “Right-sized” Level of Documentation

• Create an iterative process for plan maintenance.

– When writing and revising plans, involve subject
matter experts from each area to ask the relevant
questions:
o What resources are needed?
o Who initiates and approves resources?
– Emphasize that business continuity must be an
organization-wide effort
o Come together on a periodic basis to ensure
the business continuity plan is up to date and
accurately reflects the risk profile of the
organization
– Develop a “right-sized” plan - Less emphasis on the
volume of documentation and more emphasis on
training employees

Source: www.disastersrus.org/katrina/ACP_Hurricane_Katrina_Observations.pdf

23 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 C) Plan Integration

Trends & Considerations

• Includes Crisis Management, Business

Resumption, and IT Disaster Recovery
• Integration with ESH processes
• Requires linkage and alignment between
components of the plan through a comprehensive
and consistent definition of “recovery strategy”
• Consistency in format and approach facilitates
plan integration and execution

24 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 D) Technology Considerations

Trends & Considerations

• Changes in technology architecture should be

incorporated into the recovery strategy and
subsequent recovery plans
− Cloud architecture may enable alternative IT
environments that streamline recovery plans
and reduce reliance on traditional recovery site
models.
• “XaaS” models need to be understood and
defined in the business analysis to ensure
recovery plans are viable
• Utilize cloud services as appropriate to house and
communicate recovery plans
• Internal and external communication can be
facilitated through “social media” channels

25 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 E) Vendor Management

Trends & Considerations

• Define which vendors are critical to business

operation during the impact analysis and risk
assessment
• Implement a vendor management program to:
− Document external party roles, activities and
related controls
− Align recovery objectives
− Validate recovery capabilities
− Check the service delivery specification in
vendor contracts

26 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
Resources & Services
 Business Continuity Solutions
Business Continuity Management – Incorporates best practices & adheres to latest requirements

The analysis and evaluation of strategies, development of approaches, testing and implementing plans
that meet continuity needs from a people, processes, and IT infrastructure perspective. Our approach
incorporates industry best practices and is constantly monitored to insure adherence to regulatory
requirements.

Crisis Management &

Business Process Recovery IT Disaster Recovery
Communications
• Focus on the evaluation of • A series of actions taken to • Links critical business
people, processes, gain control of the event processes to the IT
technology, and data that quickly to minimize the resources needed to support
are vital to an operation to affects of a disruption, them (whether internal to the
mitigate business, prepare for and oversee organization or outsourced)
environmental, man-made recovery, and manage
and technology risks communications throughout
inherent to business by the event
assisting in the design and
implementation of proven
strategies

28 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Protiviti’s Updated Business Continuity FAQ

Updates Include:

• Regulatory
Requirements
• Industry
Considerations
• Lessons Learned
• Social Media
• New Trends &
Practices

Download available at:

www.protiviti.com/bcm

29 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
 Questions and Answers

Michael Porier
Managing Director, Houston, Texas

Phone: +1 713-314-5030
Email: michael.porier@protiviti.com

Powerful Insights. Proven Delivery.®

Michael Porier is a Managing Director in Protiviti's Houston IT Consulting practice, specializing in

assessing and implementing Business Continuity solutions for clients in the Energy Industry. His
expertise includes performing Business Impact Assessments to assist organization in determining their
critical business processes and working with IT to evaluate potential solutions. Michael is a national
leader for Protiviti’s BCP solutions and has presented at numerous conferences, published various
articles, and is a frequent spokesperson on this topic to industry journals and publications.

30 © 2014 Protiviti Inc.

CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.
31 © 2014 Protiviti Inc.
CONFIDENTIAL: This document is for your company's internal use only and may not be copied nor distributed to another third party.