Disaster Management (i.e.

Business Continuity)
Josef C. Mueller Associate Partner

Objective

To discuss information systems disaster management and the formation of backup and disaster recovery plans

Agenda
Introduction Disaster Recovery Approach DR Team Organization Case Study Example Disaster Recovery Services Open discussion

Introduction
What is a Disaster?
Any unplanned event that requires immediate redeployment of limited resources Sample Disasters
Natural Forces Fire Environmental Hazards Flood / Water Damage Extreme Weather Technical Failure Power Outage Equipment Failure Network Failure Software Failure Human Interference Criminal Act Human Error Loss of Users Explosions

Introduction
Some Examples of Disasters
The Chicago Flood
The underground flood of Chicago on Monday April 13, 1992 proved to be one of the worst business disasters ever. 230 buildings lost power because water threatened their underground power sources.

The World Trade Center Explosion

Businesses were forced to evacuate the World Trade Center in February 26, 1993. When a bomb exploded in the underground parking garage. Companies that were effected by the disruption were unable to remove critical equipment and documents.

The San Francisco Earthquake

The Oct 18, 1989 quake measured 7.0 on the Richter Scale. The Bay bridge had collapsed. The city had lost the main business section due to the collapse of buildings and electricity.

Introduction
Some Examples of Disasters (Contd)
Hurricane Andrew
August 22, 1992, Hurricane Andrew hit the South Florida area. Many businesses suffered physical and financial losses from the hurricane, the valuation of destroyed property was the largest in US history.

The Kobe Quake

The devastation on January 17, 1995 was the worst in the port city of Kobe where the 7.2 magnitude quake toppled roadways, wrecked docks, severed communication lines and kept the city in flames into the next day.

Oklahoma City Bombing

On April 19, 1995, a terrorist bomb exploded in front of the nine-story Alfred P. Murrah Federal Building in downtown Oklahoma City. The blast destroyed onethird of the building from roof to ground, leaving a crater eight feet deep, and 30 feet wide.

Introduction
What is a Disaster Recovery Plan?
A management document for how and when to utilize resources needed to maintain selected functions when disrupted by agreed upon incidents
Other names commonly used: Business Continuity Plan Contingency Plans Continuity Plans Emergency Response Plans Business Recovery Plans Recovery Plans

Introduction
When an incident occurs, the Disaster Recovery response activities are likely to be the following (at a high level).
Incident Assess Damage Confirm Response Strategy Transfer to Alternate Location Execute Required Functions

Prepare New Site

Transfer & Execute at New Site

Restore Primary Site Return to Normal Operations

Transfer & Execute at Primary Site

Generate Change Requests

Assess DRP Effectiveness

Introduction
What is the magnitude of an incident?
Regional Area Local Area Within 3 Blocks To The Building Within 3 Floors On The Floor Within The Room

Depending upon the magnitude of an incident, possible alternative sites include:

Within The Room Within the Building Within the Region Outside the Region

Introduction
Types of Controls
Integrity Controls Policy Methodology Staffing Education Division of Responsibility Audit Error and Change Control Reporting and Resolution Test Quality Assurance Confidentiality Controls Proprietary Information Policy Ethics Statement Need to Know, Need to Withhold Classification Scheme Records Management Handling Procedures Physical & Electronic Security Measures Availability Controls Asset Identification Interruption Analysis Controls Review Impact Analysis Data Backup Off-site Storage Avoidance Strategies Mitigation Strategies Early Detection & Notification Recovery Strategies Alternate Locations Plans and Procedures Vendor Relationships Training Testing

Introduction
Types of Strategies
Avoidance Strategy Redundant configuration to avoid incidents Site harden facilities to resist incidents Redundant utilities and hardware Automated operation recovery plan Recovery Strategy Mitigation Strategy High level recovery plan Early warning detection Contractual agreements Off-site data storage Very responsive vendor with vendors Mirrored data and relationships Very knowledgeable documents Detailed migration employees recovery plan Types of Strategy Options Hot site Cold site Self Backup Service Bureau Reciprocal Agreement

Introduction
What is a Critical Business Function?
A specific entity management has decided is so significant to the business mission, that without it, the organization cannot successfully operate after an identified time period.

Types of Impact
Financial Loss Lost Revenue Lost Sales Lost Market Share Lost Opportunity Human Interference Management Control Employee Relations Stockholder Relations Public Image Legal Exposure Contractual Liability Competitive Advantage

Extra Expense Labor Cost Recreate Lost Business Recreate Lost Data Use Manual Process Equipment Cost Hardware / software Telephones Money Cost Delayed Receivable Delayed Orders New Interest New Investments

Introduction
Criteria for a Critical Business Function
Cost of Control vs. Impact
Cost of Impact $ Timing Requirements Minutes Hours Days Weeks Quarters Special Situations

Cost of Control $

Impact
Interdependencies Inputs and Outputs

Cost

Introduction
Implementing Recovery Plans is not an easy task!
Recovery prevention techniques are inadequate Increase the level of user security awareness and education No recovery plan at all Plan is stored on the ultimate computer (in IT directors head) Establish short-term alternate processing procedures Removal of systems running on obsolete machines Recovery plans are too theoretical and not geared to the organizations needs Plans are unwieldy Recovery plans are in a written format and/or are not updated Backup not tested Plans not tested Plans are located in the computer room or the building Plans are too grandiose (EXPENSIVE) Plan does not address PCs / workstations People Factors are not taken into account

Disaster Recovery Approach

The following Life Cycle model is useful when thinking about Disaster Recovery.
Planning Activities Normal Operations Maintenance Activities
Changes from tests Up-to-Date DRP

Changes

Changes from event

Recovery Activities

Disaster Recovery Approach

Planning Implementation Recovery Disaster Strategy Recovery Development Plan Training & Testing

Scoping & Risk Assessment

Approval

Planning The primary objective for the Planning Phase is to gain management consensus on the focus areas and scope of a Disaster Recovery Plan that will address major business risks Implementation The primary objective for the Implementation Phase is to develop, test, and rollout a Disaster Recovery plan. The implementation phase could be longer or shorter, depending upon scope, approach, and staffing defined during the Scoping and Risk Assessment phase

Disaster Recovery Approach

Determine the focus areas and scope for the Disaster Recovery Plan implementation phase
Activities Management Briefing Questionnaires Interviews Focus Groups Workshop
Scoping & Risk Assessment Recovery Disaster Training Strategy Recovery & Development Plan Testing Approval

Key Deliverables Scoping and Risk Assessment Report Requirements Summary Current Capability Summary Critical Business Functions Matrix Critical Systems Matrix

Disaster Recovery Approach

Develop strategies for each of the most critical systems based upon the outcome of the Scoping and Risk Assessment phase
Activities Develop Strategies Select Spinoff Projects
Scoping & Risk Assessment Recovery Disaster Training Strategy Recovery & Development Plan Testing Approval

Key Deliverables The Recovery Strategy Report Alternatives and recommendations

Disaster Recovery Approach

Develop detailed plans for business continuity based upon the specific strategy identified for each critical system
Activities Develop Recovery Plan Key Deliverable Recovery plan includes Assessment Plan & Procedures Notification Procedure Recovery center Procedure Migration Plan (facilities, data, people) Team Organization ( Roles & Responsibilities)
Scoping & Risk Assessment Recovery Disaster Training Strategy Recovery & Development Plan Testing Approval

Disaster Recovery Approach

Develop detailed plans for business continuity based upon the specific strategy identified for each critical system (continue)
Activities Develop Maintenance Procedures Key Deliverable Maintenance Procedures include Responsibility matrix for maintenance Testing strategy How to update the Recovery Procedure Ongoing Center recovery training schedule Recovery Center Location, facilities and required component
Scoping & Risk Assessment Recovery Disaster Training Strategy Recovery & Development Plan Testing Approval

Prepare facilities and Infrastructure

Disaster Recovery Approach

Provide training to the recovery team and conduct the testing based upon the testing approach documented in the Maintenance procedure
Activities Prepare training materials Conduct & Evaluate Training
Scoping & Risk Assessment Recovery Disaster Training Strategy Recovery & Development Plan Testing Approval

Key Deliverables Training material Trained staff

Disaster Recovery Approach

Get the Disaster Recovery Plan approved and rollout to the organization
Activities Revise plan (if necessary) Approve the Disaster Recovery Plan Key Deliverable Management Sign-off Publication & Distribution of the disaster recovery
Scoping & Risk Assessment Recovery Disaster Training Strategy Recovery & Development Plan Testing Approval

DR Team Organization
An Example of Disaster Recovery Team
DRP Management Team

Disaster Recovery Director Production Application Support Disaster Recovery Coordinator

Administrative Support

Customer Liaison

Site Restoration

System Software and Database Administration Computer Operation and Off-site Storage Application Support

Security

Network Delivery Services Delivery

DR Team Organization
Examples of Data Center Roles & Responsibilities
Title
DR management Team

Roles

Responsibilities

Act as the steering committee Provide overall management support to DR team of the DR Team Responsible for strategic decision and key requirements or changes on DRP Make key decisions according to DRP Act as an advisor to the DR management team. Oversee the activities of the DR team Budget for future DR requirements Communicate with other management to deal with the business process and recovery procedures Provide the DR team with administrative resources and facilities Co-ordinate with lawyers for court cases and handle legal documents Responsible for accounting matters on DRs expenses Investigate the amount of damaged resources and insurance claims

Disaster Recovery Director

Administrative Support

Provide administration support to the DR team

DR Team Organization
Examples of Data Center Roles & Responsibilities
Title
Customer Liaison

Roles
Coordinate and coordinate with users and customers on any recovery issue

Responsibilities
Notify users and clients of the disaster Issue updates of recovery progress and expected time of recovery Help on data center migration issues and work re-allocation Declare a disaster for each critical system component or for an entire site Inform the DR team of the decision Execute DR procedures and recovery strategies Ensure that the DRP is updated and test on a regular basis Organize security control for the disaster site and alternate processing site as required Responsible for the restoration of Hosts, Servers, DB, synchronize data, etc.

Disaster Recovery Coordinator

Centralized coordination for the entire DR team

Site Restoration

Co-ordinate the recovery operations should a site be destroyed Prepare recovery and restoration of software and databases

System Software and Database Administration

DR Team Organization
Examples of Data Center Roles & Responsibilities
Title
Computer Operations and off site storage Application Support Security

Roles
Manage storage of the backups Manage applications with regard to DRP Review and monitor DR procedures Manage and monitor voice and data network

Responsibilities
Provide ready access to the required backups Ensure the backups are stored in a secure environment Manage application changes to ensure they are compliant with the DRP and vice versa Ensure the DR procedures comply with the firm security and audit policies Oversee the recovery of the communication environment Switch users to use the alternate network Co-ordinate with the communication service providers for WAN service recovery

Network Delivery

DR Team Organization
Examples of Data Center Roles & Responsibilities
Title
Service Delivery

Roles
Manage IT service delivery

Responsibilities
Oversee the service management recovery Provide helpdesk and end-user support as in DRP Work closely with Customer Liaison and Disaster Recovery Coordinator to ensure synchronization of communication channel to the users and the DR team activities.

Case Study
The Chicago Flood : Impact
One of the worst business disasters 230 buildings lost power for a couple of days Valuable government records were in jeopardy Extensive impact on electrical and computing systems The greatest financial impact on the CBOT, losing 25 billion in trading of 36 products

Case Study
The Chicago Flood : Disaster Recovery
Using Alternate Site Services approach Providing the alternate site nearly identical to the customers damaged site Implemented by Comdisco Continuity Service

The Chicago Flood : Recovery Result

Helped 2 Chicago banks resume operation within hours of evacuation 17 customers from the financial, brokerage, government and service/ distribution industries, were supported at their hot sites within half a day

Case Study
The World Trade Center Explosion : Impact
Building-wide power outage Structural damaged and employee trauma, Businesses were down Water problem due to pipes were severed Injured and Dead reports, the building was considered a crime scene

The World Trade Center Explosion : Recovery

Fiduciary Trust, a banking and financial institutes Recovery Plan The data center switched automatically to their secondary power system Moved the operation to their alternate site in NJ which equipped with a computer network nearly identical to that of the bank

Case Study
The World Trade Center Explosion : Recovery Result
System was down for Friday afternoon and was up and running by Monday morning as if nothing had happened Employees retained their usual telephone numbers Transactions went through the same as always Customers couldnt even detect that the bank was no longer operating from the World Trade Center

Example Disaster Recovery Services

Examples of Disaster Recovery Services
Alternate Sites Provide alternate site nearly identical to the customers damaged site Business Impact Analysis Provide services such as defining disaster plans and addressing exposures to business and recovery administrators Certification Provide services such as certifying qualified individuals in the discipline and promoting the credibility and professionalism of certified individuals

Example Disaster Recovery Services

Examples of Disaster Recovery Services
Education Classes Creating a base of common knowledge for the business continuity/disaster recovery planning industry through education, assistance, and the promotion of international standards On-Site Recovery Facilities Manage the mobilization of an on-call response team, prepare predesignated site, erect temporary pre-engineered structures, install mechanical and electrical systems and coordinate move-in activities Satellite Communication Provide satellite telecommunications products and services

Example Disaster Recovery Services

Service Providers : Consulting Services
Andersen Consulting www.ac.com Bell Atlantic Federal CommGuard www.commguard.com Comdisco www.comdisco.com Computer Security Consultants, Inc. www.crciweb.com GSA Disaster and Business Recovery www.gsa-gsa.com Intessera Technologies Group www. intessera.com

Example Disaster Recovery Services

Service Providers : Alternate Site Services
ARC Disaster Recovery Services www.arcdrs.com Comdisco www.comdisco.com HP Business Recovery Services www.hp.com IBM Business Recovery Services www.brs.ibm.com SunGard Recovery Services, Inc. recovery.sungard.com

Example Disaster Recovery Services

Providers : Computer Quick-ship , Hardware Replacement
El Camino www.elcamino.com

Disaster Management (i.e., Business Continuity)

Open discussion Q&A