ISACA

December 13th 2007

Auditing the Disaster Recovery Plan

What should be in a plan,

and what should not

By:
Jeffrey Blackmon CBCP, CISSP

1 ISACA 2007, Jeffrey Blackmon

 Quick Intro:
 Jeff Blackmon, CBCP, CISSP

 Started BC/DR planning in mid 80s

 Financial
 Petroleum

 Foreign Military

 Pharmaceutical

 L3 Communications, Titan Group

 Support of Federal Government Contracts
(Kansas City and DC)

2 ISACA 2007, Jeffrey Blackmon

 Format:
A little free format style

 Open Discussion

 Ask Questions

3 ISACA 2007, Jeffrey Blackmon

This may be somewhat a little different
from the regular presentations

Usually have auditors speaking to

auditors

Usually have computer people speaking

to computer people

But not in this case

4 ISACA 2007, Jeffrey Blackmon
Computer person / business person
speaking to the auditors

So expect a little different perspective

5 ISACA 2007, Jeffrey Blackmon

 Computer Staff

6 ISACA 2007, Jeffrey Blackmon

 The Auditors

7 ISACA 2007, Jeffrey Blackmon

Reason for some of the past relationships between
Auditors and the Computer people

8 ISACA 2007, Jeffrey Blackmon

 Why is BC and DR so difficult?

 May not be well defined

 Big project
 Expensive
 Very difficult to take that 1st step

9 ISACA 2007, Jeffrey Blackmon

 Topics
1. Goals and Reasons for doing Business
Continuity and Disaster Recovery
2. What are BC and DR
3. RTO/RPO
4. Good DR Plans
5. Not so Good DR Plans
6. Closing information

10 ISACA 2007, Jeffrey Blackmon

Goals and Reasons for BC and DR

11 ISACA 2007, Jeffrey Blackmon

 Principle Goals

 Provide for the safety of all

employees

 Minimize business downtime

12 ISACA 2007, Jeffrey Blackmon

 Reasons for Doing BC and DR

 Business Best Practices

 FEMA Best Practices

 Audit Requirements

13 ISACA 2007, Jeffrey Blackmon

 Reasons for Doing BC and DR
 Private Sector
 FSLIC √
 HIPAA
 OCC √
 GLBA
 Sarbanes Oxley √
 NASD 3510
 Government Sector
 FPC 65 √
 NIST 800-34
 A-123 Audit

14 ISACA 2007, Jeffrey Blackmon

 Financial Reasons
 Company Loss of $84,000 to $90,000 per
hour of downtime

 90% of companies that experience 1

week of data center down time go out
of business within 12 months

(CIO INSIGHT, IDC)

15 ISACA 2007, Jeffrey Blackmon

 More Financial Reasons
‘The cost of being unprepared’
By Jim Ellis
Energy $2,817,846
Telecom $2,066,245
Manufacturing $1,610,654
Finance/Brokerage $1,495,134
IT $1,344,461
Insurance $1,202,444
Retail $1,107,274
Pharmaceuticals $1,082,252
Banking $996,802
Food processing $804,192
Consumer $785,719
Chemicals $704,101
Average / hour $1,010,536

16 ISACA 2007, Jeffrey Blackmon

 Costs
(R. Witty, DRJ Fall 2006)

17 ISACA 2007, Jeffrey Blackmon

 High Startup Costs

18 ISACA 2007, Jeffrey Blackmon

What are BC and DR?

19 ISACA 2007, Jeffrey Blackmon

20 ISACA 2007, Jeffrey Blackmon
 DR Plan, what is it?
 IT Related

 Major disruption has occurred that is not

part of day to day SOP
 Hardware / Software requirements
 Step by step directions for full system
recovery
 Very detailed documents required

21 ISACA 2007, Jeffrey Blackmon

 DR Plan
 #1 Easy to use

 Recovery of all major Computer systems

based on Pre- determined priority (RTO)
 Details, details, details

(Hardware, software, configurations,

communications, disk storage, SAN
connections……. )
22 ISACA 2007, Jeffrey Blackmon
 BC Plan
 #1 Easy to use

 Recovery of all major business

processes
 People related
 Probably many manual processes to be
used for the short term

23 ISACA 2007, Jeffrey Blackmon

24 ISACA 2007, Jeffrey Blackmon
 Plain and Simple
 BC/DR are Risk Mitigation

 No way to eliminate all risks

 Properplanning will reduce the risks to

an acceptable level

25 ISACA 2007, Jeffrey Blackmon

RTO and RPO

26 ISACA 2007, Jeffrey Blackmon

 Recovery Time Objective (RTO)

 Themax allowable time that a business

system, application or resource is
allowed to be down or offline

 RTO is determined by business owners,

not IT department

27 ISACA 2007, Jeffrey Blackmon

 Recovery Point Objective (RPO)

 The amount of data that is acceptable to

lose since the last successful backup
was completed

 RPO is determined by business owners,

not IT department

28 ISACA 2007, Jeffrey Blackmon

 Recovery Point Objective
Recovery Time Objective
Standard Tape
Backup Recovery RTO (24 hours)

RPO (12 hours)

DISASTER

Midnight Noon Midnight Noon Midnight

Noon Monday Tuesday Wednesday
Backup Backup Backup
Tape Made Tape Made Tape Made

29 ISACA 2007, Jeffrey Blackmon

 Recovery Point Objective
Replicated Data
Recovery Time Objective
Backup Recovery RTO (12 hours, rebuild system)

RPO (2 minutes)

DISASTER
Real time replication

Midnight Noon Midnight Noon Midnight

Noon Monday Tuesday Wednesday
Backup Backup Backup
Tape Made Tape Made Tape Made

30 ISACA 2007, Jeffrey Blackmon

Find the Cost Effective Solution

Cost Effective Solution

Costs

Time

Business Interuption Cost Recovery Costs

31 ISACA 2007, Jeffrey Blackmon

 RPO / RTO Example
 Major financial institutions on mission critical
systems
 RPO = 0 hours, on some applications
 RTO = 2 hours, on some applications
 After 96 Hours, major financial institutions
will probably not recover

By Jay Ranade, CISSP, CISA, CBCP, CISM

President, Jay Ranade Consultants, Inc.

32 ISACA 2007, Jeffrey Blackmon

 RPO / RTO Example
 Major breakfast cereal producer
 RPO = 7 days
 RTO = 7 days

 Put it all into perspective

 Very regular shipments to distributors by boxcar
 Only breakfast cereal, if problems occur, then re-
ship

By DRII Classmate, 1999

33 ISACA 2007, Jeffrey Blackmon

 RPO / RTO Expectations
 ‘Usually’a large gap in management
expectations as compared to actual
recovery abilities

 Talk with technical staff

34 ISACA 2007, Jeffrey Blackmon

What a plan should look like

35 ISACA 2007, Jeffrey Blackmon

 Good DR plans
 Besure you keep in mind that DR plans
are to recover computer and network
systems

36 ISACA 2007, Jeffrey Blackmon

 NIST 800-53,
Recommended Security Controls for Federal Information System

FAMILY: CONTINGENCY PLANNING

 CP-1 CONTINGENCY PLANNING

POLICY AND PROCEDURES
 CP-2 CONTINGENCY PLAN
 CP-3 CONTINGENCY TRAINING
 CP-4 CONTINGENCY PLAN
TESTING
 CP-5 CONTINGENCY PLAN
UPDATE

37 ISACA 2007, Jeffrey Blackmon

 NIST 800-53,
Recommended Security Controls for Federal Information System

FAMILY: CONTINGENCY PLANNING

 CP-6 ALTERNATE STORAGE SITES

 CP-7 ALTERNATE PROCESSING SITES
 CP-8 TELECOMMUNICATIONS
SERVICES
 CP-9 INFORMATION SYSTEM BACKUP
 CP-10 INFORMATION SYSTEM
RECOVERY AND RECONSTITUTION

38 ISACA 2007, Jeffrey Blackmon

 Good DR plans
 Disaster definition
 Who can activate the DR plan?
 Critical computer applications
 Escalation Plans / Decision Plans

39 ISACA 2007, Jeffrey Blackmon

 Good DR plans
 Listof Recovery Team Members and
contact info
 Vendor Contact Information
 Communications Vendor Contact
Information
 Hotsite contact information
 Offsite storage contact information

40 ISACA 2007, Jeffrey Blackmon

 Good DR plans
 Hardware / Software recovery for each
and every critical system based on
RPO/RTO

 Network recovery information

 Detailed configuration information

41 ISACA 2007, Jeffrey Blackmon

 Good DR plans

 Up to date
 Information on last time this DR plan
was tested (Minimum is annually)
 Change Log to the plan
 Returning to normal operations

42 ISACA 2007, Jeffrey Blackmon

Not so Good DR Plans

43 ISACA 2007, Jeffrey Blackmon

 Not so Good DR plans
 No Executive Sponsor
 Unrealistic Budget
 (< 2% of Data Center total budget)
 Unrealisticrecovery strategy
 Not Exercised / Tested
 Testing only partial of a system
 No training
 No Priority on recovery of systems
44 ISACA 2007, Jeffrey Blackmon
 Not so Good DR plans
 Copied from another site with no
updates
 General in nature
 3 inch binder
 Overabundance of color charts and
slides
 High on fluff
 Short on useful information

45 ISACA 2007, Jeffrey Blackmon

 Not so Good DR plans
 PURPOSE
 OBJECTIVES
 SCOPE
 AUTHORITIES
 REFERENCES
 MANAGEMENT RESPONSIBILITIES
 ORGANIZATION OF THE PLAN
 DEFINITIONS
 CANCELLATION
 DISTRIBUTION
 OVERVIEW
 POLICY
 ASSUMPTIONS
 CONCEPT OF ACTIVATION
 DEPLOYMENT CONDITIONS

46 ISACA 2007, Jeffrey Blackmon

 With Logic like this

47 ISACA 2007, Jeffrey Blackmon

They may be trying to Bamboozal you!

48 ISACA 2007, Jeffrey Blackmon

 Remember
 Review the plan at a high level
 Recovery of Systems and Communications,
that is key
 Who needs to be contacted?
 Where do we go?
 Acquire equipment
 Restore Operating Systems, applications and data
 Restore Communication

49 ISACA 2007, Jeffrey Blackmon

 Remember
 Stickto the key points and don’t get
distracted by all of the rest

 Donot get bogged down in the fine

detail

50 ISACA 2007, Jeffrey Blackmon

Closing

51 ISACA 2007, Jeffrey Blackmon

Front end security vs back end BC/DR

 BC / DR activation are last resort efforts

 Risk levels go high

 Spendthe time, effort & money to

develop a very strong front end security
program to avoid a disastrous event

52 ISACA 2007, Jeffrey Blackmon

 Thank You for Attending!

53 ISACA 2007, Jeffrey Blackmon