OIT Data Center Emergency Communications Plan Action Steps

December 2009 Prepared by: Disaster Recovery Services

OIT Data Center Communications Plan

Purpose: The OIT Data Center (OIT DC) houses servers for University of Minnesota enterprise applications, supported by the Office of Information Technology (OIT), servers supported by Minnesota State Colleges and Universities (MNSCU), and servers supported by Collegiate IT departments (Tenants). Power outages to the building have highlighted the need for a coordinated emergency notification system to all three groups of OIT DC users. Scope: This communication plan outlines the process for distributing needed information efficiently and effectively to stakeholders by defined roles and responsibilities in the event of an unplanned outage. This plan does not address scheduled outages i.e., announced in the Change Request Preview Report or communicated in advance by the Office of Information Technology (OIT) and the Office of Communications and Advancement (OCA). In addition, this Communication Plan assumes that OIT and the Collegiate IT departments (Tenants) have established Server - Order of Shutdown priorities for responding to the following requirements: 1. total shutdown of all OIT DC equipment for OIT, MNSCU and Tenants 2. shutdown of all OIT DC Priority 2 equipment for OIT, MNSCU and Tenants 3. shutdown of all OIT DC Priority 3 equipment for OIT, MNSCU and Tenants This plan does not address communicating with building occupants. Communication Plan Goals: OIT Total Shutdown Protect Data The communication plan lists the steps needed to reach OIT, MNSCU, and tenants when all equipment in the OIT DC must be shut down and maintain communication during the event. OIT Power Outage Priority 2 Partial Shutdown Preserve high priority services The communication plan lists the steps needed to reach OIT, MNSCU, and tenants when systematic shutdown involves Priority 2 equipment in the OIT DC and maintain communication during the event. OIT Power Reduction Priority 3 Partial Shutdown Sustain services The communication plan lists the steps needed to reach OIT, MNSCU, and tenants when systematic shutdown will start with Priority 3 (Test / Development) equipment in the OIT DC and maintain communication during the event.

Z:\opt\scribd\conversion\tmp\scratch17154\65909908.docx Last Revised: June 11, 2010 Print Date: August 24, 2011

Page 2 of 8

Possible Scenarios: Scenario Sudden total loss of power in the OIT data center After a Priority 3 Partial Shutdown, additional reduction in power usage is needed The temperature in the Data Center is rising with no relief anticipated

Action to take Implement OIT Total Shutdown Communication Plan Priority 1 Implement OIT Power Reduction Priority 2 Partial Shutdown Communication Plan Implement OIT Power Outage Priority 3 Partial Shutdown Communication Plan

OIT Total Shutdown - steps Contact OIT, MNSCU and tenants when all equipment in the OIT DC must be shut down. 1. Data Center Support sends email Alert Level 1 (Red) Immediate Action total OIT Data Center Shutdown to oit-dcalert-l@lists.umn.edu, if listserv is down send email to dcalert@umn.edu. 2. OCA will update www.oit.umn.edu and add updates as more information is available. 3. OCA will update the banner on the System Status page http://systemstatus.umn.edu/view/Main. 4. Administrators for specific systems will update http://systemstatus.umn.edu/view/Main. 5. When OIT DC is fully functional send email Alert Level 0 (Green) Cancel OIT Data Center Alert. 6. Schedule a post event evaluation to determine what worked and what did not work. OIT Power Outage Priority 2 Partial Shutdown - steps Contact OIT, MNSCU and Tenants when systematic shutdown will continue with Priority 2 Production equipment in the OIT DC. 1. Data Center Support sends email Alert Level 2 (Red) Immediate Action Priority 2 Partial OIT Data Center Shutdown to oit-dcalert-l@lists.umn.edu, if listserv is down send email to dcalert@umn.edu. 2. OCA will update www.oit.umn.edu and add updates as more information is available. 3. OCA will update the banner on the System Status page http://systemstatus.umn.edu/view/Main. 4. Admin for specific systems will update http://systemstatus.umn.edu/view/Main. 5. When OIT DC is fully functional send email Alert Level 0 (Green) Cancel OIT Data Center Alert. 6. Schedule a post event evaluation to determine what worked and what did not work.

Z:\opt\scribd\conversion\tmp\scratch17154\65909908.docx Last Revised: June 11, 2010 Print Date: August 24, 2011

Page 3 of 8

OIT Power Reduction Priority 3 Partial Shutdown - steps Contact OIT, MNSCU and tenants when systematic shutdown will start with Priority 3 test / development equipment in the OIT DC. 1. Data Center Support sends email Alert Level 3 (Red) Immediate Action Priority 3 Partial OIT Data Center Shutdown to oit-dcalert-l@lists.umn.edu, if listserv is down send email to dcalert@umn.edu. 2. OCA will update www.oit.umn.edu and add updates as more information is available. 3. OCA will update the banner on the System Status page http://systemstatus.umn.edu/view/Main. 4. Administrators for specific systems will update http://systemstatus.umn.edu/view/Main. 5. When OIT DC is fully functional send email Alert Level 0 (Green) Cancel OIT Data Center Alert. 6. Schedule a post event evaluation to determine what worked and what did not work. Alert Levels and Messages: 1. DRS, working with MNSCU, Tenants, Data Center Support and OIT OCA, developed the contents of the email messages. See Appendix page 7 for details. 2. DRS will develop and maintain both a listserv and a departmental email list. See Appendix page 6 for details. 3. DRS will provide detailed instructions on how to use the alert email messages and lists to Data Center Support. Sign up for Alert Levels and Messages: 1. Send contact information (name, email ID, phone # and dept name) to oit-drs@lists.umn.edu 2. If contact after business hours is required, request text messaging when signing up for Alert Notifications. Please provide cell phone # and carrier information. Text messages will be sent in both listserv and email notification modes.

Z:\opt\scribd\conversion\tmp\scratch17154\65909908.docx Last Revised: June 11, 2010 Print Date: August 24, 2011

Page 4 of 8

OIT Outage Email Alert Notifications

All emails will be sent to oit-dcalert-l@list.umn.edu, if listserv is not available use dcalert@umn.edu Alert Level Use eMail Template Order of shutdown OIT
1. 2. 3. Production Priority 1 Production Priority 2 Test /Dev/DR

Order of shutdown Tenants

1. 2. 3. 4. Production-Critical ProductionImportant Production-Non Essential Test / Development Test / Development Production-Non Essential

Estimated Response Time

As specified in communication

Level 1 (Red) Critical Immediate Total OIT Shutdown Purpose: Protect Data Level 2 (Red) Severe Immediate Priority 2 Partial OIT Shutdown Purpose: Preserve critical services Level 3 (Red) Severe Immediate Priority 3 Partial OIT Shutdown Purpose: Sustain Services Level 4 (Yellow) Status of Partial OIT Shutdown Level 0 (Green) Cancel of Alert

Alert Level 1 (Red) Immediate Action Total OIT DC Shutdown

Alert Level 2 (Red) Immediate Action Priority 2 Partial OIT DC Shutdown

1. 2.

Test / Development Production Priority 2

1. 2.

15-20 minutes

Alert Level 3 (Red) Immediate Action Priority 3 Partial OIT DC Shutdown

1.

Test/Dev/DR

1.

Test / Development 15-20 minutes

Alert Level 4 (Yellow) Status of Partial OIT DC shutdown Alert Level 0 (Green) Cancel OIT DC Alert

Informational

Informational

n/a

Cancel

Cancel

n/a

Level T (Green) Alert Level T (Green) Test Testing of DC Alert Test of OIT DC Alert notification system Full text of suggested messages is in the Appendix page 7.

Test

n/a

Thunderbird Templates for DC Operations Alert level 1 Red Immediate Action Total OIT DC Shutdown Alert level 2 Red Immediate Action Priority 2 - Partial OIT DC Shutdown Alert level 3 Red Immediate Action Priority 3 - Partial OIT DC Shutdown Alert level 4 Yellow Status of Partial OIT Shutdown Alert level 0 Green Cancel Alert Alert level T Test

Z:\opt\scribd\conversion\tmp\scratch17154\65909908.docx Last Revised: June 11, 2010 Print Date: August 24, 2011

Page 5 of 8

Appendix:
Messages: Messages will be distributed using listserv oit-dcalert-l@list.umn.edu or if listserv is unavailable departmental email dcalert@umn.edu. The dcalert email list is the backup for the listserv since listserv is hosted on only one machine in OIT. Departmental email lists are stored in multiple locations. Email will be sent as cell phone text message to those who requested text message notification. The oit-dcalert-l listserv includes: y Data Center Tenants listserv maintained by Joe Zanmiller - oitdctenants y Collegiate IT Directors listserv maintained by John Sonnack - mis-it-directors y Individuals identified by the Collegiate IT Directors y OIA managers listservs maintained by Patton Fast oia-mgr and oia-sr-mgt y DRS listserv maintained by John Snider oit-drs y Departmental email lists for OIA and AAS groups, 1-Help and TAC y Other OIA and AAS individuals identified who are not on other lists y OCA contacts Kris Adelmann and Ben Neeser y Other individuals and groups as identified The dcalert departmental email list: y Individuals on the Data Center Tenants listserv maintained by Joe Zanmiller - oitdctenants y Individuals on the Collegiate IT Directors listserv maintained by John Sonnack - mis-it-directors y Individuals identified by the Collegiate IT Directors y Individuals on the OIA managers listserv maintained by Patton Fast oia-mgr and oia-sr-mgt y Individuals on the DRS listserv maintained by John Snider oit-drs y Departmental email lists for OIA and AAS groups, 1-Help and TAC y Other OIA and AAS individuals identified who are not on other lists y OCA contacts Kris Adelmann and Ben Neeser y Other individuals and groups as identified Note: TXT-U is not an option since it is only for University wide emergencies and campus closings. Notification List: The following parties are to respond to a situation in the OIT DC. y OIA Sr. Director and Sr. Manager of operations and infrastructure y OCA staff y OIA system administration staff y OIA storage administration staff y OIA database administrators y Academic Computing manager and staff y Administrative Applications and Systems manager and staff y Data center tenant technical staff y Disaster Recovery Services manager and staff y MN State Colleges and Universities operational manager and technical staff y Network and Telecommunication Services staff
Z:\opt\scribd\conversion\tmp\scratch17154\65909908.docx Last Revised: June 11, 2010 Print Date: August 24, 2011

Page 6 of 8

Suggested email message text:

Note: all emails have Reply To: cco-ops@umn.edu except Alert Level T where Reply To: oit-drs@lists.umn.edu Note: since many of those receiving these alerts are being notified by text message, all emails are very terse Subject: Alert Level 1 (Red) Immediate Action Total OIT DC Shutdown
Shutdown all tenant, OIT and MNSCU servers and equipment in the OIT Data Center. Complete the shutdown by SPECIFY TIME. See the OIT Web site, www.oit.umn.edu, for the latest status information.

41 WORDS 251 CHARACTERS

Subject: Alert Level 2 (Red) Immediate Action Priority 2- Partial OIT DC Shutdown
Using Priority Plan 2, shutdown identified tenant, OIT and MNSCU servers and equipment in the OIT Data Center. Complete the shutdown by SPECIFY TIME; reply to cco-ops@umn.edu. See the OIT Web site, www.oit.umn.edu, for the latest status information.

50 WORDS 321 CHARACTERS

Subject: Alert Level 3 (Red) Immediate Action Priority 3- Partial OIT DC Shutdown
Using Priority Plan 3, shutdown identified tenant, OIT and MNSCU servers and equipment in the OIT Data Center. Complete the shutdown by SPECIFY TIME; reply to cco-ops@umn.edu. See the OIT Web site, www.oit.umn.edu, for the latest status information.

50 WORDS 321 CHARACTERS

Subject: Alert Level 4 (Yellow) Status of Partial OIT DC Shutdown

The energy reduction from the Priority Plan shutdown is successful. No further reduction is needed at this time. See the OIT Web site, www.oit.umn.edu, for the latest status information.

39 WORDS 241 CHARACTERS

Z:\opt\scribd\conversion\tmp\scratch17154\65909908.docx Last Revised: June 11, 2010 Print Date: August 24, 2011

Page 7 of 8

Subject: Alert Level 0 (Green) Cancel OIT DC Alert

OIT Data Center has resumed normal operations; tenants can resume their normal operation. Thank you for your cooperation and assistance during this event.

31 WORDS 194 CHARACTERS

Subject: Alert Level T (Green) Test of OIT DC Alert
Test of the OIT Data Center Alert notification system. Reply including time message received.

23 WORDS 134 CHARACTERS

Z:\opt\scribd\conversion\tmp\scratch17154\65909908.docx Last Revised: June 11, 2010 Print Date: August 24, 2011

Page 8 of 8