Plastoworld Services India Pvt Ltd

Data Center Manual

Data Center Manual Version 2.0 1

Document Control

Document Reference DC manual /EN-IN-02.05

Document Description Data center Manual

Document Owner Chirag Sheth

Department Operations
Version Date Status Changes Made Author Initialled
2.0 14th November, First Release
2006

PLASTOWORLD WORLDWIDE PROPRIETARY

This document contains highly sensitive, confidential and trade secret information,
and may not be disclosed to third parties without the prior written consent of
Plastoworld Worldwide.

Data Center Manual Version 2.0 2

 Table of Contents

1. Introduction ………………………………………………………………………3

2. Process Flow………………………………………………………………………4

3. Data Center Infrastructure (Servers)……………………………………. 5

4. Data Center Infrastructure (Network)………………… ……….……….6

5. Network Description…………………………………………………………....7

6. Features of the Data Center………………………………………………….11

7. Network Security…………………………………………………………………12

8. Activities Performed in the Data Center………………………….…….. 14

Installations…………………………………………………… …..15

AS/400 Health Monitoring……………………………………..16

Windows Server Health Monitoring…………………………18

Backup and Restore in AS/400 Servers.......................20

Backup and Restore in Windows Servers...... …………..23

System Administration Activities………………..….......... 24

Network Monitoring Process…………………………………. 28

9. Escalation of Incidents in the Data Center ……………………….…… 30

10.Contact Details…………………………………………………………………… 31

Data Center Manual Version 2.0 3

Introduction

Plastoworld’s servers are located in the state of art data center of Reliance in Navi
Mumbai. Reliance Data Center provides the n+2 infrastructure redundancy, i.e.
multiple power supplies, UPS with Battery Backup and captive power plant.

Plastoworld has a complete Disaster recovery / Business Continuity Plan for

Banks running the ATM Networks and related services like Mobile recharge,
Shared ATM services, etc.

Plastoworld’s Switch ITM runs on IBM’s robust server I5/OS (AS/400) and
Plastoworld has two AS/400s running parallel to each other, where data gets
mirrored online real time, on site.

Plastoworld has a DR Site at Bangalore, which is running an AS/400, which mirrors

the backup server of primary site.

So effectively Plastoworld is running three switches parallel to each other.

Process Flow

Data Center Manual Version 2.0 4

MIMIX Application is used to mirror the data from the Production server to Backup
server and then from Backup Server to DR Server.

CITY I

MIMIX ONLINE
MIRRORING

Production Server Backup Server

-- Settlement
-- Backups

-- Settlement
- Backups
- Offsite (in city tapes to be stored).

MIMIX ONLINE
MIRRORING

MIMIX ONLINE CITY II - Banglore

MIRRORING

-- Settlement
-- Backups.

Disaster recovery Server

Data Center Infrastructure (Servers)

Data Center Manual Version 2.0 5
 No DAKC System Model No. Location OS

1 As/400 Production I-520 IDC - 1 R-4 V5R4

2 As/400 Backup I-520 IDC - 2 R-2 V5R4

3 As/400 Development 270 IDC - 1 R-3 V5R4

4 Electra AIX IBM - P - 630 IDC - 1 AIX 5.1

SUN Fire V
5 Electra-Sun IDC - 1 R-2 Solaris
240

IDC -
6 AS/400 DR 810 V5R4
BANGLORE

Proliant ML370
7 MOP IDC - 1 R-2 WIN2k
G2
Proliant DL360
8 JUN IDC - 1 R-2 WIN2k
G2
IBM Blade
9 INVAS IDC - 1 R-2 WIN2k3
Center HS20
Proliant
10 Euro Mon IDC - 1 R-1 WIN2k3
ML350 G3
Transaction Monitoring
Proliant DL380
11 + Test Server IDC - 1 R-2 WIN2k
[Application] G4

12 SDMS 1-1.2 DL360 G3 IDC - 2 R-1 WIN2k

13 SDMS - 2 - 1.5.1 HP DL 360 G4 IDC -2 R-1 WIN2k

Proliant DL 380
14 Proxy (INPR01) IDC - 1 R-1 WIN2K
G3
15 AMA2 ML570 3G IDC -2 R-1 WIN2k

16 EWIDT - Data miner DL360 G4 IDC -2 R-1 WIN2k

EWIDT - Database
17 ML370 G4 IDC -2 R-1 WIN2k
Server

Plastoworld Services IBM Blade

18 IDC -1 R-1 WIN2k
Desk (CRM) Center HS20
Data Center Manual Version 2.0 6
 Plastoworld Service HP Desktop
19 BKC WinXP
Desk Middle ware 320M
Proliant
20 ATM Next IDC -1 R-3 WIN2k
ML350 G3

IBM Blade
21 Recon IDC -2 R-1 WIN2K
Center HS20

IBM Blade
22 Ewidt-Lite
Center HS20

Central SQL DB Server

23 ML350 G3 IDC-2 R-1 WIN2k
(AMA-1)

24 Terminal Server HP IDC-1 R-3 WIN2k

25 FTP Server ML370 G2 IDC-2 R-1 WIN2k

Veritas Backup/ restore HP Server TC

26 IDC -2 R-1 WIN2K3
Server 2120
HP Server TC
27 EN Finance BKC WIN2K3
2120
Proliant
28 INDC01 BKC WIN2k
ML370 G3

29 INEX01 Proliant ML370 BKC WIN2K

30 INMU1-adp-sv1o DL380 G4 BKC WIN2K

IBM Blade
31 EWIDT WIN2K3
Center HS20
Checkpoint FW Log
32 Assembled IDC - 1 R-3 WIN2K
Server
33 Master Card Setup IDC - 1 R-2
34 VISA Rack IDC -1 R-5

Data Center Infrastructure (Network)

Routers

Cisco 3660 Router I - Leased Lines

Cisco 3660 Router II - ISDN configuration
DAX Router 1600 – IDBRT
Data Center Manual Version 2.0 7
Cisco 1800 Router - MASTERCARD

Firewall

Nokia IP330 Firewall - Active

Nokia IP330 Firewall - Redundant

Switches

Cisco 2950 Switch

Catalyst 3560 Switch I
Catalyst 3560 Switch II
Catalyst 2960 Switch I
Catalyst 2960 Switch II
16 Port KVM Switch

VPN Boxes

Cisco Pix 525 I - Active

Cisco Pix 525 II - Redundant

Network Description

All the customers of Plastoworld are connected to the two main core routers.
These two core routers are Cisco 3660 series routers.

Data Center Manual Version 2.0 8

 One is used for connecting all leased line and other for the backup ISDN.

Both these routers are running HSRP for LAN redundancy to outside network.

These routers are running OSPF as a dynamic routing protocol for routing
convergence.

For the entire customer networks, static routing is used and these static routes
are redistributed into OSPF process.

They have all their servers in 202.138.123.64/27, which is in the DMZ segment
protected by checkpoint firewall.

There is a cluster of two Checkpoint firewall in which one is active and the
other is in standby mode for redundancy.

In this DMZ segment there are multiple servers, which communicate to the
customers ATM switches, and other servers for various banking operations.

Their core router is directly connected to the checkpoint firewall and which in turn
is connected to the Layer 3 switch (Cisco3550).

All the Customers connecting via Leased lines are having ISDN as backup link for
redundancy.

When ever any leased line goes down ISDN is fired automatically within 3
seconds and is disconnected in 300 secs once the leased line comes up.

For ISDN to be connected to the ISDN router E1 PRI is being used and also 8
ISDN lines are used as a backup of E1 PRI.

Data Center Manual Version 2.0 9

Data Center Manual Version 2.0 10
Features of Data Center

Humidity, Ventilation, Air conditioning (HVAC Factor)

Direct Expansion Air conditioners are used and the condenser is located on the roof
Capacity of the AC units: 18.5 TR per Unit

A constant 22 Degree Celsius is maintained

Ambient Relative Humidity is kept at 55%
Filtration of dust particles is achieved up to 5 microns

Fire Detect and Suppress Systems

Fire Detect and Suppress System is localized with fire detectors placed on each rack.

Fire detection systems used are

1) Ionization type
2) Photoelectric type
3) Thermal Detectors

Fire Suppression system used is FM-200 fire extinguishing system (Kidde Finwal
make)

Power System

The power is obtained from two separate feeders of state run KPCTL/MSEB, these
feeders are connected to DG sets which in turn is connected to multiple UPS systems

Data Center Manual Version 2.0 11

From the UPS power is supplied to multiple PDU’s each with multiple redundant
card of 3KVA each

Diesel Generator used is K.CUMMINS make Diesel Genset with 1250 KVA (3 Nos)

Physical Security

The Physical security of the Data Center is three tiered

1) Biometric Reader
2) Smart card Access
3) CCTV Surveillance

Data Center Manual Version 2.0 12

 Network Security

Salient Features
The salient features of the Plastoworld setup, specifically addressing the
security requirements of the Plastoworld are as below:

 Layers of security protect the Plastoworld network: The routers connecting

customers use ACL’s (Packet Filtering) and the Internal Firewall uses (Stateful
Inspection).

 All the routers in the Plastoworld are configured with access-lists that prevent
traffic from one customer to flow to the other customer’s network. This
ensures that the Plastoworld network is not used as a transit network to
access one customer’s network from another’s.

1. Access Security on Switches and Routers

 Access to the routers and switches is done through login

authentication for the initial “user mode” access.

 The privileged second level “enable mode” password differentiates

users with appropriate rights based on the user credentials supplied.
o This includes different privilege mode levels and appropriate
password (currently four different levels are defined)
o An Access Control Server (TACACS+) validates the login
authentication centrally, which is a Security server having its
own database of users and the attributes that define the login
passwords and access privileges.
o The TACACS+ Server also authorizes each and every command
the user tries to execute, based on his privilege before being
executed on the device.
o All commands, the start and stop time-stamps etc., are also
logged by the TACACS+, for auditing and monitoring of
activities.

 The Telnet and console accesses are secured and controlled by

TACACS+ Server.

 The secondary TACACS+ provides for continuous availability of AAA

security feature, in case primary TACACS+ fails. In addition, local
“username password” feature is also enabled, if connectivity to either
of the security fails, thereby ensuring security is not breached in all
case.

Data Center Manual Version 2.0 13

 2. Configuration Level Security

The following section details the security that has been implemented by
fine-tuning and hardening the configuration of routers and switches.

o Access Control Lists (ACL’s) are implemented on all routers as a method to

filter unwanted traffic.
These access control lists (Packet Filters) are implemented to restrict traffic only
From/To certain hosts or service and to prevent un-authorized access from entering
or passing through other customers networks.

o All the traffic coming from external customer networks is forced to go through
firewall, which protects and prevents Plastoworld’s internal network from any
external attack or security breach.

o Secure IPSEC tunnel for HDFC customer is configured as per client’s security
requirement.

o The backup ISDN lines for leased lines are configured with PPP CHAP
authentication, which allows for a secure client router authentication with
Plastoworld’s core routers EURORT1 and EURORT2.

o By default, there are many unused and un-necessary services running on

Cisco routers and switches. These settings give rise to vulnerabilities, which
can be exploited by an attacker to obtain information or gain un-authorized
access into the network. Default services like CDP, Chargen, Echo, IP Source
routing, Proxy ARP, directed broadcasts, ICMP unreachable, mask replies etc
have all been disabled.

o Telnet access-lists to limit administrative access to the router and switches

have been implemented.

o SNMP access-lists to limit SNMP access to the router and switches have been
implemented.
o Exec-timeout of 2 minutes 30 seconds has been enabled on the Telnet,
auxiliary as well as console ports.

o TCP keepalives on incoming connections has been enabled (with service tcp-
keepalives-in). This helps to guard against both malicious attacks and
"orphaned" sessions caused by remote system crashes.

o Login banners have been implemented on the routers and switches.

o Un-necessary services like HTTP, Finger, and bootp have been disabled.

o Access-list violations are recorded to aid in analysis of un-wanted traffic.

Data Center Manual Version 2.0 14

 3. Routing Protocol Level Security

The Core routers (EURORT1 and EURORT2) use OSPF for communicating
network reachability.
 These OSPF routes are exchanged using secure route authentication
(through secure MD5 HASH), which verifies that routes learnt are
indeed through authenticated valid OSPF peers.

 The Serial, BRI and dialer interfaces are configured as passive

interfaces. This ensures that Core routers don’t announce any routes
to unwanted OSPF neighbors.

4. Miscellaneous Settings

The Cisco discovery protocol is disabled on all interfaces thereby preventing the
neighboring devices in learning about directly connected devices, which might
result in sharing of device information to wrong hands of the hacker.

The PPP session that results when an ISDN call is established between the
Plastoworld routers and their customer routers is authenticated using PPP CHAP
authentication. This ensures that only authorized customer routers with a correct
CHAP password can connect to the Plastoworld network.

Data Center Manual Version 2.0 15

 Activities Performed in the Data Center

Activities

Installation Monitoring Network

Backups
Upgradation of Servers Monitoring

Installation/Upgrade - System Administrator, Network Team

Monitoring of servers - System Administrator, Network Team, Helpdesk Team

Backups - System Administrator, Helpdesk Team

Network Monitoring - Network Team

Data Center Manual Version 2.0 16

Installations
All kinds of installations and upgrades are done in the data center, which requires
Console access to the Server

The common Installation activities performed are

 OS installation

 PTF installation

 Service pack Installation

 Disk Installation

 Hardware Installation

 Server Installation

 Router Installation

 Modem Installation

Full system backup of AS/400 Servers and RS6000 Server is conducted through their
respective consoles as TCP/IP services has to be put down during the backup process

Data Center Manual Version 2.0 17

AS/400 Health Monitoring
In the Data center there are three AS/400 Servers (520) running OS/400 V5R3

1) Production – IBM – 9406-520

2) Backup – IBM – I – 520
3) Development IBM – 9406-270

The Data center AS/400 activities involves monitoring all the aspects of the three
servers like

1) Disk Status
2) CPU % utililization
3) ASP % utilization
4) Subsystem Status
5) Job Queue Status
6) Work Problem Entries
7) SRC codes
8) Network Status
9) Output queue Management
10) Shared Pool Utilization
11) Tape Drive Status
12) Last night backup status

The complete health monitoring is done four-five times in a day.

Hourly Monitoring

Two parameters are checked each and every hour

1) ASP % utilization
2) CPU % utilization
3) Display Message

A program is scheduled in an hourly basis generating the current CPU and ASP
status

Any unusual reading is crosschecked and the activity responsible for elevation or
decline in the normalcy is either halted or terminated

Any hardware related issue is reported as an entity of Incident reporting under the
guidelines prescribed in the incident management Policy. (Refer Incident
management Document)

MIMIX Monitoring

The integrity of Data replication application (MIMIX) is monitored 24/7; any

mirroring delay is escalated to the MIMIX Application administrator with the detailed
report from the logs.

Data Center Manual Version 2.0 18

 Sample Morning Checklist of AS/400 Servers

1. Check the system asp and cpu utilization using WRKSYSSTS 89.88/10.4

2. Check the DASD status using WRDSKSTS and confirm that all are ACTIVE OK

3. Check the subsystem status using WRKSBS and confirm that all subsystems are active OK

4. Use the WRKPRB command to detect new problems within the system OK

5. Check the SYSOPR message queue for any Message needing Reply OK

6. Check the logs for Daily and Weekly backup for successful completion OK
The spool files for daily and weekly backup can be obtained from D5AOUTQ outq

7. Check the scheduled jobs for successful submission using WRKJOBSCDE OK

8. Check that the jobq’s are all in the RLS mode by using WRKJOBQ command OK

9. Ping the Test Server to assure connectivity OK

10. Check MIMIX for Data Synchronization and Inactive Objects OK

11. Check the Message Queue of OMS400 to detect the Delay in the mirroring Process Done

12. Check the jobs of Subsystem’s MIMIXSBS, see that none

Of them are in MSGW state except the job OMSJRNMGR under OMS400 subsystem OK

13. Check the file QPDSPSTS using the command WRKSPLF Vijay to obtain the CPU Done
Utilization and ASP status in the interval of 60 minutes

Comments

Marginal Increase in ASP Utilization

Date:

Checked by

Verified by

Data Center Manual Version 2.0 19

Windows Servers Health Monitoring
There are seven windows Production Servers in the Data Center

The production Servers are

MOP
JUN
INVAS
Euro Mon
Transaction Monitoring + Test Server [Application]
SDMS 1-1.2
SDMS - 2 - 1.5.1
Proxy (INPR01)
AMA2
EWIDT - Data miner
EWIDT - Database Server
Plastoworld Services Desk (CRM)
Plastoworld Service Desk Middle ware
ATM Next
Recon
Ewidt-Lite
Central SQL DB Server (AMA-1)
Terminal Server
FTP Server
Veritas Backup/ restore Server
EN Finance
INDC01
INEX01
INMU1-adp-sv1o
EWIDT

The Servers are running windows 2000 & windows 2003 with the latest patches
on IBM / HP/Compaq Proliant Servers.

The servers are all on RAID and an exhaustive backup Strategy is implemented for
disaster recovery or failure

The health of the systems is monitored twice every day, once in the morning and
again in the evening

Data Center Manual Version 2.0 20

The monitoring activities include,

1) Disk Space on all Drives

2) Network card status

3) Raid Status

4) CPU utilization

5) Event viewer Logs

6) Security Logs

7) Performance logs

8) Suspicious activity monitoring

Sample Disk Space Utilization Report Created Daily

Data Center Manual Version 2.0 21

Backup and restore in AS/400 Servers

 ITM-Production Server backup Frequency

 Daily

The entire backup

In daily we are taking Selected User Libraries backup on LTO.
program is pre-written CL codes, we just call them at required time.
 Weekly

The entire backup

In weekly we are taking Selected User Libraries backup on LTO.
program is pre-written CL codes, we just call them at required time.
 Monthly

In Monthly we are taking All User Libraries backup + Configurations + Security Data.

 Quarterly

In every quarter we are take Entire System Backup. We do it on manual basis by

taking prior downtime.

 LTO Tape Recycling Procedure

1. Daily backup tape we are preserving for one week. (Ex. Monday tape will be use on
next Monday backup.) One LTO Tape require for each day backup and it takes
around 30 MIN. to complete after settlement.

2. Weekly backup tape we are keeping for one month. Out of 5 weeks we are
preserving last week tape till Quarterly backup complete. One LTO Tape require for
each week backup and it takes around 30 MIN. to complete after settlement & daily
backup.

3. Monthly backup tape will be preserve for three months. We will re-use 1 st month
backup media on 4th month backup. Out of three months last month backup media
we are preserving life long. One LTO Tape require for each month backup and it
takes around 45 MIN to complete after settlement & daily backup.

4. Quarterly backup tape (Entire System Backup) will take every three months as per
down time we will get and this backup media we are preserving life time. One LTO
tape require for each quarter backup and it takes around 2 Hrs to complete after
settlement & daily backup.

Data Center Manual Version 2.0 22

 5. Observation of Backup Activity –
Physically put the LTO Tape in server. (Device Name – TAPE02)
Check all the backup procedure.
Verify the successful completion of backup. (Through joblog it can verify)
Check the effect on server during backup.
(Constant check WRKACTJOB for CPU Utilization if exceed pl. inform to
System Administrator)

 List of Daily backup libraries. Total 35 Libraries. (No down time required.
Save while active – parameter using.)

ASIB155801, ASIC155801, ASID155801, ASIR155801, ASIT155801,

ASIB301501, ASIC301501, ASID301501, ASIR301501, LTSLIB, TVSLIB
BASE24P, B24ISOUP, DDCBPROD8, DDLBPROD8, DDLBPRODML,
DLKBPROD8, FINACLE01, IDBIDIP8, DINGBANK8, INGBANK8,
INGKIOSK8, NCEBPROD1, NDLBPROD8, NDLBPRODM8, NDUTPROD8,
NLKBPROD8, QGPL, ASILIB, MERLINLOGX, OPSQUERY, MIS,
STLFITMLIB, STLSAVFLIB, @JRNRCVLIB

 List of Weekly backup libraries. Total 33 Libraries.

(No down time required. Save while active – parameter using)

1. SAVE SECURITY DATA +

2. SAVE CONFIGURATION +
3. USER LIBRARIES:

ASIB155801 ASIC155801 ASID155801 ASIR155801 ASIT155801 BASE24P

B24ISOUP FINACLE01 ASID301501 ASIB301501 ASIC301501 ASIR301501
LTSLIB TVSLIB ASIR3015F1 ASIB3015F1 ASIB3015P1, ASIR3015P1 ASIR155814
ASIB155814, DLKBPROD8 DDCBPROD8 DDLBPROD8, NDUTPROD8,
NLKBPROD8 NDLBPROD8 IDBIDIP8, NDLBPRODM8 INGBANK8, INGKIOSK8
DINGBANK8, NCEBPROD1 STLSAVFLIB

 List of Monthly backup libraries. (No down time required)

1. ALL USER LIBRARIES (SAVLIB LIB(*ALLUSR) +

2. SAVE SECURITY DATA. +
3. SAVE CONFIGURATION DATA. +

 List of quarterly backup libraries.

Entire System Backup. (Down time required around 3 Hrs to take

backup in restricted user mode.) (GO Save – Option – 21).

Data Center Manual Version 2.0 23

Offsite Backup

The offsite backup process involves duplication of the fortnightly Weekly backup and
the Full system backup, once the duplication is done the tapes are dispatched to
Plastoworld BKC office through courier service.

The Offsite backup tapes are rotated in a monthly Basis

Archive Backup

Archive backup involves backing up of daily transactional Logs, the Archive tapes are
assigned for permanent storage

Restoration drill Activity

Weekly we are checking any two tapes of current week and restore on backup server to
check backup integrity.

We are preserving daily, weekly, monthly and quarterly backup. Daily tapes we are
keeping for one week only. This can be restore any time during week. This same applies
for weekly and monthly tapes for respective periods. Only we are preserving Quarterly
full system backup tape life long.

Data Center Manual Version 2.0 24

 October, 06 All Server Daily, Weekly & Monthly Backup Status
Backup Backup
Daily Backup of ITM, Development & Weekly Backup of ITM, Development
Date Day AIX(Electra) Servers & AIX ( Electra) Server Taken Checked Rema
By By
ITM Development AIX(Electra) ITM Development AIX(Electra)

1 Sunday Y Y Y Amit CHIRAG

2 Monday Y Y Y Amit CHIRAG

3 Tuesday Y Y Y Amit CHIRAG

4 Wednesday Y Y Y Amit CHIRAG

5 Thursday Y Y Y Amit CHIRAG

6 Friday Y Y Y Amit CHIRAG

7 Saturday Y Y Y Y Y Y Atul CHIRAG

Backup and Restore in Windows Servers

The Application used for the backup function is Veritas backup Exec for Windows
Servers Ver 9.1 and is installed in Plastoworld-3 Server.

The LTO Autoloader Tape device attached to Veritas Server located at IDC – 2.
The Backup Strategy adopted is “Grandfather – Father – Son” Media Rotation
Strategy under which fall Monthly Full system Backup, Weekly full system
Backup and Daily Differential backup.

On Sunday Nights Weekly Full System backup is scheduled and on the first day
of the month Monthly Full System Backup would be conducted.

On the weekdays (Monday to Saturday) Daily Differential Backup would be

carried out which would record only the changes occurred since the last Weekly
backup.

In the end of the year a Yearly Full System Backup would be conducted

Data Center Manual Version 2.0 25

Monthly Tapes and Yearly Tapes would be preserved Offsite for an year for the
sole purpose of Data Storage (Which can again be rotated the next year)

The weekly and Daily Tapes are meant for instant Disaster/Failure Recovery
and would be rotated

Weekly Tapes would be rotated every 4/5 weeks and Daily tapes would be rotated
every 7 days

For added Security the Weekly tapes are rendered Write protected for 4 weeks
and daily Tapes for 5 days.

The Media Rotational backup Scheme Ensures that each bit of Data for the Entire
year is preserved in Monthly/Yearly Tapes.

The Weekly/Daily Tapes ensures that the Disaster/Failure impact is minimal.

System Administration Activities

Restoration of log data

The transactional logs obtained during the settlement process is restored to the Test
ITM server for Customer related issues

Userprofile management

Creation Deletion and Management of the user profiles are done in accordance with
the User Policy and Password policy of Plastoworld.

Userprofile creation (once approved by the security officer)

Once the Security Officer Approves the creation of user profile with the specific
authority level it is created and the user is intimated about the same

Scheduled system maintenance Tasks

Scheduled maintenance like Purging, Service Pack Install, Bug fix install is carried
out in consultation with the application Team

Audit report generation

Security Audit report is generated for AS/400 Production Server and is stored in the
FTP server

Out queue management

Out queues of AS/400 Server is deleted at a stipulated interval

Maintaining the password policy

For more Details please refer the Password Policy Documents

Data Center Manual Version 2.0 26
User Assistance

Server login and system related issues of the server users are handled as a first level
of support

Daily Physical Check of the Servers

All the servers are physically checked daily to gather the status of the Fault
indicating Displays

Data Center Manual Version 2.0 27

AS400/Windows Servers Daily Physical Checklist

1) AS400 Server (Production) Status :-

2) AS400 Server (Test) Status :-

3) AS400 Tape Drive Status :-

4) MOP/JUNE/AMA/ATMNext/Proxy/FTP/Mail Server Status :-

5) Network Card Status of Windows Servers :-

6) MOP Tape Drive Status :-

7) RAID Status of Windows Servers :-

8) HSM Light Indicator Status :-

Comments

Date:-

Checked By:-

Data Center Manual Version 2.0 28

 Network Monitoring Process

Plastoworld monitoring is done from GMC on 24x7 basis. All the links and
devices are being monitored through NNM. Service desk is one of the tools,
which is integrated with NNM. So whenever any event occurs in NNM an incident
ticket is generated in Service Desk and a mail is send to the Level 1 engineer
as well as onsite engineer regarding the incident automatically for the
appropriate action.

Weekly as well as monthly reports are sent to customer on regular basis. These
reports mention about the link availability, performance and device performance.
It also contains ongoing as well as and pending issues description and status.

Management Process

Management for all the devices as well as links is done from GMC on 24x7 basis.
Whenever any link goes down a service desk ticket is mailed automatically to the
onsite engineer from Service desk.

Accordingly the onsite engineer rectifies the problem and updates the reason of
outage in GMC Level 1 engineer and he updates the ticket. After the link is up
ticket changes its state to resolve automatically but it doesn’t closes
automatically, it has to close manually by updating the reason for outage.

 Configuration Management: The process of identifying and defining

Configuration Items in a system, recording, and verifying the
completeness and correctness of Configuration Items.

 Password Management: Passwords are changed every quarterly and

keeping track of password changeover dates by recording them in file.

 Backup Management: The basic aim of backup management is to

have the backup of the more recent configurations for all the routers
installed in the network. Every night backup of all the routers in
Plastoworld backup is being taken through a script and kept in the
backup server.

 Change Management: Change Management is the practice followed

in GMC for ensuring all changes made in relation with router are
carried out in a planned and authorized manner. This includes ensuring
that there is a business reason behind each change, identifying the
specific services affected by the change, planning the change, testing
the change, and having a rollback plan should the change result in an
unexpected state of the router or accessories.

Data Center Manual Version 2.0 29

 For any Change in the routers, a Change Request Form is send to
Plastoworld for approval of the change through mail. After the
appropriate approval change is implemented within the speculated
time and is observed for a certain period of time so that it can be
rolled back if required.

 Link Management: In Plastoworld network setup almost all the

customers of Plastoworld are having ISDN as backup connectivity to
Plastoworld network. So as soon as lease line goes down, within three
seconds the isdn gets fired, thus ensuring that the wan connectivity
to the respective customer does not get interrupted.

Escalation Procedure

Escalation for any incident or any event is configured in Service Desk. It does
this by sending mails to the recipients whom are configured

Automatic notification is done for the following events:

Link Down
Link UP
ISDN Active
ISDN Disconnect
Router Down
Router UP
Firewall Down
Firewall UP
1 hr LL Escalation
2 hr LL Escalation
Switch Down
Switch Up

In these escalation Mails Onsite Engineers as well as Plastoworld Personnel’s

depending upon the criticality of the problem are marked. So that information
is passed automatically for any problem in network.

Shift Handover

Procedure for shift handover is provisioned. Whenever any Level 2 or Level 1

engineer completes his shift, he updates the shift handover document.

This documents will contains the important activities happened in his shift, pending
activities to do, also what action to be taken or is it for only passing information.
Both Onsite as well as GMC engineers have to update this document regularly

Data Center Manual Version 2.0 30

 Sample Daily Activity report (Network)

Date
Activity/Name 02Status/Time
NOV 2004 Remark Shift Eng

Day
Diebold isdn WEDNESDAY
Fired at 8.30 From Gmc-Bala Prabhakar
Diebold isdn Disc at 10.37am Rajesh called from Prabhakar
BKC.
Sbi isdn Our of order 8.30am Called SBI, spoke to Prabhakar
vikash he will be
looking after this.
Diebold isdn Fired at 5.30pm Nazia called from Prabhakar
BKC
Diebold isdn Disconnected at Nazia called from Prabhakar
3.30pm BKC

Device Status/Remark
EURORT1 Ok
EURORT1 Ok
NOKIA ( Sec) .66 Ok
NOKIA ( Pri ) .65 Ok
CISCO-PIX 501 Ok
CATALYST 2950 Ok
CATALYST 3550 Ok
CATALYST 3550 Ok
SNIFFER Machine Ok

Shift Engineer: prabhakar

Signature

Data Center Manual Version 2.0 31

 Escalation of Incidents in the Data Center

The escalation of the incidents in Data center is done in accordance with the
Incident management policy of Plastoworld

Once an Incident is identified it is notified immediately to the Incident

Response Manager (Mr. Sudhakar Dongre / Mr. Ramesh Thorat) along with
the appropriate log details

The incidents that are notified to the Incident manager are

1) Unauthorized access attempts

2) Unauthorized user activities

3) System malfunction

4) Restoration of sensitive Data

5) Disk malfunction/Over usage

6) Hardware malfunction

7) Virus/Worm Issues

8) Operating system/Software issues

9) Service pack/Bug fix notification from the operating

10)System/software vendor

Once the Incident management team sends in the recommendations it is

implemented immediately in tandem with the network Team if required.

The issues those are resolved without any escalation includes

1) Restoration of log data

2) Userprofile management
3) Userprofile creation (once approved by the security officer)
4) Network access requests (approved personals)
5) Scheduled system maintenance Tasks
6) Audit report generation
7) Out queue management
8) User library maintenance and deletion of unused files
9) Maintaining the password policy

Data Center Manual Version 2.0 32

Contact Details / Escalation Matrix

Emergency phone numbers of the Data Center

Plastoworld Hub phone Numbers

022 27780966
022 30374303

E-Mail – itsupp_india@Plastoworldworldwide.com
Phone – 022-306 47101 - 108

Escalation Matrix
First Level –

Chirag Sheth (System Administrator)

Phone – 09324288852
E-Mail – csheth@Plastoworldworldwide.com

Yogesh Satav (Network Engineer)

Phone – 09324526820
E-Mail – ysatav@Plastoworldworldwide.com

Second Level –

Sudhakar Dongre (Operation Manager)

Phone – 09322866068
E-Mail – sdongre@Plastoworldworldwide.com

Deepesh Seth (Asst. Network Manager)

Phone – 09324526934
E-Mail – dseth@Plastoworldworldwide.com

Pushpeel Andheri (Sr. Network Manager)

Phone – 09323109421

Third Level –

Ramesh Thorat (Data Centre Head)

Phone – 09324526933
E-Mail – rthorat@Plastoworldworldwide.com

Plastoworld Cage Number – 022 30374169

Data Center Manual Version 2.0 33