Table of Contents
INTRODUCTION 4
Complete ICT System Analysis
1.
General 6
1.1 Critical Items
7
1.2 Unsatisfactory
8
2.
The Audit Process
2.1 Information Technology
2.1.1Administrator Access
Recommendation
2.1.2System Administrator
Recommendation
2.1.3Database Administrator
Recommendation
2.1.4Network Administrator
Recommendation
2.2
10
11
11
12
12
13
14
15
22
24
Recommendation 25
2.2.1Internet Access
25
Recommendation 25
2.2.2SAN (Storage Area Network)
Recommendations29
2.2.3Server Room
29
2.2.4Security
29
Recommendation 30
2.2.4.1
RIF
31
2.2.4.2
Mundo Nobo 32
2.2.4.3
Nieuwe Haven
32
2.2.5Backup Procedures
33
Recommendations33
25
2.3
2.4
Software Applications 36
Recommendations34
2.4.1Critical Systems
36
2.4.1.1
VIS2000
36
Recommendations37
2.4.1.2
Pagatinu
38
2.4.1.3
Decade
38
2.4.1.4
Infor/CAMMS
39
Recommendations39
2.4.1.5
SCADA
41
Recommendation 41
2.4.2ICT Monthly Budget Report
43
Page 2 of 44
BSD -
BSD -
INTRODUCTION
FTI Consulting, Inc. (FTI) was retained by the supervisory board of Integrated
Utility Holding N.V. (Aqualectra or the Company, Client or the Board) to
perform a forensic analysis in order to detect any inefficiencies and or irregularities
within the operations of Aqualectra. FTIs work, to be performed under the direction
of the Client, was to deploy well-qualified staff appropriate to the tasks assigned by
the Client, including personnel such as IT experts, to perform such tasks as may
have been identified during the course of this Engagement with the understanding
that someone from FTI may be called upon by the Client to provide expert
testimony in the future. The Client and FTI will discuss and mutually agree on which
FTI professional would serve as an expert.
All professional conclusions are those of the FTI professionals working on this
matter.
Client will make available to FTI the documents, if necessary, for FTI to
complete the analysis. FTI performs all engagements in a legal and ethical manner
and adheres to all applicable privacy laws.
Switches
Firewalls
Routers
Servers
All other network devices
Page 3 of 44
BSD -
Infor
Decade financials
HRM Systems
VIS2000
SCADA
Mail server
Telephone Exchange
Complaint centers ( Klachten afdeling)
Other operating support systems
BSD -
This review is a dynamic process and we reserve the right to amend our findings,
recommendations, opinions and report based on comments and/or information
received subsequent to this preliminary report.
Also, please note that this report is confidential and will be issued in final form only
to the Supervisory Board by FTI. FTI takes no responsibility for the dissemination of
any or all parts of this report to parties other than the Supervisory Board or any
consequences thereof.
1. General
FTI was contracted on August 4, 2011, to investigate concerns regarding the
Information Technology Department at Aqualectra. Our report findings address
critical and unsatisfactory issues regarding the internal Aqualectra Information
Technology Departments infrastructure. There are additional concerns regarding
the infrastructure, as well as various application concerns, both in contract (SLA)
and physical access that require immediate attention. These red flags will be
reviewed in the following audit report.
The initial investigation request needed to be slightly broadened in order to avoid
oversight of a parochial nature in that mere issues of network security might only be
addressed. The team found several unanswered questions regarding information
protection, privacy governance, and information integrity that needed to be asked.
The following taxonomy categorizes the findings into two areas impacting: Critical
(regulatory and compliance issues/high risk to business materiality); Unsatisfactory
(not as severe as critical, but needing to be addressed within the next six months);
1.1
Critical Items
The following critical points were identified during the audit of Aqualectras IT
infrastructure:
Page 5 of 44
BSD -
Employees on the network and remote access servers are never verified with
currently active employees.
Encryption levels used on the firewall is set to the lowest admissible level.
The encryption key set on the firewall for the Radius servers can be hacked
using brute force attack.
Routers and the Firewalls IOS software are outdated and exposed to DOS
(Denial of Service) attacks.
Access list on the firewall is configured to allow any source from external
entities.
Password disciplines were inadequate and did not meet industry best practice
standards leading to potential security breaches on routers and firewalls.
BSD -
Single exchange database for all mail users creates a risk for quick recovery
in case of failure. No redundancy.
1.2
Unsatisfactory
The following unsatisfactory points were identified during the audit of Aqualectras
IT infrastructure:
SAN is approaching End of Life by end of year with no support. Space is also
limited and will become critical by end of year.
There is no central reporting system; each system generates its own reports.
Oracle Databases on key revenue applications are out of service and not
supported.
None of the critical applications interacts with other applications other than
batch files or scripts. INFOR and DECADE currently interact with each other
on some levels.
Page 7 of 44
BSD -
Meter usage files collected with handheld recording devices are not
encrypted and accessible to everyone with access to the network directory.
Physical controls and logical controls need to be verified and tested for
compliance.
RIF server room is almost at full capacity and will be an issue for future
growth.
Three of the four critical applications do not have any interaction among
themselves other than manual entries made by employees.
There are no camera systems in any of the three server locations - RIF,
Mundo Nobo and Nieuwe Haven.
Page 8 of 44
BSD -
2.1
Information Technology
BSD -
Our audit consisted of meeting with critical departments under the direction of the
IT manager for which we selected the following individuals to be interviewed:
System Administrator:
Julius Griffith
BSD -
Julius Griffith
Moraima Matilda
Ludwin Henriette
BSD -
SQL installations
highly
recommends
the
introduction
Administrator/Database administrator.
of
an
in-house
Junior
Systems
nature to provide redundancy for both the Systems Administrator and the Database
Administrators job responsibilities.
Page 12 of 44
Administrator
would
immediately
be
cross
BSD -
trained
by
both
the
Systems
person would immediately fulfill the redundancy need in the area of job
responsibilities for both the Systems Administrator and Database Administrator
ultimately remedying the systems responsibility back-up needed in these two areas.
2.1.3 Database Administrator
Joep Koyen is employed by Aqualectra through the outside consulting firm called
SQL Integrators. He has worked for SQL Integrators for 12 years, including four of
those years with Aqualectra. Koyen is mainly responsible for the administration of
the Oracle databases for the following key applications:
Vis2000
Infor
Decade
Based upon our preliminary interviews and observations, we have determined that
Koyen is an essential employee to the Aqualectra organization.
He is also very
knowledgeable and respected by his peers, but is also a concern for the
organization. Like Griffith, Koyen is also considered a one man show and a
potential risk to the organization should anything happen to him.
The entire management of the Aqualectra database is controlled by one person.
Although it was mentioned that SQL Integrators can send another individual in case
something were to happen to Mr. Koyen, it has come to our attention that no one at
SQL Integrators is as knowledgeable as Koyen.
modification came to a complete halt when Koyen took a leave of absence for two
months. During this time, no additions or enhancements to the system were done.
The management of the Aqualectra database is dependent solely on Koyen with no
assistant or backup to provide redundancy in his absence.
Page 13 of 44
BSD -
Recommendation
FTI
highly
recommends
the
introduction
of
an
in-house
Junior
Systems
would
immediately
be
cross
trained
by
both
the
Systems
Page 14 of 44
BSD -
The graphs below show the infrastructure of the Oracle databases for several
applications:
Page 15 of 44
Page 16 of 44
BSD -
Page 17 of 44
BSD -
Page 18 of 44
BSD -
Page 19 of 44
BSD -
Page 20 of 44
BSD -
Page 21 of 44
BSD -
BSD -
observed
them
having
difficulties
troubleshooting
issues
and
properly
diagnosing the problems. Both of these users have administrative rights for the
entire network, as well as for all routers and firewalls.
This department manages the addition and deletion of users to the network.
However, we noticed that no one monitors or is held accountable for all users on the
network. If this department is never notified of any deletions from the system, the
user will be kept in the system with the same rights and access privileges until
notified. During our interviews we questioned the monitoring of the routers and
firewalls for possible penetrations. However, from the interviews it became obvious
that monitoring of the routers and firewalls for possible penetrations is nonexistent
for the entire Aqualectra network Infrastructure.
Page 22 of 44
BSD -
Page 23 of 44
BSD -
Recommendation
The network administrators respective knowledge base should be evaluated to
determine the areas in which deficiencies exist.
The network has been built over many years into a collection of equipment
providing network services on a best effort basis. It is anticipated that the demands
being placed on the network by new business initiatives would force more and more
changes into the network and would demand service levels beyond the capabilities
of the current infrastructure. Inconsistent implementation over an extended period
of time allowed the network to develop into an unpredictable platform that is at risk
during any change or failure event.
Working in collaboration with the Network Support and Management team, our
Audit Team followed a defined process to discover all of the available relevant data.
The results from the data discovery exercise were then processed and analyzed for
all logical and physical network topology diagrams produced.
Page 24 of 44
BSD -
Another major work stream was to examine the principles applied to the overall
network architecture and design templates and measures based on the industrys
known best practices. This exercise was also able to drive out the levels of network
resilience and determine how the architecture would cope with the business
continuity planning requirements.
Recommendation
Please refer to Network Analysis Report
2.2.1 Internet Access
The Internet connection for Aqualectra is provided by NetPro through wireless
connection and a redundant landline DSL provided by UTS.
The connection
provided by NetPro is a 6 MB wireless connection to the RIF office and a firewall has
been setup at the RIF location as a central point of entry. All VPN access into the
system is made through this connection and verified against the user ID and
password provided. Members of the VPN group are provided access to the internal
network.
Recommendation
Internet connection must be re-evaluated since the 6 MBs is insufficient with the
growth of Aqualectra. Further analysis must be made to determine whether a
wireless connection should be the redundant system instead of the main landline
connection. We are also concerned with the VPN access since the list of users is not
being verified against currently active employees.
2.2.2 SAN (Storage Area Network)
Aqualectra currently has three SAN environments and is monitored by EMC and by
local EMC support. One EMC, of CX500 13 Terabytes raw data, is located at the RIF
location and supports a CX300 located at Mundo Nobo and Nieuwe Haven. The
Page 25 of 44
BSD -
current RIF SAN is approximately 79.5% full. The detail of the storage of the SAN is
as follows, reported in GBs:
VIS2000
187
Infor
424
400
Decade
224
Decos
235
File Servers
1,500
Virtual Servers
2,207
752
Total
5,928 GB
Page 26 of 44
Page 27 of 44
BSD -
Page 28 of 44
BSD -
Recommendations
Page 29 of 44
BSD -
BSD -
After preliminary review of the SAN infrastructure, we have determined the SAN
device must be purchased before December 2011. The current system is reaching
EOL with support terminating in December 2011.
The server room located at the RIF location is small and close to
outgrowing Aqualectras computer needs in the future. The HVAC system installed in
the server room is a regular non-monitored temperature system that does not
regulate humidity vs. temperature.
2.2.4 Security
Keeping updated with service packs, patches, and locking down systems can only
provide limited security. Every security plan needs to address the physical security
of the hardware by housing and passing the data through the infrastructure.
Keeping up with service packs and patches from different vendors will not prevent
an individual from walking inside the server room and ruining and/or stealing the
hardware.
First, to establish security a strict policy should be implemented to allow access to
the server room to only those employees who need to be in or enter the room.
There are three individuals who have access to the server room, who are Julius
Griffith, Moraima Matilda and Ludwin Henriette. The server room at the RIF location
Page 30 of 44
BSD -
is the only location protected by a keycard entry system. There are no camera
systems installed at any of the locations to monitor entry to the server rooms.
Recommendation
The RIF server room must be expanded or located within a computer room and/or
area that offer the potential for future expansion. Even though technology changes
tend to make hardware more space-efficient over time, the ability to expand, either
within the current footprint of the building or through additions, should be available
to accommodate possible growth as the room evolves.
Install at all three server locations - RIF, Mundo Nobo and Nieuwe Haven - a camera
system to detect entry into any of the three locations. Additionally, at the At Mundo
Nobo and Nieuwe Haven locations we recommend installing a card entry system
and removing the glass panel located next to entry door.
since the glass can be broken and the server room accessed. Also, At the Nieuwe
Haven location, we did not find any power regulators or Uninterruptable Power
Supply systems. While we understand that the power obtained at this location is
uninterruptable, the sensitive equipment should be plugged in to a voltage
regulation system.
Likewise, another aspect of security is the installation of devices to protect the
investments made, such as fire detection and suppression systems and water
detection systems. The physical environment of the server rooms should be
rigorously controlled. Air conditioning is used to control the temperature and
humidity in the server room. ASHRAE's "Thermal Guidelines for Data Processing
Environments recommends a temperature range of 1624 C (6175 F) and
humidity range of 4055% with a maximum dew point of 15C as optimal for data
center conditions.
Page 31 of 44
BSD -
2.2.4.1 RIF
The following pictures show the key access entry to the RIF server room location:
Page 32 of 44
BSD -
BSD -
Aqualectra uses Symantec Backup Exec to backup all of its servers and the SAN
environment on a daily basis. Backups are run daily for all Oracle databases, SQL
Databases, Virtual machines, User Files, Exchange database, Intranet Groups and
Share drives. Operating systems and applications are backed up once a week. On
Sundays, a full backup is run at night. All Oracle databases are compressed and also
backed to another server in another location as a secondary backup to tape. All
backups are stored on HP LTO four tapes. Daily backups are stored on four tapes
and weekly backups are stored on 7-8 tapes. All tapes are encrypted and passwords
are maintained by the three local administrators mentioned before.
Backup rotations are as follows:
Recommendations
Yearly backups should be maintained and stored offsite with the daily backups.
Monthly backups should also be maintained and Two months should be kept onsite
in a secured location and the rest should be stored off site. We highly recommend
retrieving certain files from a subset of tapes to verify their integrity. Backup logs
need to be reviewed and checked so critical servers are never discarded or ignored.
Page 34 of 44
2.3
BSD -
During our interviews and meetings with management and staff, we have found
Aqualectra does not have a Disaster Recovery Plan or a Business Continuity Plan. A
backup and Disaster Recovery Plan dating back to 2006 was provided to us,
however, the report does not reflect the logistics nor the procedures to be followed
if such disaster should occur at Aqualectra. In the report, there is no mention of
data being stored off the island in case of a disaster. In the current environment if a
disaster should occur to the island, all data would also be lost without possibility of
recovery since all the data is stored on the island.
A Disaster Recovery Plan (DRP) should define the resources, actions, tasks, and data
required to manage the business recovery process in the event of a business
interruption. The world is changing and organizations need to prepare for natural or
manmade disasters that could disrupt business processes. Customers and millions
of dollars could potentially be lost and never recovered if business processes are
disrupted, IT systems exceed their Recovery Time Objective (RTO), or data exceeds
the Recovery Point Objective (RPO).
and
critical
business
processes
depend
on
technology
BSD -
Page 36 of 44
2.4
BSD -
Software Applications
Vis2000-
Pagatinu-
Infor-
Decade-
Financial system.
Aqualectra has two methods of selling electricity to consumers, through post paid
sales (VIS2000) and prepaid sales (Pagatinu).
2.4.1.1 VIS2000
The VIS2000 application was implemented in 2000. The application was developed
by Aqualectra with the assistance from the outside consultants SQL Integrators. The
program is written in Oracle with an Oracle database. This application is maintained
by the consultants SQL Integrators and Joep Koyen as the database administrator.
During our interviews with the IT staff at Aqualectra, a discussion of the database
structure of the VIS 2000 system let us to further investigate this matter. The VIS
2000 Oracle database, which stores all of the VIS 2000 data, is running with version
10.1 and is no longer supported or maintained by Oracle.
Recently the company encountered a problem with the index optimization of the
Oracle 10.1 database, which disabled the entire system for an entire day. This is a
major concern since the VIS 2000 system is extremely vital financially to the
organization and at any point in time this incident could happen again causing the
system to fail. It is our concern as to why the IT department has not upgraded to the
latest version of the Oracle database 11.xx that includes the latest patches and
Page 37 of 44
BSD -
support. There is no additional cost in licenses since the IT department has been
paying yearly maintenance fees for years now, which include free upgrades to the
newest Oracle licenses. An upgraded Version 10.2 has been available in the market
since July 2005.
Additionally, the system is encountering various other errors. This is taking an
additional amount of space on the SAN and the entire environment must be
continuously cleared manually. The Oracle administrators are concerned with the lack
of support from Oracle since this product is no longer supported. While we
understand that the VIS2000 core system is also outdated and must be replaced, it is
imperative to upgrade the database structure immediately to allow Aqualectra to
comfortably extend its research until the new system is implemented.
Meter reading is also entered into the VIS2000 system through batch files.
Once a
meter reader enters the usage information into their portable device, a text file is
created at the end of day and stored on the network. This file is Unencrypted.
Recommendations
We recommend further analysis with the current database administrator and VIS2000
developers to determine the effects of upgrading the database with a minimal impact
on the source code and related programs. The key problem we see in the upgrade is
that the scheduling program associated with the VIS2000 must also be updated.
Several scheduling programs have been reviewed, but no decision has been made
yet. Once we determine the impact this upgrade will have on the VIS2000 and related
applications, we can begin developing an implementation plan for the database
upgrade.
evaluating all the security and group rights in VIS2000. These rights are independent
of those from the network.
BSD -
2.4.1.2 Pagatinu
Pagatinu is Aqualectras pre paid system for consumption purchases. Pagatinu is a
third party application created and maintained by Conlog. The application is also
supported by Conlog, which is located in South Africa. Clients can purchase amounts
of consumption from local branches, MCB Bank merchants and or various other
merchants throughout the island.
When a customer purchases credits through a point of sale system or any
participating merchant, it connects with the Pagatinu system and sends the terminal
number and the amount requested. Based on the terminal number, amount, date
and time, a 20 number algorithm is provided to the client from the Pagatinu system.
That number is entered into the clients terminal and the amount purchased is then
available for the client.
All sales through the Pagatinu system are reconciled with bank reports and then
sent to Decade (the financial system). There is no connectivity between the
Pagatinu system and any other system at Aqualectra. Every entry in VIS2000 is also
entered into Pagatinu. All financial information is provided to accounting through
reports.
2.4.1.3 Decade
Decade is the financial system utilized by Aqualectra for all of its financial,
accounting and General Ledger reporting.
consultants SQL integrators and is also purchased through a third party vendor. The
critical applications like VIS2000 and Pagatinu have no interface with the Decade
system and financial information is entered manually through various spreadsheets
exported from Vis2000 or Pagatinu.
Page 39 of 44
BSD -
The financial and accounting departments are provided with daily or monthly
spreadsheets listing single General Ledger and sales numbers to be entered
manually into the decade system. No account information of the client is maintained
in Decade.
On the other hand, Infor has an interface with Decade. When Infor produces
purchase orders, inventory, work orders, etc., the information is transferred to
Decade to maintain control of items purchased and items to be paid. Infor in the
only program that interacts with Decade through an automated interface system.
VIS2000 and Pagatinu send information monthly to accounting through various
spreadsheets and reports that are manually entered into system.
2.4.1.4 Infor/CAMMS
CAMMS Computerized Asset Maintenance and Management System handles assets,
work orders, inventory, purchase orders, etc.
Database supported and maintained by SQL Integrators Joep Koyen as the database
administrator.
The Infor application among other things is utilized for work orders,
purchase order and inventory control. The only interface between Infor and VIS2000
is just for work orders.
monitored.
Purchase orders are issued by Infor and matched against the invoice,
about the system is the lack of processes between the distribution and production in
which each area works independently.
BSD -
entered into the decade system from all other critical applications. Aqualectra must
focus on replacing the existing critical systems with a cross platform system
integrating all 4 critical systems into one application.
together information from all data sources within an organization (and where
appropriate, from outside the organization) to give one, holistic view of each
customer in real time. This allows customer facing employees in such areas as
sales, customer support, and finance to make quick yet informed decisions.
Application owners and application managers are not well defined in the processes
of the applications. In some cases the application owners and application managers
are de-centralized and many decisions would be delayed due to lack of
documentation or knowledge.
We highly recommend evaluating all the security and group rights in VIS2000, Infor,
Decade and Pagatinu. These rights are independent of those from the network.
The following graph shows a summary of all databases and current versions
Page 41 of 44
BSD -
2.4.1.5 SCADA
The SCADA (Supervisory Control and Data Acquisition) is a system that collects data
from various sensors at the electrical and water plant and sends this data to a
central computer that then manages and controls the data. The SCADA software
runs on four servers, of which two are online and two are backup running on
Windows servers 2003. The information server collects information from the data
collectors and sent to the servers by 13 electrical and 15 water substations. Many of
the electrical components can be controlled by the SCADA server, but the water
collectors are only for monitoring purposes.
The SCADA network utilizes the same fiber ring as Aqualectra, but is protected by its
own firewall and router.
administration. VNC and Net meeting are used by outside vendors to access the
system for maintenance and troubleshooting.
Recommendation
The SCADA network is in a highly sensitive environment where the electrical plants
can be controlled by the software. It is crucial that this network is isolated from the
rest of the organization and treated as such. Currently the network is also being
managed by the same company that provides 100% of the Aqualectra network
NetPro. It is our recommendation to evaluate this network thoroughly and provide
a vulnerability assessment on the entire SCADA network.
Page 42 of 44
BSD -
The following table shows a summary of all applications and their respective
owners:
Page 43 of 44
Page 44 of 44
BSD -