DRAFT IA Audit Report On RM Post Implementations With Changes New

NOTE: NAZIA’s changes are in blue and the items I believe
ought to be removed are in pink
Internal Audit Department

Calendar Year: 2017
Audit Report No. 3 & 4/ 2017
POST IMPLEMENTATION REVIEW OF

THE NEW ERMS
October 2017
A product of the Internal Audit Department

1
|Page
Stage Name Title Signature Sign Date
Accepted By: Mrs. Hazel Persad Director of Corporate
Services Division
Accepted By: Ms. Curlene James Manager of Information
Management
Department
Prepared By: Ms. Mary Hussein Manager of Internal
Audit Department
Overall Report Risk Rating Summary
Risk Rating Low Medium High
No. of Audit Issues: 12 2 7 3
Note: The following graphics and rating represent IA’s evaluation for the
risk levels of the issues and recommendations that were identified.
RISK SCALE KEY
The issues, risks and recommendations identified during this audit

have been rated and colour coded in accordance with the following
definitions.
Low Medium High

2
|Page
•The issue represents an opportunity •The issue is a control deficiency •The issue is a control deficiency
to improve control and processes to which represents a gap in the which represents a significant gap
support the achievement of desired design and/or operating in the design and/or operating
outcomes. effectiveness of the control affecting effectiveness of the control affecting
•The issue should be addressed the Commission's ability to address the Commission's ability to address
promptly, as time and resources relevant risks and provide relevant risks and provide
permit. reasonable assurance regarding reasonable assurance regarding the
the achievement of desired achievement of desired outcomes.
outcomes. •The issue requires an immediate,
•The issue requires prompt comprehensive, corrective action
attention to ensure the internal plan with progress to be monitored
control is designed and/or by an appropriate level of
operating effectively. management.
Considerable professional judgment is required in applying the
ratings defined and used in this report regarding individual issues
and recommendations. Accordingly, others could rate the issues and
recommendations differently and this should be born in mind when
considering this report.
Table of Contents
1.0 EXECUTIVE SUMMARY4

2.0 OBJECTIVES AND APPROACH6
3.0 SUMMARY OF FINDINGS (Arranged accordingly from High to
Low Risk)8
4.0 DETAILED REPORT: PART ONE (1)9
17.0 DETAILED REPORT: PART TWO (2)41
53
54

3
|Page
1.0 EXECUTIVE SUMMARY
1.1 An Enterprise Wide Risk Assessment of the Trinidad and

Tobago Securities and Exchange Commission (TTSEC)
identified the Post Implementation Review of the
Commission’s Electronic Records Management System
(ERMS). The launch and implementation of the new ERMS
was a collaborative effort of the Records Management
Division of the Corporate Services Division and the
Information Management Department, as one of the higher
risk areas.
1.2 Management has recently replaced the existing electronic

records management software referred to as Versatile with a
4
|Page
new Electronic Records Management System (ERMS), an
application developed by Newgen Software Technologies
Limited. Notwithstanding the notable delays in the
implementation process due to the customization of the
system, the new ERMS went live on March 14, 2016.
1.3 A post implementation review was conducted by Internal

Audit Department (IA) to ascertain an understanding of the
efficiency and effectiveness of the implementation and the
operation of the new ERMS.
1.4 It was evident from the review that the change from the
previous records management software, Versatile to new
records management software was justified. The weaknesses
of the versatile system was noted to be three (3) times more
than strengths of the system. However, a high level of
subjectivity was evident in the evaluation process in the
allocation of points due to the unstructured approach used
which would have hindered application of the criteria
uniformly. The Commission needs to assess potential
contracts on the principles of value for money, transparency
and accountability.
1.5 IA’s assessment of the ERMS project identified that Value

for Money was not achieved. The ERMS has cost the
Commission almost twice as intended for a product that has
not completely met its functionality requirements, in terms of
quality of service provided by the software and has impacted
the resources of the Commission, as it took about three and
half times the projected timeframe for its implementation.
This negative impact on resources is continuous due to the
added labour intensive demand on the RM division and
continuous cost being incurred by the Commission related to
treating with ongoing system errors.

5
|Page
2.0 OBJECTIVES AND APPROACH
2.1 The primary audit objectives of this review were to: Commented [NA1]: Make sure it ties to the detail objs in
the respective section
i. Evaluate the accomplishment of the project objectives and
determine the efficiency and
effectiveness of the operations of the new system;
ii. Determine the existence, accuracy and completeness of
records transferred and maintained; and
iii. Ascertain whether value for money was achieved in the
changeover to the new system.
2.2 The secondary audit objectives of this review were to:
i. Assess the adequacy and effectiveness of the process for the

evaluation of bids in the Award of Contract; (1)
ii. Determine the adequacy and completeness of the delivery of
the requirement needs listing as per Contract / Project after
implementation; (3)
iii. Ascertain the accuracy and completeness of the data
transferred during the migration from previous Versatile
system to the new ERMS;(4)
iv. Assess the efficiency and effectiveness of the upload of
information from hard copy files to the ERMS ensuring
timeliness and accessibility to end users;(5)
v. Assess the adequacy, effectiveness and completeness of the
process to rectify errors that arose during the project phases
and encountered by users; (6)
vi. Ensure proper authorization over the access rights of the
system in order to preserve the integrity of confidential data
and assess the adequacy and effectiveness of the
dissemination of information (how and to whom);
vii. Assess the effectiveness of access security log in credentials
preserving who accesses the system and its alignment to best
practices;
viii. Assess adequacy of Backup and Disaster Recovery

procedures and ascertained whether or not they have been
formalized, documented and tested; (2)
ix. Assess the adequacy of training of Records Management
Staff and the adequacy of reference documentation which
exists to support efficient use of the system and assess the
6
|Page
adequacy of training for IM staff in maintaining and
supporting the system. Additionally, to assess the adequacy
of the training for the staff of the Commission in the use of
the ERMS;
x. Assess the adequacy and effectiveness of the evaluation form

to obtain feedback on the weaknesses of the system and use
of such feedback and determine whether evaluation form was
within best practices;
xi. Ascertain completeness of records data by tracing the
existence of the missing files from the Versatile system
determined in previous audit (Registration of Registrants and
Securities) can now be traced to the new system; and
xii. Ascertain whether value for money was achieved in the

changeover to the new system. Determine the extent to which
funds were expended economically and efficiently and the
extent to which the related project was effective in meeting
its objectives.
 The Economy - To ascertain whether acquisition of the

Newgen ERMS was the appropriate system with the quality
and quantity at the lowest reasonable cost.
 Efficiency - To investigate whether the level of service

provided with the Newgen ERMS has improved the
Commission’s Service (Internal and External) at no
additional cost.
 Effectiveness – To determine if the goals and objective of the

ERMS are being achieved (the attainment of the right results
from the usage of resources & organizational operations).
2.3 In conducting this review, IA held discussions with the

Director of the Corporate Services Division, the Manager of
Information Management and their core staff members.
Additionally, IA reviewed the following:
i. The Commission’s new ERMS; and

7
|Page
ii.Documents pertaining to the tender evaluation and selection,
implementation and
functionality of the ERMS.
3.0 SUMMARY OF FINDINGS (Arranged accordingly from

High to Low Risk)
Ref # Recommendations Page Risk level

Number
5.0 8
Error! Reference source not found.

6.0 12
PROPER BACKUPS TO BE MAINTAINED
7.0 Error! Reference source not found. 15
COMPLETION OF THE TRANSFER OF

8.0 18
UNMIGRATED PORTION OF DATA
IMPROVEMENT IN THE USE OF SYSTEM

FOR INPUT AND RETRIEVAL OF
9.0 19
DOCUMENTS

8
|Page
Ref # Recommendations Page Risk level
Number
FINAL ERROR LOG TO BE AGREED UPON,
APPROVED, AND COMMUNICATED TO
10.0 22
NEWGEN
FURTHER GUIDANCE TO STAFF ON ERMSError!

11.0 25
Reference source not found.
OPTIMISE EFFECTS AND USE OF RESPONSES

12.0 27
FROM EVALUATION FORMS
UPLOAD OF DOCUMENTS IN THE ERMS THAT
WERE NOT RECORDED IN VERSATILE AS
13.0 29
HIGHLIGHTED IN PREVIOUS AUDITS
14.0 30
15.0 32
4.0 DETAILED REPORT: PART ONE (1) OPERATIONAL

AND FUNCTIONALITY
5.0 IMPROVEMENT IN THE TENDER EVALUATION

PROCESS FOR THE AWARD OF CONTRACT
Objective # 1: To assess the adequacy and effectiveness of the

process for the evaluation of bids in the Award of the Contract.

9
|Page
5.1 Compliant Controls
5.2 IA reviewed the document labelled Design, Supply and

Installation of an Electronic Records Management
Software (Guide to the Evaluation of the Proposal
Documents) and it was noted that the Commission utilised a
selective tendering method of five (5) vendors and utilised
established criteria in the evaluation of bids. NewGen was
successful over the two (2) other responsive bids received
from Digidata and Knowledgelake.
5.3 Findings
5.4 The above mentioned document which was equivalent to an

evaluation report was unsigned and there was no indication
of the evaluators of the submitted proposals nor the
preparer(s) of the document. However, the project team
members were listed on page 3 of the said document.
5.5 It was noted that the document sub-titled ‘Guide to the

Evaluation of the Proposal Documents’ also included the
findings for the actual evaluation of the bids. On page six (6)
of the said document was listed the deliverables which
mostly matched the criteria as shown on page 7 that was used
in determining compliance with RFP and awarding points.
5.5.1 Even though a ‘schedule of project activities Milestones and

deliverables at each phase’ was stated as a requested
deliverable, timeline for delivery was not used as a criteria
for the awarding of points.
5.5.2 Even though a ‘profile of the project team members’ was

stated as a requested deliverable, the category for experience
in the criteria only catered for ‘relevant experience with
similar contracts’ and did not consider nor awarded any
points for the experience of project team members.
10 | P a g e
5.6 It was observed that there was also no detailed breakdown of
the criteria and an explanation of how the total points would
be allocated for each category.
5.6.1 For example, under the category ‘Relevant Experience with

similar contracts’, the evaluation failed to provide a basis
used for awarding points and the requirement which would
have to be met for one bidder to score the total points, the
minimum points or less than minimum points.
5.6.2 Another example, under the category ‘Functionality and

Scalability’, each of the bidders was awarded equally 20
points for meeting 28 of the 29 items specified under this
category. Two (2) of the companies were unable to provide
the same item. Thus, it was unclear how the 29 items
corresponded to the maximum of 25 points, as for missing
one (1) item, each company lost 5 points. It should be noted
that points cannot be awarded to bidders for items not
requested in the bid specification nor captured in the RFP.
Thus, a high level of subjectivity was evident in the

evaluation process in the allocation of points, which would
have hindered application of the criteria uniformly.
5.7 The evaluation process did not cater for the utilisation of
individual score sheets to support each evaluator’s scoring of
the individual bids based on the established criteria. Thus,
absent from the report namely Guide to the Evaluation of the
Proposal Documents was an appendix of these individual
score sheets.
5.8 There was a discrepancy observed in the weighing of the cost

of proposals and the allocation of points. Under the ‘Cost of
the Tender’ category out of a possible 40 points, Newgen
was awarded 35 points, which was 5 points greater than
Knowledgelake whose bid price was only TT$71,558.77
11 | P a g e
higher as compared to Onbase (Digidata) which was
allocated 10 points less than Knowledgelake for a bid price
three (3) times greater than Newgen.
5.8.1 IA observed that page 12 of the ERMS Project Closure

Report, it indicated a project budget of TT$1,641, 195.00.
Thus, both Newgen and Knowledgelake bided below the
budgeted estimate for the project. Thus, there was no
reasonable basis used by the evaluators in awarding points
for cost as the ratio of points allocated was not consistent
with the actual bid prices.
5.9 Risks
5.10 Lack of transparency and uniformity in evaluation process

due to identified and unsupported basis used in the allocation
of points.
5.11 Overruns in budgeted cost and estimated timelines when a

holistic and value for money approaches not taken into
consideration in developing criteria and the application of
such criteria during an evaluation.
5.12 Recommendations
5.13 It is recommended that when conducting the tender

evaluations for the selection of a bidder in the award of a
contract, there must be proper evidentiary, supportive
documents to corroborate the scores awarded during an
evaluation. Supportive documents such as individual
mark/score sheets must be utilised, signed off and attached to
the evaluation reports.
5.14 The Commission must ensure that evaluations are carried out
in accordance with prior approved and or predetermined
criteria and in a manner that ensures strict compliance with
Section 24 (1) of the Integrity of Public Life Act 2000 (see
12 | P a g e
Appendix 1). These written evaluation criteria must be
established by the Executing Unit prior to the closing date
(submission deadline date) of the Bids, and must be
uniformly applied and may not be changed after opening of
bids. To ensure the criteria is uniformly applied, the
Commission should ensure a detailed breakdown of each
category of the criteria exists and an explanation of the
requirement of how points should be allocated.
5.15 A proper justifiable basis must be agreed and consistently

applied by the Commission for the weighing of points to be
allocated to the cost of proposals. A company estimate of the
contract cost should be prepared by a technically competent
officer and approved by the Tender Committee. In addition,
the role of the company estimate in the evaluation process
and the ranking of cost proposals should also be established,
approved and consistently applied.
5.15.1 CSD should request from Policy Research and Planning

Division, a policy paper on the best suited approach to be
used by the Commission in weighting cost and awarding
points under the Cost criteria for tender evaluations. In
addition, It is recommended that a guide document should be
developed which would provide consistent procedures to be
followed and the required documents (with samples) to be
maintained when carrying out evaluations of the tenders.
These procedures should be considered by the cross
functional teams responsible for revising and reviewing the
Tender Rules. The recommended method and guide
document should be approved by the Tender Committee as
an addendum to the existing Commission’s Tender Rules.
5.16 A clear distinction must be made between the criteria

document namely ‘guide to the evaluation of the proposal
documents’ which must include a detail breakdown of the
criteria and its application and the actual evaluation report
being an explanation of the tender process and reasons for
13 | P a g e
the ranking and the selection for the award of contract. The
evaluation report should be clearly labelled ‘Evaluation
Report’ and identify the specific project for which it refers to.
5.17 All project reports and documents used in the evaluation

process must be dated and signed off. This adds to the
integrity and transparency of the evaluation process and
allows for accountability within the organisation. All
divisional / departmental heads should ensure strict
compliance with section 4.0 (1.1) Records Management
Policy which requires ALL final documents to be held with
RMD. RMD should ensure that reports received from
various departments and filed with RMD are properly
finalised i.e. dated and signed off by preparer or preparer and
approver as appropriate. Additionally, it can also be
recommended that the RMD send out a quarterly reminder to
divisional heads serving as a gentle reminder to compile with
this requirement.
5.18 Management Comments
5.19 RM -
5.20 IM – IM agrees in the most with the recommendations

presented under this section.
RM Implementation Action
Cross functional team to be charged with revising the Tenders Rules and Guidelines for
evaluation to include templates and formulae to generate the scoring
RM will include at the next revision of the policies and procedures manual, that all reports to
be submitted to the RM t be signed off as final by the designated divisional/departmental head
and/or designated senior officer before being lodged with us. (Note that in so doing we are
14 | P a g e
defining what we mean by final record as opposed to being responsible for acquiring a
signature).
Name & Title Sparkle N. Ferreira Implementation Date 31st March, 2018
of Person
Responsible
IM Implementation Action
N/A
Name & Title Implementation Date

of Person
Responsible
6.0 REMEDIAL ACTION NEEDED TO ADDRESS

WEAKNESSES OF PREVIOUS SYSTEM THAT
WERE TRANSITIONED TO THE ERMS
Objective # 2: To determine the adequacy and completeness of the

delivery of the requirement needs listing as per Contract / Project
after implementation.
6.2 IA reviewed the documents labelled ‘Report on the 2014

Records Management Gap Analysis Survey’, pages 10 & 11
and ‘Electronic Records Management Proposal’ pages 2 and
3, which highlighted issues that provided a justification for
replacing the versatile system. It was noted that some of the
major issues documented in the both reports were:
 “Frozen systems,
 Database queries,
 Getting onto the system,
15 | P a g e
 Unreliable (Frequent crashes),
 Documents in the system not searchable ,
 Use of the ERMS remote desktop (portal) is not at all
reliable”, etc.
6.3 Findings
6.4 IA further tested to ensure whether the sixteen (16 )issues

/weaknesses captured in the ‘Report on the 2014 Records
Management Gap Analysis Survey’, pages 10 & 11 from the
previous ERM system (Versatile) was resolved with the
Newgen ERMS. This was done by using a sample of five (5)
Registrants which was from an earlier audit (Administration
of Registrants) where the IA team did not locate any
document for those registrants. It should be noted that the
files that were missing from the earlier audit were still not
found on the new ERMS and some of the following
issues/weaknesses from the versatile system were still
observed in the Newgen ERMS. These were:
 ‘Difficult to navigate
 Incomplete
 Disorganized
 Inflexible
 Does not meet the current physical and electronic needs of
staff
 Difficulty in accessing the documents.
 Electronic filing of documents needs to be improved to allow
for easier accessibility’.
6.4.1 Therefore, out of the sixteen (16) issues/weaknesses captured

in the project charter, of which ten (10) still was not fully Commented [2]: To be examined by RM
resolved and posed an issue with the Newgen ERMS.
6.5 IA reviewed the TTSEC Requirements Listing as per

Electronic Records Management Proposal pages 10 & 11, to
determine if the Newgen ERM system has accurately
delivered all twenty-nine (29) of the features listed in the
16 | P a g e
document. Additionally, Newgen indicated in its proposal
document to the TTSEC that it could accomplish twenty-
eight (28) requirements as they were unsure about (1)
requirement stated. In addition, it should be noted that IA
found that one (1) item was later determined by TTSEC to
not be applicable. However, ten (10) items of the twenty-
eight (28) requirements were not fully completed by the
Newgen team. Highlighted below are the unsure item, not
applicable item and not completed items:
Unsure item
i. Microsoft Office and all future Microsoft Updates
Not Applicable
i. Ability to read and access the current record structure as has
been created with Versatile
Not Completed
i. Electronic requests for file creation;
ii. Electronic logging of all Incoming Mail;
iii. Electronic management of ALL Mail for Despatch;
iv. Electronic logging & tracking of ALL Interoffice Memos;
v. Logging & Assignment of the receipt and processing of all
External requests;
vi. Barcoding of Files/Box Files/Documents/CD’s;
vii. Full Text Search;
viii. Ability to read and access the current records structure as has
been created with Versatile (import & data entry);
ix. User-friendly Access; and
x. Flexibility of the system (to adapt to change example
possible changes to the Commission’s filing structure)
6.6 Risks
6.7 Loss of valuable production time due to the weaknesses of

the previous system, Versatile being transitioned into the new
ERMS.
17 | P a g e
6.8 Recommendations
6.9 It is recommended that the Commission should approach the

Newgen team to derive a solution to mitigate the effects of
the remaining ten (10) features listed in the Requirement
listing.
6.11 RM -
6.12 IM – Recommendation noted. Commented [NA3]: Action and action date needed as to
when Newgen will be contacted on the same in the space
provided below

of Person
Responsible

of Person
Responsible
7.0 COMPLETION OF THE TRANSFER OF

UNMIGRATED PORTION OF DATA FROM
VERSATILE TO THE NEW ERMS.
18 | P a g e
Objective # 3: To ascertain the accuracy and completeness of the
data transferred during the migration from previous Versatile
system to the new ERMS.
7.1 Findings
7.2 It was noted that items from 2008 to mid-2016 were migrated
to a ‘Versatile Migration Folder’ on the ERMS. A sample of
thirty (30) items were selected from the mail log (task listing)
which were received by RM prior to 2016. IA was able to
trace twenty (20) of these items in the new ERMS, which
was representative of sixty six percent (66%) of the sample.
7.3 Of the twenty (20) items traced to the ERMS, IA noted the
length of time it took to and on average it took seven (7)
minutes per item. Due to this time gap, the end users may
encounter productivity problems which may involve longer
intervals in obtaining information needed for completion of
job deliverables. However, it was noted IA was not a regular
user of the ERMS and lacked familiarity that would have
contributed to ease of use overtime.
7.4 IA reviewed the “Project closure document” and it was

stated that 88% of the data was successfully migrated. This
was also documented and highlighted to NewGen as a
pertinent open, pending issue. Following enquiries by IA, it
was indicated that arrangements have been made by IM for
Newgen to begin an offsite migration of the remaining
records into the ERMS as of Monday July 3, 2017.
7.5 Risks
7.6 Users would spend longer intervals retrieving information

and this would lead to increased unproductive time,
especially in the case of migrated documents.
7.7 Crucial information can be permanently lost with regards to

data not successfully migrated which may lead to
19 | P a g e
reputational risk for the Commission, if required to provide
documents and these documents cannot be retrieved.
7.8 Recommendations
7.9 It is recommended that a collaborative review be done by

RMD and IM, after this current exercise by NEWGEN to
migrate the remainder of data from Versatile, to ensure
whether migration of all data from Versatile was completely
transferred.
7.11 RM -
7.12 IM – In progress
Name & Title Anderson Gyan – Lead Implementation Date T.B.D

of Person Technician Prod. Support
Responsible Anika Noel – Database
Analyst
Migration of remaining Versatile data

of Person
Responsible
20 | P a g e
8.0 IMPROVEMENT IN THE USE OF SYSTEM FOR
INPUT AND RETRIEVAL OF DOCUMENTS
Objective # 4: To assess the efficiency and effectiveness of the

upload of information from hard copy files to the ERMS ensuring
timeliness and accessibility to end users.
8.2 IA reviewed the ERMS to assess whether there was timely

upload of data from the hardcopy to the ERMS. It was noted
that items from mid-2016 to present were directly processed
to the ERMS and from this a sample of thirty (30) items were
chosen. IA with the assistance of RM traced all thirty (30)
items to the ERMS. RM was able to trace documents faster
due to their experience with the software.
8.3 Findings
8.4 IA initially conducted a walkthrough to observe the steps in

processing Registrants’ documents along with Non –
Registrant documents and noted the processing and upload
times. The following observations were made which
contributed to more a manual intensive system in the upload
stage:
8.4.1 Once document was entered onto Omniscan, it had to be

uploaded to Omnidocs where the documents are all then
linked manually; and
8.4.2 Each document was required to be labelled and properly

indexed. This aided to expedite the end user’s work as it
added more details to search engines but also resulted in
lengthy processing times due to the additional fields to enter.
8.5 Of the thirty (30) items tested in the ERMS, IA noted the
length of time it took to find these items. On average, it took
21 | P a g e
fifty one (51) minutes to locate these items resulting in under
two (2) minutes per item.
8.6 Overall, in using the ERMS, IA observed the following

issues as an End user:
8.6.1 It was difficult to search for documents using the registrant’s

name and registration acronym because there were several
different names and acronyms being used.
8.6.2 Additionally, the search function did not work constantly and
was not intuitive. That is, links to batches of documents or
related matters did not always functioned. The search
function did not drill down in terms of the specific matters
being searched for. For example, the search feature by
registrant acronym, was the most responsive feature
however, no search material was retrieved when attempts
were made to search by registrant’s acronym and a specific
key word. Also, the system didn’t contain prompts or related
options that would expedite the search time.
8.6.3 The design of the user interface presented some challenges

for example, to retrieve historical data the search lay out was
monthly and not yearly. Thus it would be time consuming to
retrieve historical data.
8.6.4 The viewing size for documents changed from page to page.
For example, an increase of the viewing size of the record on
the first page, did not automatically adjusted for the other
pages within the document.
8.7 Risks
8.8 Users would spend longer intervals retrieving information

and this would lead to increased unproductive time,
especially in the case of migrated documents.
22 | P a g e
8.9 Crucial information can be permanently lost with regards to
data not successfully migrated which may lead to
reputational risk for the Commission, if required to provide
documents and these documents cannot be retrieved.
8.11 It is recommended that reviews be done by Newgen to Commented [4]: Is this practical though?
improve the retrieval of documents and the ease of use of
information in the ERMS, making it less tedious for end
users and resolve the associated bugs in the system.
Thereby, boosting the overall efficiency of the Commission.
8.12 Additionally, it is recommended that RM staff should be

reminded to utilise the the listing of acronyms that is on the
Commission’s intranet. This would ensure that the entering
and saving of correspondences into the ERMS are
standardised with universal naming conventions. That is,
either saved by the full registrant’s name or by a consistent
acronym. This listing would aid staff in their search.
8.13 The RM team should consider providing further training to

help staff deal with unresolved issues /weaknesses until the
Newgen/IM team find the solution for these weaknesses. E.g.
Shortcuts and tips in finding documents easier using the
ERMS, leading additional assistance to help staff find
documents/files, training and or educate staff in the filing
convention the RM department uses, so staff will have a
better appreciation and understanding on how to locate files.
8.14 Additionally, by rotation of existing staff this can allow

everyone to gain a better understanding of the operational
duties within the Department and therefore reduce
monotonous duties. To facilitate this, a complete revamp of
the exisiting Job Descriptions within the RM division could
be done by the HR Department.Thus rotation between
23 | P a g e
logging, inputting and retrieving staffers could be easily
achieved.
8.16 RM -
8.17 IM –IM remains committed to working alongside RM, to

ensure that the issues experienced are reported in accordance
with the SLA requirements and resolved by NewGen’s
helpdesk team.

of Person
Responsible

of Person
Responsible
9.0 FINAL ERROR LOG TO BE AGREED UPON,

APPROVED, AND COMMUNICATED TO NEWGEN
24 | P a g e
Objective # 5: To assess the adequacy, effectiveness and
completeness of the process to rectify errors that arose during the
project phases and encountered by users.
9.1 Compliant Control
9.2 The process to treat with errors involved an email being sent
to Newgen helpdesk, the issue would be logged and a ticket
number assigned followed by a request for a team viewing
remote session.
IA randomly selected five (5) tickets submitted to NEWGEN
and in all instances tested, there were proper authorisation
granted by the Commission to commence a team viewer
remote session with NEWGEN.
9.3 Findings
9.4 IA observed that there are two (2) error logs. One kept by the
IM Division which contained thirty seven (37) issues of
which thirty two ( 32) closed and two (2) in progress and the
other log by the RM Department which contained forty eight
(48) issues of which thirty seven (37) were shown as open.
There was no coordination between these two lists to
efficiently deal with errors that occurred.
9.5 IA reviewed both logs and randomly selected Opened and

Closed issues to perform subsequent test of details following
a request that RM and IM met to produce one (1) final Error
log of all the real issues as at April 2017. IA reviewed the
unapproved final Error log and it was observed that there
were still discrepancies relating to the current status of some
issues.
9.6 IA observed that the actual dates on which issues were closed
were not recorded on the log, and thus, IA was unable to
estimate an average time it took to resolve the eleven
25 | P a g e
(11)closed issues. In addition, the error log did not contain
the ticket number or an unique sequential identifier number.
9.7 The Project Closure document reflected that the only major
opened issue to date as per IM’s error log was the issue of
Migration. All other open and in progress issues as per RM’s
Error log were not documented.
9.8 IA noted that the Newgen project team was different from
the maintenance team assigned by NewGen. This resulted in
longer time for rectification of errors. Additionally, the
vendor originated in India and this created a language barrier
when dealing with issues.
9.9 Risks
9.10 There is a risk of not accurately documenting and tracking all

real errors when two (2) error logs are maintained
simultaneously.
9.12 Due to the disparity among the initial two (2) logs and the
final unapproved log, it is recommended that IM and RM
embark on a clean-up exercise over RM’s initial Error log to
decipher which issues are actual real issues and ensure that
the same are logged with NewGen for which corrective
measures or solutions can be offered by NewGen, to aid in
remedying the situation. The final approved log should be
updated with any missing issues and pertinent information.
This final approved error log should be agreed upon by RM
and IM and signed off. This should then be approved by the
CEO and sent to Newgen for a timeline for resolving open
issues.
9.13 It is recommended that RM should maintain the error log

and create access by IM, for which, RM should report the
26 | P a g e
issues to Newgen under the new format of reporting issues.
That is, via the use of new web-ex reporting template to
Newgen to be actioned. Also, IM should facilitate the remote
sessions which will be monitored while Newgen is in
working on the system
9.15 RM -
9.16 IM – In prior discussions held with RM, it was agreed that

the maintenance of the error log would be a function of the
RM unit, a mutually accessible location would then be
utilized to store the document, thus allowing IM access to
same. The maintenance of an error log for the ERMS has not
been a function of IM and in this regard, IM disagrees with
recommendation 9.13
Name & Title of Person Implementation Date

Responsible
Name & Title of Person Implementation Date

Responsible
10.0 PERIODIC REVIEWS TO BE DONE OVER ACCESS

RIGHTS
27 | P a g e
Objective # 6: To ensure proper authorization over the access
rights of the system in order to preserve the integrity of confidential
data and assess the adequacy and effectiveness of the dissemination
of information (how and to whom).
10.2 IA reviewed the Security Matrix and Classification Grid for

three (3) random folders within the ERMS. Three (3) random
staff members’ access rights as per each grid were tested
against the systems. It was noted that the sampled tested
revealed that access rights were being adhered to.
10.3 Findings
10.4 However, there were currently no documented periodic

reviews of users and their assigned access rights conducted
for the ERMS. Currently, RM review access rights whenever
a new folder has to be created and when new employees join
the Commission.
10.5 Risks
10.6 Lack of documented periodic reviews would lead to loss of

integrity over the ERMS since Access rights may not be
consistent with a user's current job function.
10.7 Increased risk of unauthorised access through active

application accounts for separated users that are not cleaned
up / deleted.
10.9 It is recommended that Access rights reviews should be

performed at specific intervals (e.g. quarterly, bi-annually or
annually) to detect active accounts belonging to separated
employees and to detect users whose access rights are
28 | P a g e
inconsistent with their job functions. These reviews should
be a joint effort between process owners, the administrators
of the respective application and the IM division.
Documentation of the results of these reviews should be
retained for audit purposes.
10.10 It is also recommended that the Commission’s DRAFT

Operational Risk Manual 3.0 dated 27th March 2015 should
be reviewed, approved, implemented and communicated to
all stakeholders.
10.12 RM -
10.13 IM – Reviewed. Recommendations noted.
Name & Title Implementation

of Person Date
Responsible

of Person Date
Responsible
29 | P a g e
11.0 CONSISTENCY IN INTERVALS FOR CHANGING
PASSWORDS FOR NON ROOT AND ROOT
ADMINISTRATORS
Objective # 7: To assess the effectiveness of access security log in

credentials preserving who accesses the system and its alignment to
best practices.
11.1 Findings
11.2 IA observed that both the end users and RM users ERMS
logins were directly linked to their Windows log in
credentials. Therefore, every time users changed their
Windows password, the login information for ERMS
changed automatically. IA noted from the “Draft
Operational Risk Manual ’’, Section 7.3.3: User Password
Management which stated that “all user passwords must
comply with the management standards specified within this
Operational Risk Manual. General Standards are that
passwords must be changed on a regular basis, based upon
how frequently the password is used. All end user passwords
and non –root administrative passwords should be changed
every forty five (45) days or less. All root administrator
passwords should be changed every thirty (30) days or less”.
11.3 IA enquired from IM the frequency in which both end users

and RM users were required to change their passwords and it
was noted that both were required to change their passwords
every sixty (60) days. Though unapproved, this is not in
keeping with the Draft Operational Risk Manual.
11.4 Risks
30 | P a g e
11.5 If End Users and RM Users passwords are not changing their
passwords at the recommended expiry dates, increases the
risk of unauthorised access to their system through possible
password leakage.
11.7 All administrator accounts belonging to RM Staff should be

set to have a password that expires according to the DRAFT
Operational Risk Manual once this manual is approved.
11.8 All administrative accounts belonging to End users should be

set to have a password that expires according to the DRAFT
Operational Risk Manual once this manual is approved.
11.10 RM -
11.11 IM – In the absence of an approved and updated Operational

Risk Manual, password management is guided by the
policies defined within Active Directory. In this regard, IM
disagrees with the recommendations raised. It should also be
noted that reducing the maximum password age, introduces
additional risk, as the tendency to utilize weak passwords,
store passwords in clear sight, etc. increases. The optimum
password management solution should thus take into
consideration, these varying factors.
31 | P a g e
of Person Date
Responsible

of Person Date
Responsible
12.0 PROPER RECORDS OF BACK UPS TO BE

MAINTAINED
Objective # 8: To assess the adequacy of Backup and Disaster

Recovery procedures and ascertain whether or not they have been
formalized, documented and tested.
12.2 A discussion with IM personnel was conducted on the

Commission’s back up process which revealed that backups
were done automatically on a weekly basis from Sunday
morning to Thursday evening unto five (5) random tapes
based on the availability as the tapes were re-writable / re-
useable as well, back up is also done on Friday and Saturday.
12.3 Findings
12.4 When the TTSEC’s Messenger returns the previous batch of

tapes to the IM Division and it goes to storage and is re-
written. The IM personnel indicated that the information on
32 | P a g e
the ‘Offsite back up storage record of tape movement” form
for the old batch of tapes returned would be verified to the
software generated numbering / coding on the cartridge label.
However, there is no physical evidence on TTSEC’s copy of
the form reflecting that this check was done. Therefore, there
is an issue that there is no signage area on TTSEC’ copy of
the form to reflect that a check was done to verify the codes
of the tapes returned matched to the codes stated on the form.
12.5 KPMG conducted an Information Technology review of the

Commission’s IT systems and procedures in 2015. One of
the areas reviewed were Disaster Recovery and Business
Continuity Planning with the audit being conducted and
completed in 2016. Out of the three (3) recommendations
that was proposed by KPMG, all issues are outstanding to be Commented [NA5]: Nazia to remember to update using
the most recent follow up comments
implemented by the Commission. Of these three (3) issues,
the priority for remediating these areas for improvement for
the said recommendations were all of low importance.
Please refer to Appendix 2 for a Table Displaying the Issues

and Recommendations following KPMG’s review. This table
also reflects IA’s Follow up on the Implementation statues of
each Recommendations along with Management’s comments
for non –implementation on due dates.
12.6 Risks
12.7 Lack of accountability when collecting previous batch of

tapes.
12.9 When the previous batch of tapes are returned to TTSEC by

the Messenger, there must be a signing field created on
TTSEC’s copy of the “Offsite back up storage record of tape
movement “form to reflect a signature for the person
receiving the tapes form the messenger, verifying that the
33 | P a g e
identification details as per the numbering on the tapes is the
same as per the identification details as per the “Offsite back
up storage record of tape movement” form.
12.11 RM -
12.12 IM – .

of Person Date
Responsible

of Person Date
Responsible
13.0 FURTHER GUIDANCE TO STAFF FOR ERMS
Objective # 9: To assess the adequacy of training for Records

Management Staff and the adequacy of reference documentation
which exists to support efficient use of the system and assess the
34 | P a g e
adequacy of training for IM staff in maintaining and supporting the
system. Additionally, to assess the adequacy of the training for the
staff of the Commission in the use of the ERMS.
13.2 Following the walkthrough that was done on the ERMS with
RM users, IA observed that the staff received the training
from the relevant software provider. Additionally, IA
reviewed the ‘User Manual for Process Workflow in ERMS
Solution’ and it was noted that the document was detailed
and it included visual screen shots which made it easy to
follow and understand.
Also it had a step by step approach to scanning, uploading
and indexing documents for both Omniscan and Omnidocs.
13.3 The RM Division created an ‘ERMS Desktop Manual’ in

2015. This document was detailed and also has a step by step
guide throughout the ERMS as it denotes how to search
using many of its in –built search features, such as request a
record directly from the system without having to fill the
user’s mail box and so on.
13.4 IA also reviewed signed training registers showing evidence

of training sessions conducted by RM to the staff of the
Commission’s end users during the period March 14th to 16th
2016.
13.5 Findings
13.6 IA reviewed the training plan from email threads between

TTSEC and NewGen and the Project Meeting Minutes from
the Project Status Weekly report dated 8th July 2015. It was
noted that as per the minutes, there were milestones and
status of the training sections were listed as closed areas.
After reviewing the training plan as well as the NewGen
designed testing scripts, it was stated in the said documents
35 | P a g e
that functional staff were trained on the 25 th and 26th June
2015 and, the end user staff members were trained on the
22nd, 23rd, and 24th June 2015. However, no evidence was
provided by RM as to the individuals trained, whether or not
this included all relevant personnel and who conducted the
training.
13.7 The User Manual for Process Workflow in ERMS provided

by Newgen did not contain any FAQs (Frequently Asked
Questions).
13.8 The User Manual for Process Workflow in ERMS provided

by Newgen did not contain any instances or circumstances
where known user errors can occur and the possible
solutions.
13.9 Risks
13.10 Loss of productivity time, when the User Manuals do not

include FAQs, worked examples of possible errors which can
occur along with the solutions and links to helpful articles.
13.11 Loss of productivity time of end users due to unfamiliarity

with system, if not adequately trained or have access to a
user friendly manual.
13.13 It is recommended that the RM develop a concise user

friendly guide for the ERMS to be used by the Commission
staff, and which would contain FAQs to address known
issues ascertained from responses to the RM’s Omisuite
Evaluation Forms and other tips for users. This guide should
be placed on the SPICE and accessible to all staff of the
Commission.
36 | P a g e
13.14 A refresher training session on the use of the ERMS by end
users should be done by RM since the last one held was over
a year ago.
13.16 RM -
13.17 IM – Reviewed and noted
Name & Title of Implementation

Person Responsible Date

14.0 OPTIMISE EFFECTS AND USE OF RESPONSES

FROM EVALUATION FORMS
Objective # 10: To assess the adequacy and effectiveness of the

evaluation form to obtain feedback on the weaknesses of the system
and use of such feedback and determine whether the evaluation form
was within best practices.
37 | P a g e
14.2 IA reviewed the Evaluation Form created by the RM
Department for ascertaining feedback on the ERMS. It was
observed that the form was adequate when compared to best
practice and the guide provided by the “Software Evaluation:
Criteria Based Assessment”, taken from the Software
Sustainability Institute.
14.3 Findings
14.4 IA selected a representative sample of seven (7) staff

members and evaluated the current prevalent user issues and
changes made one (1) year later after RM’s initial evaluation
exercise done in July 2016. The following were observed:
14.4.1 Significant percentage decline in daily or regular usage of

system. This decrease can be earmarked to the numerous
issues expressed by staff on why they use the system less i.e.
searching difficulties (only 5% of what was searched for
usually displayed), ‘not user friendly’ and ‘it’s easier to
request hard files’.
14.4.2 Both questionnaires, done a year apart, revealed that users’

opinions still held that information was difficult to locate.
14.4.3 The majority of the main problems and issues highlighted

from the previous questionnaire were still present in the
responses that IA received from sampled users. This may
indicate that little or no progress have been made to resolve
these problems or issues within the ERMS.
14.4.4 Some of the main improvements solicited from the previous

RM’s questionnaire was still presented as current issues or
improvements needed in the ERMS when questioned by IA.
14.5 Risks
38 | P a g e
14.6 Non-cooperation of staff in participating in future
questionnaires and evaluation exercises when responses
doesn’t bring about change and issues seem to go unresolved.
14.7 Loss of the effectiveness of the anticipated system when

numerous users’ issues goes untreated.
14.9 Priority should be given to devise hard and soft solutions to

best address or mitigate issues encountered by the end users.
Responses to questionnaires should be revisited and a plan of
action should be determined and implemented.
14.11 RM -
14.12 IM – Reviewed and noted.

of Person Date
Responsible

of Person Date
Responsible
39 | P a g e
15.0 UPLOAD OF DOCUMENTS IN THE ERMS THAT
WERE NOT RECORDED IN VERSATILE AS
HIGHLIGHTED IN PREVIOUS AUDITS
Objective # 11: To ascertain completeness of records data by

tracing the existence of the missing files from the Versatile system
determined in previous audit (Registration of Registrants and
Securities) can now be traced to the new system.
15.1 Findings
15.2 IA randomly chose seven (7) items that were untraced to

Versatile from our previous audit ‘Review of Registrants’
done by IA in 2015 and eight (8) items that were untraced to
Versatile from our ‘Review of registration of Securities’ audit
also done by IA in 2015 .Of the fifteen (15) sampled items,
only one (1) was found which took one (1) minute to locate.
Overall, it took 104 minutes to look for items to ascertain
whether or not they were available in the system. Enquires of
RM staff indicated that the missing items as indicated in
previous audits were still a work in progress item.
15.3 Risks
15.4 There is a risk that the reputation and integrity of the

Commission could be at stake when asked to provide
documents and such documents cannot be found.
15.6 Following from IA’s previous reviews, it was recommended

that RM should consult with DR & CF to attempt to locate
these missing documents and an inventory list of the residual
40 | P a g e
missing documents should be compiled and submitted to the
General Counsel to determine the possible impact of not
having these documents. This recommendation is currently
being worked on and is still a work in progress item.
15.8 RM -
15.9 IM – Reviewed and noted.


16.0 DETAILED REPORT: PART TWO (2) VALUE FOR

MONEY
17.0 USE OF CONSISTENT APPROACH WHICH

FACTORS IN VALUE FOR MONEY
Objective # 12: To ascertain whether value for money was

achieved in the changeover to the new system. Determine the extent
41 | P a g e
to which funds were expended economically and efficiently and the
extent to which the related project was effective in meeting its
objectives.
17.1 The Post Implementation ERMS Value-For-Money (VFM)
audit was intended to examine how well the Trinidad and
Tobago Securities and Exchange Commission (TTSEC)
managed the project and its activities.
17.2 The initial approved budget (fiscal 2014/2015) for the ERMS
project was TT$1, 641, 195. It was however noted that some
components of the project were undertaken within the
2014/2015 fiscal year, while others were undertaken within
2015/2016.
Objective # 12 (a): The Economy - To ascertain whether

acquisition of the Newgen ERMS was the appropriate system with
the quality and quantity at the lowest reasonable cost.
17.3 Findings
17.4 IA reviewed the evaluation report namely ‘Design, Supply

And Installation of an Electronic Records Management
Software For the Trinidad and Tobago Securities and
Exchange Commission- Guide to the Evaluation of the
Proposal Documents’ and the supporting information which
accompanied the said report, for the three (3) companies
whose bids were evaluated by TTSEC. IA assessed the
information provided about each company in the said report
to evaluate TTSEC’s decision on whether or not it was the
best proposed choice in terms of cost, time taken to complete
each project and the fulfilment of most or all of the proposed
ERMS requirements.
17.5 In comparing the cost of each proposed system, it was noted

that Digi data was the most costly with a proposed cost of
TT$4,036,391; InfoTech proposal cost was TT$1,478,754,
(TT$2,557,632) less than Digi data’s proposal and Newgen
42 | P a g e
proposal was the least expensive at TT$1,407,195,
(TT$2,629,196) less than the Digi data’s proposal. It should
be noted that the Annual Support Cost/Maintenance and
Support cost was factored into the bids for each company for
the first year. Going forward the Maintenance and Support
cost would be incurred on an annual basis.
17.6 Cost was one (1) of the main factors in the decision for the
award of contract. The time taken to complete these projects
was not allocated any points by TTSEC when awarding this
contract. It should be noted that the following timeframes
were estimated for the completion of the project: Digi data
proposed timeframe was twenty (20) weeks, InfoTech
proposed six (6) weeks and Newgen proposed fourteen (14)
weeks. Infotech offered to complete the project in the least
time while the other two (2) companies proposed timeframes
that were more than twice and thrice that of Infotech.
17.7 Newgen was awarded the contract at the proposed cost of

$1,407,195 and the proposed timeframe of fourteen (14)
weeks in which to complete the ERMS project. However,
the project ran for sixty-four (64) weeks, an overrun of fifty
(50) weeks. The project started its kick-off meeting on
March 02nd, 2015 and had an agreed project Closure date of
June 1st 2016. A conservative approach was taken by IA and
therefore an allotment was made for a total of fifteen (15)
days for vacations, sick days and emergencies which reduced
this overrun to forty-seven (47) weeks, which was used in
IA’s calculation. To ascertain the overrun cost, IA reviewed
all the overtime, additional responsibility allowances and
utilised an estimate of the time allotted by each department to
the project during the period. This variance overrun cost to
the Commission was estimated at TT$868,856.35.
17.7.1 The real cost of the Newgen ERMS was estimated at

TT$2,787,077.24 as depicted in Table 1 below. This
43 | P a g e
took into account the contract cost of TT$1,407,195,
other cost for hardware of TT$234,000 and an
unplanned cost of TT$148,435.29 (a supplementary
approval was granted this additional sum by the
Tenders Committee for the unforeseen technological
costs for JBOSS Web and Application Servers
(clustering) for the purpose of high system availability),
hotel accommodation cost of TT$ 123,130.89, air
passage TT$5,388.70, staff associated variance overrun
cost of TT$868,856.35. This brought the overall cost/
real cost of the Newgen ERMS to about
TT$2,787,077.24 ($1,379,882.24) over the contract
price, which was almost twice the original bid cost and
$ 1,145,882.24 over the budgeted cost.
Table 1: Showing the breakdown of the Real Cost

incurred by TTSEC for the ERMS
Cost item Details Cost TTD
Newgen Contract cost: Implementation Efforts 519,000.00
77,850.00
Software Cost NewGen Software 583,050.00

Licenses
Recurring Year 1 Maintenance and 227,295.00
Support
1407,195.00
Other Cost:
Hardware cost Dell Hardware 234,000.00
Acquisition
Approved budget(fiscal 2014/2015 1641,195.00
Other unplanned cost JBOSS Web and 148,435.29

Application server
Revised project budget 1789,630.29
HOTEL ACCOMODATION The Normandie Hotel 37,388.96
44 | P a g e
AIR PASSAGE Trafalgar Travel Limited 5,388.70
Staff overruns:
IM's Responsibility For periods between 10,009.88
allowance and overtime (April 2015 to Mar 2016)
RM's Overtime. For periods between 8,342.04
(April 2015 to Mar 2016)
Salary Overrun
Overrun IM Dept. 574,558.03

Overrun CSD Dept. 275,946.40
STAFF ASSOCIATED VARIANCE 868,856.35
COST
TOTAL REAL COST OF THE NEWGEN
ERMS Commented [CJ6]: IM found that there was significant
TT$2,787,077.24
benefit in seeking to derive the true cost of the project and a
believe that a methodology should be determined for
17.8 The IA team reviewed the list of proposed ERMS calculating the cost to the Commission with cost overruns on
requirements that the TTSEC specified for the Commission future projects. As the process utilized for the purpose of this
report was not based on actual work done but rather
new ERMS. The list used by TTSEC was generated from the estimated figures, it was felt that these figures in total did not
limitation of the old ERMS Versatile system and then current represent the “real cost the newgen erms”
needs of the Commission. After reviewing the requirement

breakdown which accompanied the evaluation report, it was
indicated by TTSEC that Newgen was the only one (1) out of
the three (3) companies that could have provided all twenty
(29) requirements. Whereas for both Digi data and InfoTech, Commented [NA7]: We need to change the wording
perhaps from “real” to estimated
it was indicated that they could not have provided one (1) of
the requirement, Space Management.
17.9 Additionally, after the Newgen ERMS was implemented a

series of issues and errors occurred on a weekly basis. These
issues were monitored by both the RM and IM department.
As at June 2017, there were approximately fifty (50) issues
as stated in RM’s report “1 Year Implementation Review of
the Omni Suite (ERMS)”. Of these, approximately thirteen
(13) have been resolved. IM also indicated in its “Post
Implementation Review of the ERMS Application” on page7
that an average of two (2) hours per day, two (2) times per
week is spent on troubleshooting issues relating to the
ERMS. As indicated in the said IM report this was due to the
45 | P a g e
high volume of issues occurring on the ERMS. IA
nonetheless noted that the Commission was not required to
pay any additional cost directly to Newgen team to solve
these issues.
17.9.1 However IA noted that, the time spent documenting,

reporting and troubleshooting these issues are being done at a
cost to the Commission. IA estimated the associated cost to
be TT$4,414.56 for troubleshooting issues on a monthly
basis and $52,974.00 annually. This estimation took into
account two (2) members from RM and one (1) member
from IM. Therefore, the estimated maintenance cost for the
ERMS is the fixed annual maintenance cost of $227,295 and
the annual estimated associated cost for treating with errors
of TT$52,974, which amounted to a total of TT$280,275 per
annum.
17.10 Conclusion
17.11 Under the Economy objective, the Commission compared

two (2) of the three (3) main factors used to ascertain the
achievement of this objective, which was cost and fulfilling
the requirements (see further analysis under 16.24 (iv)) as
the Commission did not factor in time towards the allocation
of points for the awarding of the contract. The Newgen
ERMS cost was the lowest compared to the other three (3)
companies evaluated by the Commission and it was
purported by Newgen to have been able to provide all the
proposed ERMS requirements in TTSEC assessment of the
Newgen proposal. However, the time and cost overruns
resulted in the commission incurring additional direct and
variable costs, and other associated recurring cost due to the
abundance of errors occurring on a monthly basis. Therefore,
with all of the three (3) main factors (cost, time taken to
complete the project and completion TTSEC requirements)
46 | P a g e
not having been achieved, this Value for Money-Economy
objective was not achieved for the ERMS project.
Objective # 12 (b): Efficiency - To investigate whether the level of

service provided with the Newgen ERMS has improved the
Commission Service (internal and external) at no additional cost.
17.12 Findings
17.13 IA in collaboration with members of the Record

Management team compared the upload times under the
Newgen and Versatile ERM systems. As a result of the
interviews and field work done, IA noted that the Newgen
system takes almost three (3) times longer to upload
documents to the Newgen ERMS compared to the versatile
system. This makes it less efficient and has created new
challenges on the demand of the man power of the RM team
with an associated risk of error due to the level of human
intervention. As a result the RM team is currently seeking to
employ additional resources to the department to help reduce
this increase labour demand on the current RM team. If
granted, this will further cost the Commission approximately
TT$120,000 in salaries annually for two (2) additionally
clerical employees. Additionally, it should be noted that
although the system takes three (3) times longer to upload
documents compared to versatile, the Newgen ERMS adds
value through the indexing and search features which have
provided more search fields and criteria for locating
documents compared to the old versatile system.
17.14 The IA team developed a questionnaire to determine if the

Newgen ERMS had provided added benefits for staff of the
Commission in the access of files and information from the
RM department. The questionnaire was based on multiple
efficiency questions such as on frequency of use, success rate
of searches, difficulties encountered etc. The questionnaire
was distributed to members of staff from seven (7)
47 | P a g e
departments: the Disclosure, Registration and Corporate
Finance (DR&CF), Market Regulation and Surveillance
(MR&S), Compliance and Inspections (C&I), Human
Resource (HR), Policy Research and Planning (PR&P),
Finance and Library.
17.15 The following information was compiled from the

questionnaire:
17.16 From the sample chosen only 29% of the sample used the
ERMS on a daily basis, 13% used it at least once weekly,
29% use it on a monthly basis and another 29% sampled
never used the system. If the results of the sample was
extrapolated it shows a large percentage staffers does not use
the system and or rarely use it. This was further tied to the
question “How often do you find what is searched for?” this
results indicated that 60% of the five (5) sample users, had a
70% success rate. A follow question to this was “Is it easier
to request the hard files on use the ERMS?” 100% of the
sample selected they would request the hard files. The reason
for this was to: save time in locating documents, the ERMS
searching feature was not reliable, thus requesting the hard
copy files was a double check and only the cover pages of
documents were located on the ERMS.
17.17 Additional question was asked to ascertain what made this

ERMS system difficult to use. Most of the sample answered
by highlighting the difficulty to find documents using the
ERMS, the search feature does not work properly, a
document with multiple pages were uploaded separately and
not as one document and the ERMS was not user friendly.
Thus, presenting the issues of lack of confidence and
inaccuracies in the system.
17.18 Conclusion
48 | P a g e
17.19 Under the Efficiency objective, the ERMS system does not
appear to have achieved any of the efficiency goals. With
respect to uploading documents to the Newgen ERMS it took
three (3) times longer with little additional benefit from the
previous Versatile ERMS. With the additional time and steps
needed to upload documents, a request from the Director of
Corporate Service Department was made for additional staff
and if granted would add cost to the Commission to an
already over budgeted system. Furthermore, the impression
on staff minds was that the system was difficult to use for
retrieving information and because of this, staff preference
was to request the hard copy files. This high demand for the
hard copy documents poses a further strain on the RM
Department.
Objective # 12 (c): Effectiveness – To determine if the goals and

objectives of the ERMS are being achieved (the attainment of the
right results from the usage of resources & organizational
operations).
17.20 Achieved objectives
17.21 IA reviewed the eight (8) goals and objectives of the ERMS
outlined in the proposal document ‘Electronic Records
Management Proposal’ page 4 and the ‘Project Closure
Report’ page 14. After reviewing the eight (8) goals and
objectives it should be noted, that three (3) of the objectives
were achieved, three (3) were not achieved and two (2) were
partially achieved. IA reviewed and verified that the
following three (3) achieved objectives were attained.
i. The procurement of a system that will be able to fully

integrate into the existing technological environment.
ii. The facilitation of remote access to the ERMS where
required.
iii. The ability to certify, accept and route e-documents.
49 | P a g e
17.22 Findings
17.23 The objectives which were partially completed and not

achieved are discussed below:
17.24 Partially Completed
iv. The ability of the proposed system to meet the minimum

general, security and functional requirements.
The ERMS has met the minimum general requirements,
however, there were still some concerns that the RM team
expressed. Which was, on one (1) occasion a member of staff
was able to access folders that they were not supposed to
have access to. Although this was a one off occasion, the
security of the system needed to be tested to ensure that this
does not reoccur. After the system went live, testing
continued with the RM and IM team to ensure no
reoccurrence.
The ERMS functional requirements to date are still partially
completed. Note that the ‘Electronic Records Management
System Functional Requirements Matrix’ report made
reference to twenty-nine (29) features requested, seventeen
(17) was completed, ten (10) was not met, one (1) labelled as
unsure by the Commission and one (1) as not applicable.
Therefore, with forty-one percent (41%) of the system
requirements not delivered this objective is hence, partially
completed.
v. More flexible full text and advanced search capabilities

for all documents inclusive of scanned documents.
Due to the detailed indexing of all records uploaded to the
ERMS, there is advanced search capabilities within the
system. However, in the report ‘Electronic Records
Management System Functional Requirements Matrix’ item
21, Newgen, indicated after the implementation of the project
that “this feature is not their strongest search feature and as
50 | P a g e
such should not be the primary search option for the staff.
Only about 5% of data may be displayed when a search is
conducted. It is therefore not reliable.” Considering the
aforementioned, the search capabilities of the ERMS is not
reliable and should not be relied upon to ensure all
documents search for are located. This objective was
therefore labelled as partially completed because of the
limitation of the search capabilities.
17.25 Not Achieved
vi. To employ ERMS to support integration, timely and

effective decision making, and improved services to users.
The Commission’s SharePoint communication platform was
not integrated into the ERMS, nor was the requirements with
respect to mail logging, notification, etc completed via
ERMS, thus excel sheets such as the mail log and the task
listing are maintained to fulfil these purposes. The RM
department efficiency level to decrease. E.g. processing time
has increased thus resulting in delays as it pertains to when
records are made accessible to staff across the Commission
as well as interdepartmental communication. As a result,
service has not improved as it takes longer to make
documents available for timely and effective decisions
making.
vii. The integration of the chosen application to the

Commission’s SharePoint communication platform.
The SharePoint communication platform was not integrated
with the ERMS making it difficult for the SharePoint records
and some of the versatile records to being migrated into the
new ERMS. As of the report date, there were at least five
percent (5%) of outstanding versatile data to be migrated in
to the ERMS.
viii. The ability to certify, accept and route e-documents.

51 | P a g e
The ability to certify a document (certify a document as a
true copy of the original by getting it signed and dated),
Accept and route e-documents (An electronic document is
any electronic media content (other than computer programs
or system files) that are intended to be used in either an
electronic form or as printed output).
After reviewing the Electronic Records Management
Proposal’ page 4 and the ‘Project Closure Report’ page 14.
It was noted that ‘the ability to certify a document’ and
‘accept and route e-documents’ was an objective of the
Commission that was not delivered. These features were
however linked to Omniflow, which was not a module
provided by the Newgen team. IA could not find any
explanation or evidence on why this module was not
provided to achieve this objective.
17.26 Conclusion
17.27 Under the Effectiveness objective, the ERMS system did not
fully achieved five (5) of its eight (8) objectives. Although
the Commission was aware of these issues, to date no
solution was determined to fully complete the partially
achieved objectives or fulfil the objectives that were unable
to been achieved. Therefore, with sixty-three percent (63%)
of the goals and objectives not fully achieved, the Value for
Money Effectiveness objective was not achieved in this
ERMS project.
17.28 Overall Conclusion
17.29 IA assessment of the ERMS project identified that Value for

Money was not achieved. The ERMS has cost the
Commission almost twice as intended for a product that has
not completely met its functionality requirements in terms of
quality of service provided by the software and has impacted
the resources of TTSEC, as it took about three and half times
52 | P a g e
the projected timeframe for its implementation. This negative
impact on resources is continuous due to the added labour
intensive demand on RM department and continuous cost
being incurred by the Commission related to treating in
ongoing system errors.
17.30 Recommendation
17.31 A greater emphasis should be placed on taking a holistic

approach to project evaluation and award as well as in the
implementation assessment at the close out phase of the
project, that take into consideration the overall hard and soft
costs to be incurred and the resulting value/ benefit to be
derived to the Commission. The method utilised should
ensure the achievement of the best combination of price and
quality to meet the particular need of the Commission within
an acceptable timeframe.
APPENDIX 1 Table Displaying Section 24 (1) of the Integrity of

Public Life Act 2000
24. (1) A person to whom this Part applies shall ensure that he
performs his functions and administers the public resources for
which he is responsible in an effective and efficient manner and
shall—
(a) be fair and impartial in exercising his public duty;
(b) afford no undue preferential treatment to any group or

individual;
(c) arrange his private interests whether pecuniary or

otherwise in such a manner as to maintain public
confidence and trust in his integrity.
53 | P a g e
APPENDIX 2 Table Displaying the Issues and
Recommendations following KPMG’s review.
This table also reflects IA’s Follow up on the Implementation statues
of each Recommendations along with Management’s comments for
non –implementation on due dates.
Issue/ Findings Recommendations Status of Due Management’s
Implementation Dates comments for
non –
implementation
on due dates
Establish formal IM Management should Outstanding May On conduct of
procedures for document and 2017 the
restoration testing of implement procedures Department’s
backups.IM’s draft for restoration testing of strategic
Operational Manual backups for critical planning
contains a policy for systems. exercise at the
Business Continuity end of 2016, it
Management as it relates was noted that a
to Computer Contingency revision to the
Plans. While there is a Audit timelines
Backup and Recovery was warranted,
policy, there are no given the
documented procedures number of
to support the policy. Strategic
projects in
which the
Division was
engaged on an
annual basis. As
such all dates
have since been
revised.
54 | P a g e
non –
implementation
on due dates
Backup media are sent to Restoration testing Outstanding June The item is
off-site storage (G4S) should be performed on 2017 linked to the
every Friday and rotated a regular basis, current
on a weekly basis. It was preferably every quarter, virtualization of
indicated by the Systems to ensure the integrity, the server
Analyst that backup tapes availability and network, which
are tested in a test reliability of the data in would impact
environment, however case of an emergency. the current
the test results are not Where possible, backup defined backup
documented. It was also media should be strategies. The
noted that backups are restored and tested in a current SAN
not encrypted when sent test environment similar Upgrade and
offsite. to the “live” production Virtualization
environment. project is due to
Additionally, the results complete by
of recovery testing June 1st.
should be documented
for audit purposes.
Although, IM Data on backup tapes Outstanding May On conduct of
Management indicated should also be encrypted 2017 the
that the Commission had to prevent unauthorized Department’s
an active SLA with a access to sensitive strategic
vendor for the 2013- information held. planning
2014 period that provided exercise at the
for Quarterly reviews of end of 2016, it
the backup application was noted that a
and an annual restore revision to the
exercise, no evidence was Audit timelines
seen of restoration testing was warranted,
of backups given the
number of
Strategic
projects in
which the
Division was
engaged on an
annual basis. As
55 | P a g e
non –
implementation
on due dates
such all dates
have since been
revised.
56 | P a g e

DRAFT IA Audit Report On RM Post Implementations With Changes New

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

DRAFT IA Audit Report On RM Post Implementations With Changes New

Diunggah oleh

Hak Cipta:

Format Tersedia

NOTE: NAZIA’s changes are in blue and the items I believe

ought to be removed are in pink

Internal Audit Department

POST IMPLEMENTATION REVIEW OF

A product of the Internal Audit Department

Overall Report Risk Rating Summary

Risk Rating Low Medium High

No. of Audit Issues: 12 2 7 3

RISK SCALE KEY

The issues, risks and recommendations identified during this audit

A product of the Internal Audit Department

1.0 EXECUTIVE SUMMARY4

A product of the Internal Audit Department

1.1 An Enterprise Wide Risk Assessment of the Trinidad and

1.2 Management has recently replaced the existing electronic

1.3 A post implementation review was conducted by Internal

1.5 IA’s assessment of the ERMS project identified that Value

A product of the Internal Audit Department

2.2 The secondary audit objectives of this review were to:

i. Assess the adequacy and effectiveness of the process for the

viii. Assess adequacy of Backup and Disaster Recovery

x. Assess the adequacy and effectiveness of the evaluation form

xii. Ascertain whether value for money was achieved in the

 The Economy - To ascertain whether acquisition of the

 Efficiency - To investigate whether the level of service

 Effectiveness – To determine if the goals and objective of the

2.3 In conducting this review, IA held discussions with the

i. The Commission’s new ERMS; and

A product of the Internal Audit Department

3.0 SUMMARY OF FINDINGS (Arranged accordingly from

Ref # Recommendations Page Risk level

Error! Reference source not found.

7.0 Error! Reference source not found. 15

COMPLETION OF THE TRANSFER OF

IMPROVEMENT IN THE USE OF SYSTEM

A product of the Internal Audit Department

FURTHER GUIDANCE TO STAFF ON ERMSError!

OPTIMISE EFFECTS AND USE OF RESPONSES

4.0 DETAILED REPORT: PART ONE (1) OPERATIONAL

5.0 IMPROVEMENT IN THE TENDER EVALUATION

Objective # 1: To assess the adequacy and effectiveness of the

A product of the Internal Audit Department

5.2 IA reviewed the document labelled Design, Supply and

5.4 The above mentioned document which was equivalent to an

5.5 It was noted that the document sub-titled ‘Guide to the

5.5.1 Even though a ‘schedule of project activities Milestones and

5.5.2 Even though a ‘profile of the project team members’ was

A product of the Internal Audit Department

5.6.1 For example, under the category ‘Relevant Experience with

5.6.2 Another example, under the category ‘Functionality and

Thus, a high level of subjectivity was evident in the

5.8 There was a discrepancy observed in the weighing of the cost

A product of the Internal Audit Department

5.8.1 IA observed that page 12 of the ERMS Project Closure

5.10 Lack of transparency and uniformity in evaluation process

5.11 Overruns in budgeted cost and estimated timelines when a

5.13 It is recommended that when conducting the tender

5.15 A proper justifiable basis must be agreed and consistently

5.15.1 CSD should request from Policy Research and Planning

5.16 A clear distinction must be made between the criteria

5.17 All project reports and documents used in the evaluation

5.18 Management Comments

5.20 IM – IM agrees in the most with the recommendations