How-to Audit How-to for a project

Geo-Spatial Project Learning Institute 11-300 Earl Grey Dr Suite 209 Kanata, Ontario K2T 1C1 www.iGPLI.net

Larry Cooper, GPLI Larry.Cooper@IGPLI.net www.IGPLI.Net

GPLI 1001: Rev 1.1 (www.iGPLI.net)

2008 GPLI

Agenda
Audience Poll Audit Terminology Audit Conditions The Audit Process Questions?

Audience Poll
How many of you have taken over a project that is in progress?
How many got to do a project audit? How many felt they were at a disadvantage from not being able to do an audit?

If you did get to do an audit, did you find everything was ok:
Yes? No?

How many of you have done an audit of a completed project otherwise known as a Post Implementation Review?

Audit Terminology

Objective versus Subjective

Objective:
undistorted by emotion or personal bias; based on observable phenomena; "an objective appraisal"; "objective evidence"

Subjective:
taking place within the mind and modified by individual bias; "a subjective judgment"

What is Audit?
Independent review and examination of records and activities to assess the adequacy of system controls, to ensure compliance with established policies and operational procedures, and to recommend necessary changes in controls, policies, or procedures. Definition can be applied to almost anything that is auditable: financial audit; quality audit; project audit; etc There is an expectation that the opinion of the auditor that is expressed in the final audit report will be objective.

Types of Auditors
There are two types of auditors:
Internal auditor- are employees of a company who assess and evaluate its system of internal control. To maintain independence, they present their reports directly to the Board of Directors or to Top Management. They provide functional operation to the concern. External auditor are independent staff assigned to assess and evaluate an organizations systems of internal control or to perform other agreed upon evaluations. They are called upon from the out side of the company.

Types of Project Audit

Informal
New project manager No indication that project is in trouble Need to take stock of where the project really is as opposed to where it is reported to be Can apply same criteria as formal audit but likely no need for depth and breadth or a formal report

Formal
Project is in trouble Sponsor agrees that an audit is needed Sensitivities are high Need to be able to prove conclusions via substantiated evidence

Audit Conditions

Audit Conditions
When to do an Audit Audit Circumstances Purpose of a Project Audit What should be covered? Who should do the Audit?

When to do an Audit
Change in Project Managers Project in Trouble Post Implementation Review Random

Audit Circumstances
Project Manager was removed Project Manager quit Delivery Issues Budget/Contract Issues Dysfunctional project teams Client/Customer complaints The fact that an audit is required or has been requested results in heightened sensitivities on the part of the sponsor, the project team and other stakeholders Expect some resistance and turbulence!

Purpose of a Project Audit

Determine if project is on-track or not If off track, determine why and offer specific advice on how to get it back on track OR whether the project ought to be cancelled Determine whether what caused the issues are systemic and how to avoid them in the future

What should be covered?

Project Management Deliverables All Project Deliverables Contracts

Who should do the Audit?

That depends.
A PMP should do the Project Management portion SMEs should do the rest*
Architecture Specialists Systems Integration Specialists Information Modeling/Architecture Specialists IT Service Manager or ISO 20000 Consultant for ITSM projects Contracts or Procurement specialists Etc.

PM would own the final report * A single person can do the audit if they have the requisite knowledge and
experience

The Audit Process

The Audit Process

Initiation and Planning Enquiry Reporting (Structure) Final Report

Initiation and Planning

With the project sponsor agree on:
The scope of the audit (whats in, whats not) The Basis of Opinion (more later) The Documents and References The Questions to be asked

Enquiry
Obtain the Project Organization and Structure Identify who should be interviewed Obtain all identified Documents and References Conduct Interviews
Others may be identified during the interviews

Review all documents and references

May be added to as Audit progresses

Reporting
Daily/Weekly updates on progress/issues for Sponsor Identify obstacles and how to overcome them Provide sponsor with critical project findings as soon as they can be substantiated helps mitigate the OMG factor and blaming the messenger when the final report is submitted At about the 75% marker provide a draft audit report for the sponsor to gauge their acceptance on key findings and wording

Structure of an Audit Report

Executive Summary
Audit Focus Basis of Opinion Audit Findings Recovery Plan

Documents and References Used (overall) Individual Audit areas with individual findings
Area of Interest and Basis of Opinion Documents and References used Statements of Fact Findings

Audit Focus
Clear statement on what is being audited:
Eg. The focus of the audit was on the project management deliverables :
Project plan Project Schedule Work Breakdown Structure Etc.

as well as the core project deliverables:

Systems Integration Architecture and Strategy, Business Process Model Information Model Developed Software Etc.

Basis of Opinion
Financial Audit:
We conducted our audit in accordance with International Standards on Auditing. Those standards require that we plan and perform the audit to obtain reasonable assurance about whether the financial statements are free of material misstatement. An audit includes examining, on a test basis, evidence supporting the amounts and disclosures in the financial statement. An audit also includes assessing the accounting principles used and significant estimates made by management, as well as evaluating the overall financial statement presentation. We believe that our audit provides a reasonable basis for our opinion.

Basis of Opinion for a Project Audit

The audit was conducted based on normal industry expectations for the area being
audited as follows:
The core project management deliverables were audited for conformance to the Project Management Institutes Project Management Body of Knowledge (PMBOK) at a basic level only for the Project Plan, the WBS, the Project Schedule, Project Estimates, Resourcing, and Project Change Control. The core project deliverables were assessed according to whether they in fact were identified as being required and if they were, what would they reasonably be expected to cover. The documents and references that were used are fully listed at the beginning of the audit areas, as well as where they are individually referenced. They were found at <> as well as on the server that hosted the referenced application. Simple Statements of Fact that any other PMP, ITIL Service Manager, ISO 20000 Consultant or IT professional would have made on documents and references as compared against normal industry expectations based on the review of the documents and references. An interpretation of Statements of Fact as to what they has led to or will lead to in the ability of project to deliver normally expected outcomes for an implementation in an IT Service Management Context. That comparison determines what is stated as the findings for the area being audited.

Every attempt was made to conduct the audit itself and to present the audit findings without prejudice.

The objective here is to show that any other person of the same qualifications would reasonably have reached the same conclusions in the audit report.

Audit Findings
Clear statements that establish, based on the analysis of the interview results and the documents and references by appropriate SMEs, the true state of the project and its core deliverables as compared to what was previously reported. Must stick to the facts that can be proven i.e. a professional (objective) rather a personal (subjective) opinion Must be such that any other person of the same qualifications for the area of findings would have reasonably concluded the same thing

Recovery Plan
If the project is to be continued, it is likely expected that you are to propose a recovery plan:
Highlight what can be carried forward it gives some hopehowever slim Be honest about what cannot be carried forward Realize that you will likely be asked to execute the plan

Audit Team Size/Timeline

Audit team can 1 or more but should not likely be more than 3 Timelines can range from a few days to a few weeks or longer depending on the size of the project

Sample questions Project Management

http://web.princeton.edu/sites/ppo/AuditWorksheet.dot

Sample Questions Project Deliverables

Sample Audit Report Content

Project WBS
Basis of Opinion:
A Project WBS is defined as: The WBS is a tool for defining the hierarchical breakdown of work required to deliver the products of a project. Major categories are broken down into smaller components. These are sub-divided until the lowest required level of detail is established. The lowest units of the WBS become the activities in a project. The WBS defines the total work to be undertaken on the project and provides a structure for all project control systems. By having a WBS that is tied to identified deliverables it is then possible to easily establish project dependencies (including dependencies on other projects), build activity time estimates, and determine appropriate project resources based on the skills that are needed for success.

Documents and References

Ref A. Deliverables, Effort and Cost Estimates document Ref B. Management Submission Ref C. Project Schedule in MS Project

Statements of Fact
Statements of Fact:
The WBS was somewhat contained at Ref A in that some of the identified deliverables also had activities associated with them, and at Ref C which is the closet to a project WBS that was attempted The WBS at Ref C does not appear to have been built using Ref A or Ref B as the source but rather it appears there was an attempt to link the three after the fact Deliverables that were identified at Ref A or Ref B were not necessarily shown in Ref C Most deliverables had no associated WBS that broke the work down into smaller components that could be easily identified Where work breakdowns did occur, it was not to a sufficient level of detail for work planning purposes for a project schedule as defined activities Most of the WBS at Ref C was activity-based with no discernable deliverable at the top of each activity hierarchy

Findings
For all intents and purposes a project WBS was never created as the project
deliverables were never properly enumerated or defined. For those that were, the WBS was clearly at too high a level as it was not broken down into actual activities, while in other areas only activities were defined with no associated deliverables This has led to or will lead to the following:
A project WBS that does not represent either the identified or required project deliverables An inability to develop a project schedule with appropriate time estimates, dependencies, or resourcing (see 4.4 Project Schedule below) Without a proper WBS we in fact cannot be sure what to report or whether what we report is indicative of what the project ought to be delivering or in point of fact is delivering. A general inability to know when a deliverable is completed even when the deliverable itself was clearly identified Confusion on the part of project team members on what they are to produce and how they will know when they are done

Information and Data Modeling

Basis of Opinion: Data modeling is the act of exploring data-oriented structures. Like other modeling artifacts data models can be used for a variety of purposes, from high-level conceptual models to physical data models. In most instances, you are likely to see three basic styles of data model:
Conceptual data models. These models, sometimes called domain models, are typically used to explore domain concepts with project stakeholders. Conceptual data models are often created as the precursor to LDMs or as alternatives to LDMs. Logical data models (LDMs). LDMs are used to explore the domain concepts, and their relationships, of your problem domain. This could be done for the scope of a single project or for your entire enterprise. LDMs depict the logical entity types, typically referred to simply as entity types, the data attributes describing those entities, and the relationships between the entities. Physical data models (PDMs). PDMs are used to design the internal schema of a database, depicting the data tables, the data columns of those tables, and the relationships between the tables.

Documents and References

Ref F: Product Information Model, Vendor x Ref G: Corporate Information Modeling Guidelines xxx.doc Ref J: Project Information Models

Statements of Fact
Statements of Fact:
The information model and guidelines as evidenced at Ref F and Ref G does not align with Ref J and indeed resulted in the products base information being modified from its intended purpose Ref J does not reflect Ref G neither substantively nor in intent Neither Ref J nor what was present in the instances accessible for review appropriately represented the products base information at Ref G as had been agreed with the project team. Indeed it was mostly focused on the layers that it had been agreed that the sister project team would handle. The products base information at Ref G and as illustrated in the diagram on the previous page appears not to have been used. Of the instances that were created with project developed information models and that was documented in Ref J, when compared to the data to be moved over from the existing system, it was found to be severely lacking.

Findings
As the tool came with a proven information model, which formed the
basis of its selection, it seems odd that it was not substantively implemented or followed within the project. Indeed, not only was it not followed, the actual model diagram was modified without noting why the modifications were made and without any input from the vendor (verbally confirmed with them). There is little evidence that the information model that was developed within the project would be suitable for use in any operational context. At best, when compared to the data in existing operational systems it was to replace, it was illustrative of how much work still needed to be done. This has led to or will lead to the following:
Inability to develop coherent training materials Inability to integrate with other operational systems Having to make an either/or decision for which one to implement as they are for all intents and purposes, incompatible views

Conclusion
Audits, whether in the financial or project arenas, must be objective and free of bias They must establish the Basis of Opinion The Enquiry phase should use prepared questions to test for evidence to establish the Statements of Fact The Statements of Fact must be just that The findings must be based on Statements of Fact (evidence) that can be demonstrated There should be a recovery plan proposed that you as a PM can live withbecause you likely will have to!

Questions?