Docstoc

Post Implementation Review Audit Program

Document Sample
Post Implementation Review Audit Program Powered By Docstoc
					Audit Program Subject: Post Implementation Reviews

Version 1.0 Version Date: 15 October 2001

Author: David M Burbage IT Auditor Internal Audit Unit Western Sydney Area Health Service Locked Bag 7118 North Parramatta NSW Australia

e-mail: David_Burbage@wsahs.nsw.gov.au

Background The audit will be undertaken as part of the 2000 / 2001 Audit Plan. Scope and Period of Audit To perform a post-implementation review The audit reviewed outcomes of project/s (both computerised and non-computerised) undertaken within the organisation, to verify whether the project/s achieved the desired results and met the strategic outcome measures predicted by the organisation, within the planned costs and schedules. The review covered the period 1 January 1999 to 1 December 2000. In order to determine the success of a project, the organisation needed to evaluate the performance of the system after it is implemented. Through the Post-Implementation Review (PIR), the organisation determined whether the project met its objectives and, further, evaluates the development and management processes that supported the project. The review also compares actual, total project expenditures to the project cost estimates. The PIR also determined whether the technology or change of work practice investment is yielding the expected benefits to the processes, products or services supported by the organisation. Approach The audit was conducted by : * The circulation of a questionnaire to users, Project Managers, Data Centre Staff and Systems Administrators of 5 projects to ascertain the level of acceptance, ease of use, appropriateness of training, access to user documentation, etc. Interviews with project sponsors, project managers (internal and external where available), systems administrators, data centre and ISD staff to ascertain the status, maturity, robustness and overall capability of the project. Interviews with the project sponsor and other relevant staff to ascertain the financial and other benefits realisation.

*

*

Post Implementation Review Overall Audit Findings

Group 1. Contractor/s

Control Consideration 1.1 Have contract details changed during the project life-cycle? 1.2 Have cost and time parameters been met during the project life-cycle? 1.3 Have professional standards and work standards been met during the project life cycle? 1.4 Have all promised modules and functionality been supplied and implemented? 1.5 Have contractors / sub contractors provided adequate acquittance of claims, documentation of work undertaken, outcomes, etc? 1.6 Have contracts been monitored? 1.7 Have contractors displayed adequate knowledge and expertise in relation to the project? 2.1 Have Team members been provided with an effective working environment and intra-/ inter-/ discipline support? 2.2 Have all disciplines been kept in the mainstream of change and functionality at all times? 2.3 Has the team maintained good documentation during the project life cycle? 2.4 Have issue logs been maintained, and have these been ratified prior to sign off? 2.5 Were the project team adequately trained? 3.1 Have users been trained adequately? 3.2 Have users accepted the system enthusiastically? 3.3 Have users concerns or problems been adequately addressed? 3.4 Have user expectations been met satisfactorily? 3.5 Have user manuals been produced and are they readily accessible? 4.1 Has management displayed adequate support and control over the project? 4.2 Have management revised policies and procedures

2.

Project Team

3.

Users

4.

Management

Post Implementation Review or standards? 4.3 Have obligations of all parties been monitored and validated?

Post Implementation Review

5.

ISD

5.1 Has a monitoring process been established to determine the efficacy of the system? 5.2 Has the system proved stable since go-live? 5.3 Is a service level agreement in place for the system? 5.4 Have all parties been satisfied with the level of service to date? 5.5 Has the system integrated effectively with other systems? 5.6 Has liaison with the vendors been effective and professional? 5.7 Has vendor support been adequate, effective and timely? 5.8 Has security within the system been effective? 5.9 If the system has been a joint effort between two or more vendors, have they worked effectively and united? 5.10 Have appropriate technical manuals been produced relating to the system installed. 6.1 Have payments relating to the project / system been based on contractual terms and conditions? 6.2 Have cost reports been viewed, verified and validated? 6.3 Has the system changed substantially from what was originally promised? Has this affected the Area=s strategies, viability or work practices? 6.4 Have outstanding issues been resolved to the satisfaction of the Area? 6.5 Have promised benefits been realised? 6.6 Has a method been implemented to capture promised / anticipated savings? 6.7 If the project was implemented to support any statewide strategy, has this been achieved? 6.8 Have performance agreements been substantiated for contractors, vendors, etc? 6.9 Have milestones been signed off legitimately? 6.10 Has project documentation been retained and maintained? 6.11 7.1 7.2 Has a system administrator been appointed? Have all users applied for, and received, access

6. Project Sponsor

7. System Administrator

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review

Group

Control Consideration rights to the system? 7.3 Is there a hierarchical level of access for users, super users, etc? 7.4 Has a criteria been set for enabling super user status, etc? 7.5 Are issues, problems, errors, faults logged and escalated, where necessary, to ISD or vendor

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review

Group

Control Consideration

Risk Analysis
Risk Analysis Objective Control Audit Test / Questionnaire

1. The application as tendered was at variance from what was functional at go live. 2. User expectations have not been realised. 3. Training has been inadequate.

1, 2

1.1 - 1.7, 2.1 - 2.5, 3.2 - 3.4, 4.1, 4.3, 5.5, 5.9, 6.3, 6.4, 6.7, 6.9, 6.10 1.3, 1.4, 2.2, 3.2 - 3.4, 4.2, 5.4, 5.7, 6.3, 6.4, 6.5, 6.7, 7.5 2.2, 2.5, 3.1, 3.5, 5.10, 5.11, 7.6

2.1, 2.2, 3.1, 3.2. 8.1, PA 3.2, 4.1 - 4.3, DC, EU, PA 5.1 - 5.3, DC, PA, EU 6.1 - 6.3, DC, PA, EU 7.1 - 7.7, 8.2, PA, EU 9.1, 9.2, DC, PA, EU 9.1 - 9.3, DC, PA 10.1 - 10.3, DC, PA 11., PA 12.1 - 12.4, PA

2

1, 2

4.

Manuals are inadequate and out of date.

1

3.5, 5.10

5. There is no ongoing administration of the system. 6. The application is frequently unavailable or does not work . 7. The application requires constant application of patches. 8. No SLA was agreed to, or it is not operating. 9. The project has not been capitalised

1

3.3, 5.1, 5.6 - 5.8, 6.4, 7.1 - 7.5.

1, 2

1.4, 2.4, 3.3, 3.4, 4.3, 5.1 - 5.5, 5.7, 6.4, 6.8, 7.5 1.4, 2.2, 2.4, 3.3, 4.3, 5.1, 5.2, 5.7, 6.3, 6.4, 6.8, 6.10, 7.5 3.4, 4.3, 5.1, 5.3, 5.4, 5.7, 6.4, 6.8, 6.10, 7.5 1.2, 4.1, 6.1, 6.2, 6.6, 6.10 1.2, 1.5, 1.6, 4.1, 6.1, 6.2,

1

1, 2

3 1, 3

10. The budget to actual costs were not properly reconciled. 11. Savings outlined in the business case have not been realised.

1, 2, 3

1.2, 1.5, 1.6, 4.1, 6.1, 6.2, 6.5, 6.6,

12.4, 13.1, 13.2, PA

Key: DC PA Data Centre Staff Project / System Administrator / Sponsor

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review EU End User

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review

Audit Tests
Location / Unit Department Working Paper Ref Completed By & Date Audit Result

AUDIT TESTS
1. Previous Management Letters Determine if a previous audit has been performed (internal / external) and that all recommendations and proposed actions have been implemented. 2. Application Functionality 2.1 Review Business Case / project initiation documentation / IPS and determine, by questionnaire and interview with the Project Sponsor, whether the application / system / project as installed / implemented at go live stage varied adversely to the inital proposal. 2.2 Review Change Control documentation to ascertain level of changes to original specification 3. Pre Go-Live Issues Review Project Steering Committee minutes / other documents to ascertain the issues current at go-live. 3,1 Were all issues referred to Steering Committee for go-live decision? 3.2 Have all issues been subsequently addressed to the Area=s satisfaction? 4. User Expectation By questionnaire: 4.1 Ascertain whether user expectations have been met. 4.2 Ascertain whether ISD expectations have been met 4.3 Ascertain whether project sponsor=s expectations have been met

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review

5. Training By questionnaire and Interview: 5.1 Users. Ascertain a. Whether users received training b. Whether the training was timely, adequate and pertinent c. Whether users received take away workbooks 5.2 ISD. Ascertain a. Whether ISD staff (particularly data centre staff) received training in the package. b. Whether processes are in place to identify deficiencies in training (e.g. constant calls from users / system administrators for advice on how to perform basic functions). 5.3 System Administrator. Ascertain a. Whether System Administrators received adequate and timely training. b. Whether processes are in place to identify deficiencies in training (e.g. constant calls from a single user for advice on how to perform basic functions). 6. Manuals By questionnaire and interview: 6.1 Process users: a. Do users have user manuals; b. Are manuals kept up to date; c. Are manuals readily available to all users. 6.2 Systems Administrators: a. Do System Administrators have appropriate technical manuals; b. Are the manuals kept up to date; c. Is access restricted to valid users only? 6.3 ISD / Data Centre: a. Do staff have appropriate technical manuals; b. Are the manuals kept up to date? 7. Administration By questionnaire and interview ascertain: 7.1 Is there a designated administrator of the system / process? 7.2 Are relevant records kept of:

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review

Location / Unit

Department Working Paper Ref Completed By & Date Audit Result

AUDIT TESTS
a. Down time; b. Application issues, faults, errors; c. User profiles; d. Upgrades to process, access, documentation? 7.3 Are access rights reviewed and modified where necessary? 7.4 Is there an ongoing review of the SLA to determine need for revision? 7.5 Is the Administrator the custodian of the project documentation, and is such documentation stored securely? 7.6 Have performance statistics been maintained? 7.7 Have all users submitted an authorised access agreement document? 8. Change Control 8.1 Have all changes from the original specification been logged and authorised? 8.2 Have all changes to the application since golive been logged and authorised? 9. System Stability 9.1 Has the system proved robust and: a. Has downtime been minimal; b. Have users expressed concern over the system (speed, durability, poor outcomes); c. Are there substantial outstanding issues with the vendors? 9.2 Are there ongoing problems with the maintenance of the system (software / hardware / peripherals)? 9.3 Have upgrades been provided in accordance with contractual obligations? 10. 10.1 10.2 Service Level Agreement Is there a service level of agreement? Have service levels been met?

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review

Location / Unit

Department Working Paper Ref Completed By & Date Audit Result

AUDIT TESTS
10.3 11. Has the SLA proved adequate? Has the project been satisfactorily capitalised?

12. Cost reconciliation 12.1 Were budget to actual reconciliations performed during the life of the project? 12.2 Were reconciliations presented to the Project Steering Committee during the project life cycle? 12.3 Was a final reconciliation performed and submitted to the Steering Committee at go-live time? 12.4 Are reconciliations performed regarding running costs and to indicate the cost effectiveness of the system? 13. Cost benefit savings identified in the business case or other pre-project documentation: 13.1 Have been achieved and validated. 13.2 Have yet to be achieved due to the time frame for such savings, and there is a mechanism in place to capture such savings. Questionnaire : DC - Data Centre Staff

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review Name: Tel: Application: Position: Date

1. Is there an Service Level Agreement in existence for this application between the Data Centre and the system users / owners?

Yes ( ) Attach Copy

No ( ) Attach reason

Have all parties been satisfied with the level of service to date? Yes ( ) No ( ) Expand upon answer if necessary:

2. Has the system integrated effectively with other systems?

3. Has the application presented any specific problems for the Data Centre?

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review

4. Has the vendor provided appropriate support?

5. Did the vendor provide adequate training to maintain the application?

6. Did the vendor provide adequate documentation to maintain the system?

7. Has a monitoring process been put into place to determine the efficacy of the application? 8. Has the application shown any signs of instability? If the system was a multi-Vendor System 10. Did they work effectively and united? Did the collaboration produce problems?

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review Questionnaire : PA - Project / System Administrator / Sponsor (Please circle appropriate)

Name: Tel: 1. Are issues, problems, errors, faults: Application: Logged?

Position: Date Yes ( ) Please indicate location of logs and readiness for review: No ( ) What method is used to record / monitor issues?

escal ISD ated to: Vendor

2. Did the vendor provide adequate training to administer and maintain the system?

Yes ( )

Please provide details:

No ( )

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review 3. Have all users applied for, and received, access rights to the application? Yes ( ) Please provide details:

No ( )

4. Have access rights been granted according to a set criteria?

Yes ( )

Please provide details:

No ( ) 5. Have protocols been developed to justify granting of super user, supervisor, etc status? Yes ( ) Please provide details:

No ( ) 6. Has any follow up been undertaken to ensure users are confidant in their use of the application (including reviewing issue logs to ascertain users who are poor adaptors of the system)? Yes ( ) Please provide details:

No ( )

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review 7. Has there been an unacceptable level of patches / fixes applied by the vendor, or is the system showing signs of instability or unusability? Yes ( ) Please provide details:

No ( ) 8. Is there a Service Level Agreement agreed to with ISD and the vendor, and is it working effectively? Yes ( ) Please provide details:

No ( )

9. Was the application as tendered the same that went live

All tendered modules and features in place?

Yes ( )

Please provide details

No ( )

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review Capacity of system as per tender validated in use? Yes ( ) Please provide details

No ( ) 10. Have user expectations been met, or are users complaining regarding the functionality of the application? Yes ( ) Please provide details:

No ( ) 11. Were the team kept in the mainstream of change and functionality during the project and were they provided with an effective working environment with intra- / interdisciplinary support? Yes ( ) Please provide details:

No ( )

12. Were the project team adequately trained for the project?

Yes ( )

Please provide details:

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review No ( ) 13. Is the documentation from the project securely stored and maintained, and have issue logs been validated and updated since go-live? Yes ( ) Please provide details:

No ( ) 14. Has management provided adequate support for the project, and is this support ongoing? Yes ( ) Please provide details:

No ( ) 15. Have policies and procedures and standards been revised as a result of the implementation of the new system? Yes ( ) Please provide details:

No ( ) 16. Have obligations of all parties been monitored to ensure compliance and capability? Yes ( ) Please provide details:

No ( )

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review 17. Contracts / Contractors Have payments relating to the project / system been based on contractual terms and conditions? Yes ( ) Please provide details:

No ( ) Have performance agreements been substantiated for contractors, vendors, etc.? Yes ( ) Please provide details:

Yes ( ) 18. Have cost reports been viewed, verified and validated? Yes ( ) Please provide details:

No ( )

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review 19. Has the system changed substantially from what was originally promised? Has this affected the strategies, viability or work practices? Yes ( ) Please provide details:

No ( )

20. Were outstanding issues resolved to the Area=s satisfaction?

Yes ( )

Please provide details:

No ( )

21.

Refer Business Case / other documentation developed before implementation,

have benefits promised been realised?

Yes ( )

Please provide details:

No ( ) Is the project on track to achieve future benefit realisation Yes ( ) Please provide details:

No ( )

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review Has a method to capture promised savings been implemented? Yes ( ) Please provide details

No ( ) 22. If the project was implemented to support any statewide strategy, has this been achieved? Yes ( ) Please provide details:

No ( ) 23. Have performance agreements been substantiated for contractors, vendors, etc? Yes ( ) Please provide details:

No ( ) 24. Have milestones been signed off in accordance with the contract? Yes ( ) Please provide details:

No ( )

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review Questionnaire : EU - End User Name: Tel: 1. Did the application meet your expectation? Application: As promised prior to implementation of project Yes ( ) Position: Date Please provide details

No ( ) As operated after implementation Yes ( ) Please provide details

No ( ) 2. Training Did you receive training prior to the application go live Yes ( ) Please provide details:

No ( )

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review Was the training adequate, timely and relevant? Yes ( ) Please provide details

No ( ) Were you given adequate work books for the training Yes ( ) Please provide details

No ( )

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review 3. User manuals Do you have access to an up to date user manual? Yes ( ) Please provide details:

No ( ) Is it usable and comprehensive? Yes ( ) Please provide details:

No ( ) 4. Usage of the application Is the application easy to use? Yes ( ) Please provide details:

No ( ) If you have problems, do you know who to contact? Yes ( ) Please provide details

No ( ) Are problems resolved to your satisfaction? Yes ( ) Please provide details

WESTERN SYDNEY AREA HEALTH SERVICE Post Implementation Review No ( ) Is the application frequently unavailable or erratic? Yes ( ) Please provide details

No ( )


				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:349
posted:4/1/2009
language:English
pages:28