Post-Implementation Review Audit Program Contributed 1/21/00 by Geoff Storms 1. Determine if the business proposal/contract for the system included all relevant information, including: * Reasons for the project (Business Requirements) * Scope of the project * Costs and benefits of the project * User requirements (software, hardware, communications, operating environment, other pertinent technical constraints) 2. Determine if the project had an established project team, including a leader from IS project area? * Did the project team have the level of authority to make the decisions concerning the project? * Did the project team have the appropriate level of expertise? In the technical and business area? * Did the project team include members from the user areas as well as systems development, vendors, computer operations, audit, legal, compliance and all other appropriate area? 3. Determine if there was a feasibility study/plan prepared that met requirements and details the project plan as required. * Was the feasibility study written and approved by management? * What were the evaluation criteria? * Evaluation Decision (documentary evidence of the results of the evaluation) * Evaluation final report (a doc describing how the candidate systems fared including: a summary of findings, conclusions drawn, evaluation of recommendations) * Does the study detail the scope of the project * Had the project budget been included? * Had the appropriate level of management reviewed and approved the study? * What provisions if any had been made for overruns, delays, and changes? 4. Determine if the project plan was followed and any deviations documented, including extensions of the schedule. * Are all deviations documented? * Did the project team and management approve all extensions? * Were relevant parties notified of any extensions or changes to the project plan? 5. Determine if detailed user requirements have been developed 6. Evaluate the quality of information (to ascertain the timeliness of system response to recordable events, the degree of accuracy in recording these events, and the timely understandable presentation of the information to users) * Gauge responsiveness to events by calculating the average time between the occurrence of an event and its reflection in the system
* Examine error rates on edits and file maintenance reports as well as observe the keying habits of data entry clerks. Errors may mean that data are not being captured accurately at the source, that the design of the system does not encourage accurate data entry, or that there are management or training problems in the data-entry department. * Evaluate clarify of systems reports * Interview users to evaluate system quality 7. Determine if the system was adequately tested prior to implementation, the test plan includes all aspects of the new system, and all unexpected results are thoroughly resolved. * Did the test team develop a test plan? * Has the test plan been written * Will there be system and acceptance tests? * Are users included in the testing? * Will all aspects of the system tested, as outlined in the detailed requirements, including, but not limited to; * Data entry * Editing * Reports * Calculations * Error reporting * Interfaces with other systems * Network communications * Print handling * Are all critical functions tested? * Are all existing capabilities tested? * Are all changes tested? * Will testing be completed prior to implementation? * What were the criteria for the termination of parallel run? * Does the test plan allow for re-testing of errors and changes? * Determine if volume and stress testing was done 8. Evaluate the adequacy of system documentation * Is the old system documented and understood? * Verify that an up-to-date user instruction manual exists and is used * Systems and programming documentation is current and follows installation standards * Concise, well-written operating instructions exist and are used by computer operators * Interview users to get a feel of user satisfaction with the system * Are the specification documented? * Data files * Interfaces * Procedures * Screens * Reports * Documents * Are all existing accounts, products and services known and documented? * Consider the following attributes for source code availability: * Where the source code is kept * Who has access to the source code * Whether it is on-line, off-line or archive stored * How quickly it can be retrieved for inspection
9. Determine if the results of the new system met the original objectives as stated in the business proposal * Processing * Determine at the project's conclusion whether the project met the objectives defined in the original proposal. * Were the expected benefits of the new system realized? * Does the system perform as expected? * If there were differences found in expectations and actual results, were they investigated and dispositions noted? * If there were inefficiencies noted, were they documented and the dispositions noted? * Cost/Benefit * Determine if the cost/benefit analysis was correct. * Compare the actual vs. budgeted costs and benefits * Were the actual costs within reason * Were price changes and increases in volumes taken into account in the original estimate? * Evaluate the reasons for the differences in actual vs. budgeted costs and benefits. * Do the reasons for the variances appear valid? * User Satisfaction - determine if the user is satisfied with the new system and evaluate what points should be considered in another project. * Determine if the user is satisfied with the operations of the new system. * Are there problems or inefficiencies in the new system that can be corrected? * Is there a need for ongoing training? * Have all problems been corrected? * Does the system meet the user requirements? * Does the system provide all the required information? * Conduct a survey of the users (a sample is acceptable) to determine if training was sufficient, the system is operating as expected, reports are providing the required information, problems are being resolved. * Application Controls (Determine that the application was adequately designed to meet the functional business requirements) * Review any changes to the application during all phases of the project to determine if they significantly changed the project's original goal. * Are all changes documented? * Are all changes reviewed and approved by the project team? * Does upper management approve all changes that affect the project scope? * For any significant changes, is the project re-evaluated to determine the feasibility/cost/benefits? * Is the system written in a common programming language? * Are there programming resources available to maintain the system? * Determine if the program contains adequate controls over data integrity. Are there controls built into the system, including edits, verifications, etc.? * Input Design (Determine if the in0put requirements are adequately defined and documented) * Review the documentation for the input requirements of the system to ensure it includes editing and validation, security provisions, control totals, and appropriate authorizations
* Determine if the input file definitions are defined and documented * Have the files been defined, including all record layouts? * Have the databases been defined? * Have the security levels been established and defined for file and database access? * Determine if the application allows for batch or control totals. Are the totals logged? Can the control totals be reconciled between input and output? * Determine if provisions have been made for data preparation and computer processing errors to be reviewed and reentered correctly. Can the errors be detected and corrected prior to completion of the processing cycle? * Who will be performing the review process? Frequency? * Who will be performing the re-entry function? * Determine if provisions have been made for any internal tables or parameter files to be periodically reviewed by the user for accuracy. What is the frequency of the review? Who makes changes to these files? How are errors corrected? * Determine if error files are to be reviewed to determine the extent and type of outstanding errors for trend analysis purposes. Who will be doing the analysis? Frequency of reviews? What type of errors will be included in the review? * Processing Design (to determine if the processing requirements are defined and documented adequately) * Determine if written procedures have been prepared that explains all error codes and messages and corrective action for each. Are all error codes/messages included? Error codes for operators as well as data entry should be included. * Determine if the application has provisions that prevent concurrent file/record updates. Is the file/record locked when one user is accessing in update? Are there appropriate error messages provided? * Output Design (Determine if the output requirements are defined and documented adequately) * Review the adequacy of the documentation for output requirements. Has information been classified as restricted, confidential, public...etc. * Determine if the output provides the users with the ability to control and ensure the completeness, accuracy, and authorization of the data. Do the reports include the ability to trace the originator of each transaction? Is there a means to verify the information included on the reports? * Have the calculations used to develop data (accruals, fees, rates, etc.) been checked for accuracy? * Determine if provisions have been made for the user to scan output reports/datasets/files to detect obvious errors. These can include missing files, unreasonable values, incorrect report dates, formats, etc. * Determine if service level agreements between operations/vendor and the user are in place or are being negotiated. These should include response time & system "up" time * Interfaces (To determine if there is adequate security and controls over the application interfaces.) * Which system does the new system interface with? How does it interface? * Were all interfaces tested adequately? * What are the controls over the application interfaces?
10. Determine if a training plan was developed for the project and if user training was adequate * Was a training plan developed? * Is the training plan written? * Was training completed prior to implementation of the system? * Will critical personnel be trained early in the training? * Will there be staff trained to train other? * Will there be technical training for operators? * Are all aspects of the system covered in the training : * Data entry * Backups * Management reporting * Disaster recovery * User operations * Computer operators * Balancing and reconciliation 11. Examine the operating system software under the following criteria: * Suitability * Security * Change control and upgrade mechanism * Documentation * Support * Testing * Performance * Installation * Maintenance 12. Audit trails and procedures. To determine if there are adequate and effective audit trails and reports designed in the system. * Are audit reports listed on the report distribution list? Is the audit department satisfied with the information produced on the audit reports? Will the information satisfy audit needs? * Review the schedule of reports to determine if they include: * Error reports * Logs of all logon attempts * Logs of all invalid signon attempts * Balancing reports * Transaction registers * Determine the security and integrity of the audit trail reports. Can users input information that will alter the audit trail reports? Are the reports distributed and reviewed by the appropriate people? 13. Security (To determine if the security over the hardware, software, and data is adequate, and determine if the data security procedures are in place) * Physical Security (To determine if the physical security over the hardware and software is adequate) * Are the items secured in some way? * Are they locked in a room, limited access area, or some way controlled? * Are the terminals, PCs in a locked, inaccessible area, kept away from public and unauthorized users? * Is there control over the modem? * Where are the diskettes stored?
* Are the diskettes/storage media locked in a fireproof cabinet? * For systems using a modem, determine if there is a fully dedicated line and the security surrounding the modem use. * Are there verification procedures used when using the modem? * Are terminals inactivated or locked during non-business hours? * Data Security (To determine if the data security procedures are in place) * Review the data security access levels and assignments to determine if they appear adequately controlled. Are there varying levels of security access for different types of transactions: - inquiry only, update nonmonetary transactions, update financial transactions, add/delete records. Are the levels appropriately assigned to the user department staff? Does management appro0ving transactions have the authority to input the transactions? * Who has the ability to change passwords? * Are the password assignments controlled by the user department or data security? If controlled by the user department, does the staff member also have authority to input transactions? * Review the system password access - are passwords masked, encrypted, stored in a visible file? * Determine if there are controls to log and monitor all sign-on attempts, both valid and invalid. Is access to the system monitored? * Determine if the application has controls in place to prevent unauthorized access to the system. Does the system lock out after a certain number of invalid sign-on attempts? Is both a password and logonid required for access to the system? * Backup and Recovery Procedures (To determine if there are adequate backup and recovery procedures developed for the system) * Determine if there are procedures developed for disaster recovery and restart for the system. Have recovery/restart procedures been written? * Determine if there are procedures developed for periodic backup of the system. * How often will backups be done? * How long will backups be kept? * What media will backups be done on? * Have the backup procedure been written? * Will there be daily backups? Weekly? Monthly? Quarterly? Yearly? * Will the backups be shipped to off-site storage? * Database Integrity (To determine if there is adequate security over the database and interfaces. * Determine if there are adequate controls in place that prevent unauthorized access to the application source and object code. Are programmers prevented from accessing production code? Is access to the code logged? 14. System Documentation (To determine if the system documentation is complete) * Determine if an operations manual has been prepared prior to implementation of the system. Does the manual include complete instructions on the system operation? Are the operator's manuals readily available to all operators? * How has the system changed the way in which operations are performed?
* How has the system receive? * How has the system users receive? * How has the system * How has the system organization? * How has the system
changed the accuracy of information that users changed the timeliness of information and reports changed the completeness of information? changed the interactions between members of the changed productivity?