Audit of the Design and Operation of the Sampling

Document Sample
Audit of the Design and Operation of the Sampling Powered By Docstoc
					   Audit of the Design and
Operation of the Sampling Plan




           AUDIT REPORT




        Project # 08/09 01–01




                prepared by
    the Audit and Evaluation Directorate




             OCTOBER 2008
                                                                                                                   PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN                                                                                                AUDIT REPORT



                                                           Table of Contents


1.0         SUMMARY ..................................................................................................................................3

      1.1 AUDIT OBJECTIVE ..........................................................................................................................3

      1.2 AUDIT OPINION ..............................................................................................................................3

      1.3 STATEMENT OF ASSURANCE ..........................................................................................................3

      1.4 SUMMARY OF RECOMMENDATIONS .................................................................................................3
2.0         AUDIT REPORT ..........................................................................................................................5

      2.1 BACKGROUND ...............................................................................................................................5

      2.2 AUDIT OBJECTIVES, SCOPE AND APPROACH ...................................................................................6

      2.3 FINDINGS, RECOMMENDATIONS AND MANAGEMENT RESPONSE .......................................................6

            2.3.1 EFFECTIVENESS OF EXISTING SYSTEM ...................................................................................6

            2.3.2 COST-EFFECTIVENESS OF EXISTING SYSTEM ..........................................................................8

APPENDIX - MANAGEMENT ACTION PLAN ..................................................................................................11




                 AUDIT AND EVALUATION DIRECTORATE
                                                                                                                                                2
                                                                                 PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN                                                            AUDIT REPORT


1.0   SUMMARY
1.1   AUDIT OBJECTIVE
      The objective of the audit was to evaluate the design and operation of the monitoring system
      used to identify higher-risk transactions for review and validation.

1.2   AUDIT OPINION
      In our opinion, the design and operation of the sampling plan require significant
      improvements.

1.3   STATEMENT OF ASSURANCE
      In my professional judgment as Chief Audit and Evaluation Executive, sufficient and
      appropriate audit procedures have been conducted and evidence gathered to support the
      accuracy of the opinion provided and contained in this report. The opinion is based on a
      comparison of the conditions, as they existed at the time of the audit, against pre-established
      audit criteria, and is only applicable to the particular entity examined. The evidence was
      gathered in compliance with Treasury Board policy, directives and standards for internal audit.
      The evidence has been gathered to be sufficient to provide senior management with the proof
      of the opinion derived from the internal audit.

1.4   SUMMARY OF RECOMMENDATIONS
      The Accounting, Financial Reporting and Policies Division is responsible for the Agency’s
      payment requisition monitoring system, subject to internal policies and directives. However, it
      also relies on the co-operation of managers and financial clerks and officers with responsibility
      for financial operations in the sectors concerned to improve the efficiency of the account
      verification process and to periodically review the sampling plan in an efficient and cost-
      effective manner.

      Our audit showed that the sampling plan adequately identifies most higher-risk transactions
      for account verification purposes, since most payment requisitions have been coded high-risk.

      Further to our review of the design and operation of the sampling plan, we recommend
      • that the sampling plan be reviewed and approved, annually and on a need basis, by the
        Chief Financial Officer (CFO);
      • that the various transaction categories’ degree of risk be reviewed;
      • that periodic analytical reports be prepared; and
      • that a correction and follow-up mechanism for the errors detected be developed with the
        staff of the sectors concerned.




            AUDIT AND EVALUATION DIRECTORATE
                                                                                                      3
                                                             PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN                                       AUDIT REPORT


     Signature of the Chief Audit and Evaluation Executive


     Original signed by Jean-Guy Desrosiers




     Audit team member


       Jimmy Cheung




           AUDIT AND EVALUATION DIRECTORATE
                                                                                4
                                                                                   PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN                                                             AUDIT REPORT


2.0   AUDIT REPORT
2.1   BACKGROUND
      The Canadian Space Agency (CSA) policy is to pay on time amounts actually owed to third
      parties. Account verification mechanisms must be designed and used in such a way as to
      ensure their effectiveness, with due regard for the risks related to the characteristics of each
      payment requisition and to monitoring requirements.

      Sampling design
      In April 2003, a manual sampling plan was adopted to identify, for review and validation,
      payment requisitions coding high, medium or low risk based on the nature of the transactions.
      Under this plan, 13 of the 20 transaction categories were coded high-risk.

      The sampling plan is based on a feasibility study that took place between October 1, 2002,
      and February 28, 2003.

      Sampling operation overview

      Sector financial operations                Accounting Services accounting clerks do a spot
      clerks enter the payment                   check (a few key items) or a full check (according to
      requisition data into SAP,                 the checklist under sect. 34) of documents received
      then send the hard copy                    from the sectors under the sampling plan, then send
      documents to Accounting                    the verified documents to Accounting Services
      Services                                   financial officers for approval before payment




      Accounting Services                        Accounting Services financial
      financial officers transfer                officers verify certain key
      SAP data entered by                        elements of the documents
      sectors' financial operations              verified by the clerks before
                                                                                             Payment
      clerks into another interface              affixing their initials and marking
                                                                                             made
      for payment purposes                       "sect. 33" on the documents and
                                                 approving the payment in SAP




            AUDIT AND EVALUATION DIRECTORATE
                                                                                                         5
                                                                                PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN                                                          AUDIT REPORT


2.2   AUDIT OBJECTIVES, SCOPE AND APPROACH
      The purpose of the audit was to evaluate the design and operation of the monitoring system
      used to identify higher-risk transactions for review and validation.

      The audit dealt with the design and operation of the sampling plan instituted in April 2003. We
      also examined the breakdown of payment requisitions by transaction category for fiscal year
      2007–2008.

      Various audit processes were employed, including staff interviews and reviews and analyses
      of documents and records. A review of the existing sampling plan and the main departmental
      policies and directives was done in order to develop audit criteria.

2.3   FINDINGS, RECOMMENDATIONS AND MANAGEMENT RESPONSE

2.3.1 EFFECTIVENESS OF EXISTING SYSTEM
      The Accounting, Financial Reporting and Policies Division is responsible for the Agency’s
      payment requisition monitoring system, subject to internal policies and directives. Hence, we
      expected to find an effective system including the following main elements:
      • a sampling plan reviewed and approved annually and on a need basis by the CFO;
      • a sampling plan embodying the various transaction categories and the appropriate risk
        level and degree of significance;
      • a sampling plan whereby transactions in each transaction category could be selected; and
      • a sampling plan whereby all high-risk payment requisitions that were sensitive or for large
        amounts could be identified for account verification purposes.
      In general, management did draw up a sampling plan whereby higher-risk transactions could
      be identified for account verification purposes. Indeed, our exam revealed that most payment
      requisitions had been coded high-risk.

      However, we would like to point out certain findings that require management attention.

      SAMPLING PLAN
      The sampling plan, including the sampling parameters (i.e. an acceptable level of confidence
      and a 10% maximum acceptable error rate (MAER)) should be updated and approved,
      annually and on a need basis, by the CFO.

      Our discussions revealed that the annual sampling plan update by the CFO had not been
      done since April 2003.

      RECOMMENDATION
      i) The Finance Directorate should review the sampling plan annually and on a need
         basis, including the sampling parameters, and have it approved by the CFO.




            AUDIT AND EVALUATION DIRECTORATE
                                                                                                      6
                                                                                              PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN                                                                          AUDIT REPORT


     MANAGEMENT RESPONSE
     We agree with the recommendations and have already taken steps to rectify the situation.

     SAMPLE
     For account verification purposes, the sampling plan should take into account such elements
     as transactions involving large sums or those involving high risk.

     Our audit brought to light the lack of any sampling, for review and validation purposes, of
     journal vouchers and requisitions for interdepartmental settlement, as no risk level was
     assigned to these two transaction categories, which do involve large amounts. Table A gives
     some representative data.

      TABLE A – OTHER TRANSACTIONS
                                                                    FISCAL 2007–2008
      TRANSACTION                  NUMBER OF TRANSACTIONS
      CATEGORY                             PROCESSED                                         VALUE
      Journal vouchers                         Note                Transactions of up to $1 million; totalling a few million.
      Interdepartmental                       1,200                Transactions of up to $1.3 million; totalling $21 million.
      settlements
      Note: Since the information is not available, we estimated that the number of transactions processed might involve
            several hundred journal vouchers (for example, miscoding, breakdown of monthly costs, PAYEs). Journal
            vouchers are cross-charges.


     RECOMMENDATION
     ii) The Finance Directorate should review these transaction categories’ degree of risk
         to enable proper transaction sampling for account verification purposes. However,
         high-risk transactions and those involving large sums must be verified individually.

     MANAGEMENT RESPONSE
     Although our sampling plan does not deal with interdepartmental settlements (ISs) or journal
     vouchers (JVs), we do have certain controls in place (verification of delegations for JVs and
     ISs over $100,000). We have not made it a priority to include these verifications in the plan,
     because we consider the risk associated with the transactions to be lower than for
     transactions involving payments to outsider suppliers. For instance, JVs are usually
     corrections or transfers between responsibility centers, where a manager has already signed
     a Section-34 authorization for payment. However, starting November 1, 2008, all JVs deemed
     to involve risk (change of parliamentary credit, change of ledger account affecting a ledger
     involving risk [eg, hospitality, ex gratia payments, etc]) and all ISs over $100,000 will be fully
     verified. These kinds of transactions will be added and documented in our next sampling plan.




           AUDIT AND EVALUATION DIRECTORATE
                                                                                                                        7
                                                                                PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN                                                          AUDIT REPORT


2.3.2 COST-EFFECTIVENESS OF EXISTING SYSTEM
     We expected to find a cost-effective system including the following main elements:

      • statistical data kept;
      • periodic analytical reports, in particular on errors detected, non-compliance, and the
        various corrective measures to be taken regarding the errors detected in order to
        periodically review the sampling plan, including the sampling parameters; and
      • proper communication between Accounting Services and the managers and financial
        officers and clerks in the sectors, to improve the account verification process.

     There is a system in place, the Central Accounting Verification System (CAVS), that allows
     statistical data to be kept, such as the number of errors detected.

     However, we would like to point out certain findings that require management attention.

     PERIODIC ANALYTICAL REPORTS
     The sampling plan should include a periodic analysis of the findings, followed by
     recommendations on the corrective measures to be taken. Moreover, an evaluation should be
     done at regular intervals of the statistical sampling method chosen for each type of transaction
     to make sure that it is still appropriate and adequate. That evaluation should contain
     information on the causes of errors and provide a mechanism for correction and follow-up with
     stakeholders.

     Our audit brought to light the lack of periodic analytical reports on the CAVS, for example a
     report on errors detected that could be used as a periodic review of the sampling plan, to
     adjust the error rate and re-evaluate the risk associated with each transaction category; this
     would have had an impact on the sampling volume for verification purposes. Better
     communication about error rates between Accounting Services and the sectors concerned
     ought to improve the quality of payment requisitions.

     We also noted that no corrective action had been taken on the errors detected. Neither had
     any follow-up been done in that regard with the sectors concerned. As regards to errors
     detected in the area of travel expenses, an email was sent to the travellers and to the sector
     clerks concerned. It is up to the travellers to advise their managers if necessary.

     Moreover, our audit revealed that the error rate was in excess of the MAER, particularly for
     travel expenses, training costs and membership fees, in spite of the corrective measures
     already identified in the 2007–2008 action plan on the audit of travel, conference and
     hospitality expenses.

     Table B gives details on the payment requisitions audited and processed by transaction
     category, together with the error rates.




           AUDIT AND EVALUATION DIRECTORATE
                                                                                                      8
                                                                                                    PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN                                                                                  AUDIT REPORT



 TABLE B – PAYMENT REQUISITIONS

                                                                                  2007–2008

                                                   RISK     REQUISITIONS   REQUISITIONS     %           ERRONEOUS
 TRANSACTION CATEGORY                             CODED       AUDITED       PROCESSED     AUDITED      REQUISITIONS   ERROR RATE

 Travel within Canada (more than $2,000)          High

 Travel within Canada (between $1,000 and       Medium
 $2,000)

 Travel within Canada (less than $1,000)          Low

 Travel to the United States (more than           High         1,261         4,006         7%             613           49%
 $1,000)
 Travel to the United States (less than           Low
 $1,000)
 International travel                             High

 Non-public servants travel                       High

 Hospitality                                      High          22                                          4           18%

 Grants & contributions                           High          94                                          9           10%

 Contracts                                        High         2,831                                      391           14%

 Conferences                                      High           5                                          0            0%

 Relocations                                      High          23                                          3           13%

 Training                                         High          40          13,494        28%              15           38%

 Acquisition card & petty cash purchases          High          69                                         18           26%
 Membership fees & ex gratia payments             High           2                                          1           50%

 Invoices (more than $10,000)                     High

 Invoices (between $2,001 and $10,000)          Medium
                                                               1,826                                      225           12%

 Invoices (less than $2,000)                      Low

                                                   TOTAL       6,173        17,500        35%            1,279          21%

 It should be noted that payment requisitions do not include salaries, journal vouchers or interdepartmental settlements.




               AUDIT AND EVALUATION DIRECTORATE
                                                                                                                            9
                                                                            PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN                                                      AUDIT REPORT


     RECOMMENDATIONS

     The Finance Directorate should
     i) ensure that periodic analytical reports from the CAVS are done to justify revising or
        not revising the sampling plan, including the sampling parameters, and are also
        used to communicate with the staff of the sectors concerned; and
     ii) ensure that there is a mechanism for the correction of errors detected and follow-up
         with the staff of the sectors concerned, to improve the effectiveness of the account
         verification process.

     MANAGEMENT RESPONSE
     We agree with the recommendations and have already taken steps to rectify the situation.




           AUDIT AND EVALUATION DIRECTORATE
                                                                                                10
                                                                                                                                                    PROJECT # 08/09 01-01


AUDIT OF THE DESIGN AND OPERATION OF THE SAMPLING PLAN                                                                                                         AUDIT REPORT


                                                            APPENDIX - MANAGEMENT ACTION PLAN


 REF.           Recommendations              Responsibility Identified                                DETAILS OF ACTION PLAN                                    Timetable
                                             Organization    Function
 2.3.1 Effectiveness of existing
       system
  i)    The Finance Directorate should       Finance       Chief         In 2006–2007, Information Technology (IT) developed a new system (CAVS) to            JUNE 30,
        review the sampling plan annually    Directorate   Financial     enable the accounting team to calculate erroneous transactions. The system            2009
        and on a need basis, including the                 Officer       was supposed to allow us to:                                                          (APPROVAL BY
        sampling parameters, and have it                                 1) calculate error rates by type of error and transaction category;                   THE CFO AND
        approved by the CFO.                                                                                                                                   IMPLEMENT-
                                                                         2) produce reports to implement corrective measures and follow-up with the            ATION OF THE
                                                                         parties involved in the verification process; and                                     NEW
                                                                                                                                                               SAMPLING
                                                                         3) annually adjust the parameters of our sampling plan and have it approved by
                                                                                                                                                               PLAN)
                                                                         the CFO.
                                                                         The reports did not allow us to do those things. The error definitions were not
                                                                         explicit enough and the errors were not calculated correctly.
                                                                         As of September 30, 2008, we had reviewed the description and classification of
                                                                         errors (critical, non-critical). We met with the head of the CAVS at IT to correct
                                                                         the problems identified above, which will allow us to perform activities 2) and 3).
                                                                         Beginning in fiscal year 08–09 and at the end of each subsequent fiscal year,
                                                                         we will evaluate the sampling plan and adjust the parameters as needed. This
                                                                         plan will be approved by the CFO as of June 30 of each year to enable us to
                                                                         consider all of the transactions from the previous fiscal year.




            AUDIT AND EVALUATION DIRECTORATE                                                                                                                                  11
                                                                                                                                                      PROJECT # 08/09 01-01


AUDIT OF THE DESIGN AND OPERATION OF THE SAMPLING PLAN                                                                                                           AUDIT REPORT


 REF.           Recommendations                Responsibility Identified                                DETAILS OF ACTION PLAN                                    Timetable
                                               Organization    Function
  ii)   The Finance Directorate should         Finance       Chief         Our sampling plan already takes large-sum and high-risk transactions into
        review these transaction categories’   Directorate   Financial     consideration.
        degree of risk to enable proper                      Officer
        transaction sampling for account                                   As for JVs and ISs, here is what we intend to do:
                                                                                                                                                                 NOVEMBER 1,
        verification purposes. However,                                    All JVs deemed to involve risk (change of parliamentary credit, change of ledger      2008
        high-risk transactions and those                                   accounts affecting a ledger involving risk [eg, hospitality, ex gratia payments,
        involving large sums must be                                       etc]) and all ISs over $100,000 will be fully verified. For fiscal year 2008–2009,
        verified individually.                                             those transactions will be analyzed for a five-month period, ie, from November 1
                                                                           to March 31.
                                                                           We will then review the sampling plan for the transactions described above and        JUNE 30,
                                                                           for the other kinds of transactions already included in the plan at the intervals     2009
                                                                           indicated in recommendation 2.3.1 i) of the action plan.
 2.3.2 Cost-effectiveness of existing
       system
  i)    The Finance Directorate should         Finance       Chief         In 2006–2007, IT developed a new system (CAVS) to enable the accounting
        ensure that periodic analytical        Directorate   Financial     team to calculate erroneous transactions. The system was supposed to allow us
        reports from the CAVS are done to                    Officer       to:
        justify revising or not revising the                               1) calculate error rates by type of error and transaction category;
        sampling plan, including the
        sampling parameters, and are also                                  2) produce reports to implement corrective measures and follow-up with the
        used to communicate with the staff                                 parties involved in the verification process; and
        of the sectors concerned.
                                                                           3) annually adjust the parameters of our sampling plan and have it approved by
                                                                           the CFO.
                                                                           The reports did not allow us to do those things. The error definitions were not
                                                                           explicit enough and the errors were not calculated correctly.
                                                                           As of September 30, 2008, we had reviewed the description and classification of
                                                                           errors (critical, non-critical). We met with the head of the CAVS at IT to correct
                                                                           the problems identified above, which will allow us to perform activities 2) and 3).
                                                                           a)   IT will modify our CAVS by December 31, 2008;                                    DEC 31, 2008

                                                                           b)   Accounting will accumulate data in the CAVS database from January 1,             MAY 31,
                                                                                2009, until the end of fiscal year 2008–2009;                                    2009




            AUDIT AND EVALUATION DIRECTORATE                                                                                                                                    12
                                                                                                                                                       PROJECT # 08/09 01-01


AUDIT OF THE DESIGN AND OPERATION OF THE SAMPLING PLAN                                                                                                           AUDIT REPORT


 REF.           Recommendations                 Responsibility Identified                                DETAILS OF ACTION PLAN                                   Timetable
                                                Organization    Function

                                                                            c)   We will communicate the results of the analysis (rate of error by transaction   JUNE 30,
                                                                                 type) to the Sector Financial Operations Division and take corrective           2009
                                                                                 measures to reduce the error rate (notice of information or training); and
                                                                            d)   The data accumulated in b) will serve as a starting point for reviewing our     JUNE 30,
                                                                                 sampling plan in 2.3.1 i).                                                      2009

  ii)   The Finance Directorate should          Finance       Chief         Beginning in 2008–2009 and in subsequent years, we will produce reports on a         1ST REPORT
        ensure that there is a mechanism        Directorate   Financial     quarterly basis, ie, as of June 30, September 30, December 31 and May 31 (at         FOR FISCAL
        for the correction of the errors                      Officer       the end of P12-2).                                                                   YEAR 2009–
        detected and follow-up with the staff                                                                                                                    2010 AS OF
        of the sectors concerned, to                                        We will communicate the results of the analysis (error rate by transaction type)     JULY 31,
        improve the effectiveness of the                                    in the month following the publication of the report to the Sector Financial         2009
        account verification process.                                       Operations Division and take corrective measures to reduce the error rate
                                                                            (notice of information or training).




            AUDIT AND EVALUATION DIRECTORATE                                                                                                                                    13