Get documents about "Audit of the Design and Operation of the Sampling
Document Sample
Audit of the Design and
Operation of the Sampling Plan
AUDIT REPORT
Project # 08/09 01–01
prepared by
the Audit and Evaluation Directorate
OCTOBER 2008
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN AUDIT REPORT
Table of Contents
1.0 SUMMARY ..................................................................................................................................3
1.1 AUDIT OBJECTIVE ..........................................................................................................................3
1.2 AUDIT OPINION ..............................................................................................................................3
1.3 STATEMENT OF ASSURANCE ..........................................................................................................3
1.4 SUMMARY OF RECOMMENDATIONS .................................................................................................3
2.0 AUDIT REPORT ..........................................................................................................................5
2.1 BACKGROUND ...............................................................................................................................5
2.2 AUDIT OBJECTIVES, SCOPE AND APPROACH ...................................................................................6
2.3 FINDINGS, RECOMMENDATIONS AND MANAGEMENT RESPONSE .......................................................6
2.3.1 EFFECTIVENESS OF EXISTING SYSTEM ...................................................................................6
2.3.2 COST-EFFECTIVENESS OF EXISTING SYSTEM ..........................................................................8
APPENDIX - MANAGEMENT ACTION PLAN ..................................................................................................11
AUDIT AND EVALUATION DIRECTORATE
2
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN AUDIT REPORT
1.0 SUMMARY
1.1 AUDIT OBJECTIVE
The objective of the audit was to evaluate the design and operation of the monitoring system
used to identify higher-risk transactions for review and validation.
1.2 AUDIT OPINION
In our opinion, the design and operation of the sampling plan require significant
improvements.
1.3 STATEMENT OF ASSURANCE
In my professional judgment as Chief Audit and Evaluation Executive, sufficient and
appropriate audit procedures have been conducted and evidence gathered to support the
accuracy of the opinion provided and contained in this report. The opinion is based on a
comparison of the conditions, as they existed at the time of the audit, against pre-established
audit criteria, and is only applicable to the particular entity examined. The evidence was
gathered in compliance with Treasury Board policy, directives and standards for internal audit.
The evidence has been gathered to be sufficient to provide senior management with the proof
of the opinion derived from the internal audit.
1.4 SUMMARY OF RECOMMENDATIONS
The Accounting, Financial Reporting and Policies Division is responsible for the Agency’s
payment requisition monitoring system, subject to internal policies and directives. However, it
also relies on the co-operation of managers and financial clerks and officers with responsibility
for financial operations in the sectors concerned to improve the efficiency of the account
verification process and to periodically review the sampling plan in an efficient and cost-
effective manner.
Our audit showed that the sampling plan adequately identifies most higher-risk transactions
for account verification purposes, since most payment requisitions have been coded high-risk.
Further to our review of the design and operation of the sampling plan, we recommend
• that the sampling plan be reviewed and approved, annually and on a need basis, by the
Chief Financial Officer (CFO);
• that the various transaction categories’ degree of risk be reviewed;
• that periodic analytical reports be prepared; and
• that a correction and follow-up mechanism for the errors detected be developed with the
staff of the sectors concerned.
AUDIT AND EVALUATION DIRECTORATE
3
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN AUDIT REPORT
Signature of the Chief Audit and Evaluation Executive
Original signed by Jean-Guy Desrosiers
Audit team member
Jimmy Cheung
AUDIT AND EVALUATION DIRECTORATE
4
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN AUDIT REPORT
2.0 AUDIT REPORT
2.1 BACKGROUND
The Canadian Space Agency (CSA) policy is to pay on time amounts actually owed to third
parties. Account verification mechanisms must be designed and used in such a way as to
ensure their effectiveness, with due regard for the risks related to the characteristics of each
payment requisition and to monitoring requirements.
Sampling design
In April 2003, a manual sampling plan was adopted to identify, for review and validation,
payment requisitions coding high, medium or low risk based on the nature of the transactions.
Under this plan, 13 of the 20 transaction categories were coded high-risk.
The sampling plan is based on a feasibility study that took place between October 1, 2002,
and February 28, 2003.
Sampling operation overview
Sector financial operations Accounting Services accounting clerks do a spot
clerks enter the payment check (a few key items) or a full check (according to
requisition data into SAP, the checklist under sect. 34) of documents received
then send the hard copy from the sectors under the sampling plan, then send
documents to Accounting the verified documents to Accounting Services
Services financial officers for approval before payment
Accounting Services Accounting Services financial
financial officers transfer officers verify certain key
SAP data entered by elements of the documents
sectors' financial operations verified by the clerks before
Payment
clerks into another interface affixing their initials and marking
made
for payment purposes "sect. 33" on the documents and
approving the payment in SAP
AUDIT AND EVALUATION DIRECTORATE
5
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN AUDIT REPORT
2.2 AUDIT OBJECTIVES, SCOPE AND APPROACH
The purpose of the audit was to evaluate the design and operation of the monitoring system
used to identify higher-risk transactions for review and validation.
The audit dealt with the design and operation of the sampling plan instituted in April 2003. We
also examined the breakdown of payment requisitions by transaction category for fiscal year
2007–2008.
Various audit processes were employed, including staff interviews and reviews and analyses
of documents and records. A review of the existing sampling plan and the main departmental
policies and directives was done in order to develop audit criteria.
2.3 FINDINGS, RECOMMENDATIONS AND MANAGEMENT RESPONSE
2.3.1 EFFECTIVENESS OF EXISTING SYSTEM
The Accounting, Financial Reporting and Policies Division is responsible for the Agency’s
payment requisition monitoring system, subject to internal policies and directives. Hence, we
expected to find an effective system including the following main elements:
• a sampling plan reviewed and approved annually and on a need basis by the CFO;
• a sampling plan embodying the various transaction categories and the appropriate risk
level and degree of significance;
• a sampling plan whereby transactions in each transaction category could be selected; and
• a sampling plan whereby all high-risk payment requisitions that were sensitive or for large
amounts could be identified for account verification purposes.
In general, management did draw up a sampling plan whereby higher-risk transactions could
be identified for account verification purposes. Indeed, our exam revealed that most payment
requisitions had been coded high-risk.
However, we would like to point out certain findings that require management attention.
SAMPLING PLAN
The sampling plan, including the sampling parameters (i.e. an acceptable level of confidence
and a 10% maximum acceptable error rate (MAER)) should be updated and approved,
annually and on a need basis, by the CFO.
Our discussions revealed that the annual sampling plan update by the CFO had not been
done since April 2003.
RECOMMENDATION
i) The Finance Directorate should review the sampling plan annually and on a need
basis, including the sampling parameters, and have it approved by the CFO.
AUDIT AND EVALUATION DIRECTORATE
6
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN AUDIT REPORT
MANAGEMENT RESPONSE
We agree with the recommendations and have already taken steps to rectify the situation.
SAMPLE
For account verification purposes, the sampling plan should take into account such elements
as transactions involving large sums or those involving high risk.
Our audit brought to light the lack of any sampling, for review and validation purposes, of
journal vouchers and requisitions for interdepartmental settlement, as no risk level was
assigned to these two transaction categories, which do involve large amounts. Table A gives
some representative data.
TABLE A – OTHER TRANSACTIONS
FISCAL 2007–2008
TRANSACTION NUMBER OF TRANSACTIONS
CATEGORY PROCESSED VALUE
Journal vouchers Note Transactions of up to $1 million; totalling a few million.
Interdepartmental 1,200 Transactions of up to $1.3 million; totalling $21 million.
settlements
Note: Since the information is not available, we estimated that the number of transactions processed might involve
several hundred journal vouchers (for example, miscoding, breakdown of monthly costs, PAYEs). Journal
vouchers are cross-charges.
RECOMMENDATION
ii) The Finance Directorate should review these transaction categories’ degree of risk
to enable proper transaction sampling for account verification purposes. However,
high-risk transactions and those involving large sums must be verified individually.
MANAGEMENT RESPONSE
Although our sampling plan does not deal with interdepartmental settlements (ISs) or journal
vouchers (JVs), we do have certain controls in place (verification of delegations for JVs and
ISs over $100,000). We have not made it a priority to include these verifications in the plan,
because we consider the risk associated with the transactions to be lower than for
transactions involving payments to outsider suppliers. For instance, JVs are usually
corrections or transfers between responsibility centers, where a manager has already signed
a Section-34 authorization for payment. However, starting November 1, 2008, all JVs deemed
to involve risk (change of parliamentary credit, change of ledger account affecting a ledger
involving risk [eg, hospitality, ex gratia payments, etc]) and all ISs over $100,000 will be fully
verified. These kinds of transactions will be added and documented in our next sampling plan.
AUDIT AND EVALUATION DIRECTORATE
7
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN AUDIT REPORT
2.3.2 COST-EFFECTIVENESS OF EXISTING SYSTEM
We expected to find a cost-effective system including the following main elements:
• statistical data kept;
• periodic analytical reports, in particular on errors detected, non-compliance, and the
various corrective measures to be taken regarding the errors detected in order to
periodically review the sampling plan, including the sampling parameters; and
• proper communication between Accounting Services and the managers and financial
officers and clerks in the sectors, to improve the account verification process.
There is a system in place, the Central Accounting Verification System (CAVS), that allows
statistical data to be kept, such as the number of errors detected.
However, we would like to point out certain findings that require management attention.
PERIODIC ANALYTICAL REPORTS
The sampling plan should include a periodic analysis of the findings, followed by
recommendations on the corrective measures to be taken. Moreover, an evaluation should be
done at regular intervals of the statistical sampling method chosen for each type of transaction
to make sure that it is still appropriate and adequate. That evaluation should contain
information on the causes of errors and provide a mechanism for correction and follow-up with
stakeholders.
Our audit brought to light the lack of periodic analytical reports on the CAVS, for example a
report on errors detected that could be used as a periodic review of the sampling plan, to
adjust the error rate and re-evaluate the risk associated with each transaction category; this
would have had an impact on the sampling volume for verification purposes. Better
communication about error rates between Accounting Services and the sectors concerned
ought to improve the quality of payment requisitions.
We also noted that no corrective action had been taken on the errors detected. Neither had
any follow-up been done in that regard with the sectors concerned. As regards to errors
detected in the area of travel expenses, an email was sent to the travellers and to the sector
clerks concerned. It is up to the travellers to advise their managers if necessary.
Moreover, our audit revealed that the error rate was in excess of the MAER, particularly for
travel expenses, training costs and membership fees, in spite of the corrective measures
already identified in the 2007–2008 action plan on the audit of travel, conference and
hospitality expenses.
Table B gives details on the payment requisitions audited and processed by transaction
category, together with the error rates.
AUDIT AND EVALUATION DIRECTORATE
8
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN AUDIT REPORT
TABLE B – PAYMENT REQUISITIONS
2007–2008
RISK REQUISITIONS REQUISITIONS % ERRONEOUS
TRANSACTION CATEGORY CODED AUDITED PROCESSED AUDITED REQUISITIONS ERROR RATE
Travel within Canada (more than $2,000) High
Travel within Canada (between $1,000 and Medium
$2,000)
Travel within Canada (less than $1,000) Low
Travel to the United States (more than High 1,261 4,006 7% 613 49%
$1,000)
Travel to the United States (less than Low
$1,000)
International travel High
Non-public servants travel High
Hospitality High 22 4 18%
Grants & contributions High 94 9 10%
Contracts High 2,831 391 14%
Conferences High 5 0 0%
Relocations High 23 3 13%
Training High 40 13,494 28% 15 38%
Acquisition card & petty cash purchases High 69 18 26%
Membership fees & ex gratia payments High 2 1 50%
Invoices (more than $10,000) High
Invoices (between $2,001 and $10,000) Medium
1,826 225 12%
Invoices (less than $2,000) Low
TOTAL 6,173 17,500 35% 1,279 21%
It should be noted that payment requisitions do not include salaries, journal vouchers or interdepartmental settlements.
AUDIT AND EVALUATION DIRECTORATE
9
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND
OPERATION OF THE SAMPLING PLAN AUDIT REPORT
RECOMMENDATIONS
The Finance Directorate should
i) ensure that periodic analytical reports from the CAVS are done to justify revising or
not revising the sampling plan, including the sampling parameters, and are also
used to communicate with the staff of the sectors concerned; and
ii) ensure that there is a mechanism for the correction of errors detected and follow-up
with the staff of the sectors concerned, to improve the effectiveness of the account
verification process.
MANAGEMENT RESPONSE
We agree with the recommendations and have already taken steps to rectify the situation.
AUDIT AND EVALUATION DIRECTORATE
10
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND OPERATION OF THE SAMPLING PLAN AUDIT REPORT
APPENDIX - MANAGEMENT ACTION PLAN
REF. Recommendations Responsibility Identified DETAILS OF ACTION PLAN Timetable
Organization Function
2.3.1 Effectiveness of existing
system
i) The Finance Directorate should Finance Chief In 2006–2007, Information Technology (IT) developed a new system (CAVS) to JUNE 30,
review the sampling plan annually Directorate Financial enable the accounting team to calculate erroneous transactions. The system 2009
and on a need basis, including the Officer was supposed to allow us to: (APPROVAL BY
sampling parameters, and have it 1) calculate error rates by type of error and transaction category; THE CFO AND
approved by the CFO. IMPLEMENT-
2) produce reports to implement corrective measures and follow-up with the ATION OF THE
parties involved in the verification process; and NEW
SAMPLING
3) annually adjust the parameters of our sampling plan and have it approved by
PLAN)
the CFO.
The reports did not allow us to do those things. The error definitions were not
explicit enough and the errors were not calculated correctly.
As of September 30, 2008, we had reviewed the description and classification of
errors (critical, non-critical). We met with the head of the CAVS at IT to correct
the problems identified above, which will allow us to perform activities 2) and 3).
Beginning in fiscal year 08–09 and at the end of each subsequent fiscal year,
we will evaluate the sampling plan and adjust the parameters as needed. This
plan will be approved by the CFO as of June 30 of each year to enable us to
consider all of the transactions from the previous fiscal year.
AUDIT AND EVALUATION DIRECTORATE 11
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND OPERATION OF THE SAMPLING PLAN AUDIT REPORT
REF. Recommendations Responsibility Identified DETAILS OF ACTION PLAN Timetable
Organization Function
ii) The Finance Directorate should Finance Chief Our sampling plan already takes large-sum and high-risk transactions into
review these transaction categories’ Directorate Financial consideration.
degree of risk to enable proper Officer
transaction sampling for account As for JVs and ISs, here is what we intend to do:
NOVEMBER 1,
verification purposes. However, All JVs deemed to involve risk (change of parliamentary credit, change of ledger 2008
high-risk transactions and those accounts affecting a ledger involving risk [eg, hospitality, ex gratia payments,
involving large sums must be etc]) and all ISs over $100,000 will be fully verified. For fiscal year 2008–2009,
verified individually. those transactions will be analyzed for a five-month period, ie, from November 1
to March 31.
We will then review the sampling plan for the transactions described above and JUNE 30,
for the other kinds of transactions already included in the plan at the intervals 2009
indicated in recommendation 2.3.1 i) of the action plan.
2.3.2 Cost-effectiveness of existing
system
i) The Finance Directorate should Finance Chief In 2006–2007, IT developed a new system (CAVS) to enable the accounting
ensure that periodic analytical Directorate Financial team to calculate erroneous transactions. The system was supposed to allow us
reports from the CAVS are done to Officer to:
justify revising or not revising the 1) calculate error rates by type of error and transaction category;
sampling plan, including the
sampling parameters, and are also 2) produce reports to implement corrective measures and follow-up with the
used to communicate with the staff parties involved in the verification process; and
of the sectors concerned.
3) annually adjust the parameters of our sampling plan and have it approved by
the CFO.
The reports did not allow us to do those things. The error definitions were not
explicit enough and the errors were not calculated correctly.
As of September 30, 2008, we had reviewed the description and classification of
errors (critical, non-critical). We met with the head of the CAVS at IT to correct
the problems identified above, which will allow us to perform activities 2) and 3).
a) IT will modify our CAVS by December 31, 2008; DEC 31, 2008
b) Accounting will accumulate data in the CAVS database from January 1, MAY 31,
2009, until the end of fiscal year 2008–2009; 2009
AUDIT AND EVALUATION DIRECTORATE 12
PROJECT # 08/09 01-01
AUDIT OF THE DESIGN AND OPERATION OF THE SAMPLING PLAN AUDIT REPORT
REF. Recommendations Responsibility Identified DETAILS OF ACTION PLAN Timetable
Organization Function
c) We will communicate the results of the analysis (rate of error by transaction JUNE 30,
type) to the Sector Financial Operations Division and take corrective 2009
measures to reduce the error rate (notice of information or training); and
d) The data accumulated in b) will serve as a starting point for reviewing our JUNE 30,
sampling plan in 2.3.1 i). 2009
ii) The Finance Directorate should Finance Chief Beginning in 2008–2009 and in subsequent years, we will produce reports on a 1ST REPORT
ensure that there is a mechanism Directorate Financial quarterly basis, ie, as of June 30, September 30, December 31 and May 31 (at FOR FISCAL
for the correction of the errors Officer the end of P12-2). YEAR 2009–
detected and follow-up with the staff 2010 AS OF
of the sectors concerned, to We will communicate the results of the analysis (error rate by transaction type) JULY 31,
improve the effectiveness of the in the month following the publication of the report to the Sector Financial 2009
account verification process. Operations Division and take corrective measures to reduce the error rate
(notice of information or training).
AUDIT AND EVALUATION DIRECTORATE 13
Related docs
