Docstoc

EDF Evaluation_Final_Report_August_11_2009

Document Sample
EDF Evaluation_Final_Report_August_11_2009 Powered By Docstoc
					   Evaluation of the
Environmental Damages
        Fund

        Final Report




         July 2009
Audit and Evaluation Branch              Evaluation of the Environmental Damages Fund


Report Clearance Steps

 Planning phase completed                                  November 2008
 Report sent for management response                       June 2009
 Management response received                              July 2009
 Report completed                                          July 2009
 Report approved by Departmental Evaluation Committee      July 2009
 (DEC)

Acronyms used in the report
 AEB           Audit and Evaluation Branch
 ADM           Assistant Deputy Minister
 CATI          Computer-assisted Telephone Interviewing
 CA            Contribution Agreement
 CAPE          Community Action Programs for the Environment
 CEPA 1999     Canadian Environmental Protection Act, 1999
 DEC           Departmental Evaluation Committee
 EC            Environment Canada
 ECCAP         Environment Canada Courts Awards Project
 EDF           Environmental Damages Fund
 EPAM          Environmental Protection Alternate Measures
 FTE           Full-time Equivalents
 G&C           Grant and Contribution
 MBCA 1994     Migratory Birds Convention Act, 1994
 MIS           Management Information System
 OPG           Outcome Project Group
 OPP           Outcome Project Plan
 OPSC          Outcome Project Subcomponent
 RDG           Regional Director General
 RMAF          Results-based Management and Accountability Framework
 RMS           Results Management Structure




Acknowledgments

The Evaluation Project Team, including Robert Tkaczyk and Conrad Winn, the lead
researcher from COMPAS Research Inc., led by William Blois under the direction of the
Director, Shelley Borys, would like to thank those individuals who contributed to this
project and particularly all interviewees who provided insights and comments crucial to
this evaluation.




Prepared by COMPAS Research Inc. and the Evaluation Division, Audit and Evaluation
Branch.




Environment Canada
                                            Table of Contents
EXECUTIVE SUMMARY ...................................................................................................i
1.0 INTRODUCTION....................................................................................................... 1
2.0 BACKGROUND ........................................................................................................ 1
  2.1    Environmental Damages Fund (EDF): Origins, Functions and Evaluation ....... 2
       2.1.1      Origin and Legal Setting ........................................................................2
       2.1.2      Program Delivery ..................................................................................3
       2.1.3      Eligible Organizations and Priority Projects ...........................................4
       2.1.4      National Implementation of the EDF ......................................................5
  2.2    EDF Administration .......................................................................................... 6
       2.2.1      Roles and Responsibilities ....................................................................7
       2.2.2      Monetary Contributions to Date and Prospective Financial Resources ..8
3.0 EVALUATION DESIGN ........................................................................................... 11
  3.1 General Purpose.................................................................................................. 11
  3.2 Evaluation Approach and Methodology ................................................................ 12
       3.2.1      Document Review ...............................................................................12
       3.2.2      Review of Administrative and Financial Files .......................................12
       3.2.3      Key Informant Interviews .....................................................................14
       3.2.4      Challenges and Limitations .................................................................15
4.0 FINDINGS ............................................................................................................... 16
  4.1 Relevance ............................................................................................................ 16
  4.2 Design and Delivery ............................................................................................. 19
       4.2.1      Program Delivery ................................................................................21
       4.2.2      Administration of EDF Projects............................................................25
       4.2.3      Environmental Protection Alternate Measures (EPAMs) ......................31
       4.2.4      Accountability ......................................................................................32
       4.2.5      Performance Measurement and Reporting ..........................................34
       4.2.6      Strengths and Challenges ...................................................................36
  4.3    Success ......................................................................................................... 38
       4.3.1      Project Delivery ...................................................................................42
       4.3.2      Communications and Program Promotion ...........................................44
       4.3.3      Partnerships ........................................................................................46
       4.3.4      Proposal Planning and Development ..................................................47
       4.3.5      Unintended Outcomes.........................................................................49
  4.4    Cost-effectiveness ......................................................................................... 49
       4.4.1      Minimum Project Size ..........................................................................52
5.0 CONCLUSIONS ...................................................................................................... 53
6.0 RECOMMENDATIONS ........................................................................................... 54
7.0 MANAGEMENT RESPONSE .................................................................................. 55
ANNEXES ..................................................................................................................... 63
  Annex 1: EDF Program Logic Model .......................................................................... 64
  Annex 2: Summary of Terms and Conditions for EDF Contribution Agreements ....... 65
  Annex 3: Evaluation Issues and Data Sources .......................................................... 66
  Annex 4: Bibliography ................................................................................................ 67
  Annex 5: Administrative (Project) File Assessment Guide ......................................... 69
  Annex 6: Interview Guides ......................................................................................... 73
  Annex 7: Summary of Findings .................................................................................. 87




Environment Canada
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund




EXECUTIVE SUMMARY
Environment Canada‘s (EC) Evaluation Division, Audit and Evaluation Branch,
contracted COMPAS Research Inc. to conduct an evaluation of Environment Canada‘s
Environmental Damages Fund in 2008–2009. This project was selected for inclusion in
Environment Canada‘s Risk-based Audit and Evaluation Plan 2008–2011, approved by
the Departmental Evaluation Committee in May 2008, to fulfil a requirement for an
evaluation prior to the renewal of the terms and conditions of the Environmental
Damages Fund program.

The Environmental Damages Fund (EDF) was first established in 1995 to provide a
general framework and guidelines for the management of monies stemming from the
prosecution of environmental infractions under federal law in cases where the courts
have stipulated that the awards are to be used for specific purposes. The purpose of
any contribution to the EDF is to remediate pollution-damaged natural environments in a
cost-effective way, in accordance with conditions specified by the courts or other
compensation regimes. Funding to the EDF may come from court awards or out-of-court
settlements, voluntary contributions, and contributions from national and international
liability funds.

The objectives of the EDF are achieved through financial support to appropriate
restoration projects or alternative activities that are proposed by eligible organizations.
Program delivery flows from four streams of activity and outputs related to:
communications and promotion; partnerships; proposal planning and delivery; and
project delivery. The three broad outcomes of the program are:

     Biodiversity is conserved and protected;
     Sustainable use and management of natural capital is ensured; and
     Risk to the environment posed by toxic and harmful substances is reduced.

EVALUATION ISSUES

The Evaluation of Environment Canada’s Environmental Damages Fund assessed the
relevance, success, cost-effectiveness, and design and delivery of the program. The
evaluation was designed to determine whether the program is:

     aligned with and contributing to federal government priorities, and whether it
      addresses actual needs (relevance);
     achieving or on track to achieving its intended outcomes (success);
     using the most appropriate and efficient means to achieve its outcomes (cost-
      effectiveness/alternatives); and
     designed and delivered in the best possible way (design and delivery).

METHODOLOGY

Data were collected for the evaluation using multiple lines of evidence. These included a
document/file review, a review of project files (n = 35), and a total of 116 key informant


Environment Canada                                                                        i
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


interviews conducted with Environment Canada personnel (n = 46), funding recipients (n
= 40), judicial respondents (prosecutors, judge and federal counsel) (n = 6),
environmental academics (n = 9) and potential funding recipients (n = 15). Key
challenges to conducting this evaluation included the absence of a centralized
information management system, limited information with which to assess cost-
effectiveness, and the absence of a comparison group for funded projects.

EVALUATION FINDINGS

Overall, evaluation findings indicate that the EDF is highly relevant to the Government‘s
mandate and departmental priorities. The program, however, is not adequately
resourced to ensure its consistent regional implementation and in particular, its
promotion among the judiciary and potential project proponents. The program
nonetheless demonstrates progress toward the achievement of outcomes in a cost-
effective manner. Program improvements may be realized through the dedication of
adequate resources for program delivery in all regions, refinement or development of
existing tools and processes (e.g., application forms, training), the initiation of a
communications strategy targeting key stakeholder groups, and stronger national
oversight to ensure consistency of program delivery.

Evaluation findings are summarized in the following sections by evaluation issue.

a) Relevance

The EDF is highly relevant in terms of supporting federal priorities related to
environmental enforcement and departmental priorities related to natural resource
rehabilitation, environmental quality improvement and conservation, research and
development, and education and awareness. The vast majority of key informants feel
the EDF serves an important role and does not duplicate other existing programming.
As well, the evidence suggests the program is the only federal mechanism of its kind for
managing court awards and financial penalties arising from federal environmental
enforcement activities, is expected to play an expanded role in the management of court
awards prosecuted under different federal statutes, and supports the court‘s use of
creative sentencing in environmental damage cases.

b) Design and Delivery

The EDF is generally perceived to be delivered as designed, although low familiarity with
the program and a lack of supporting documentation for its use in the courts may reduce
its application when prosecuting environmental infractions.

With the exception of Atlantic Canada, the absence of dedicated resources has resulted
in less than full implementation of the EDF program in the regions, such that the pace at
which funds are being diverted to the program through the courts outstrips the
program‘s ability to allocate funds to EDF projects. The solution to uneven regional
implementation is generally thought to involve an increase in human resources
dedicated to the program, which should improve program promotion in the courts and
among potential applicants, accessibility of staff to support funding recipients and the
ability of program staff to respond to court-imposed funding conditions.




Environment Canada                                                                          ii
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


Awareness of the EDF is low among both the courts and potential applicants to the
program, and this appears to be largely the result of a lack of dedicated resources. Low
program awareness is thought to limit the number of court awards being diverted to the
program, lead to a higher rate of court-imposed conditions, and result in fewer
applications for EDF funding. Possible strategies for raising awareness among the
courts include training for prosecutors, communications activities targeting judges, and
engaging enforcement officers in efforts to educate the judiciary.

The evaluation also found the following with respect to various specific aspects of the
EDF‘s delivery:

      The application process: is generally perceived to be appropriate, although
       some funding recipients and EC personnel feel it is burdensome. EDF program
       personnel are thought to be quite helpful in navigating the application process,
       but their uneven accessibility and expertise may dampen the effectiveness
       support to applicants.
      Partnership requirements: Program documentation is not clear about whether
       partnerships are a requirement for EDF-funded projects and assessments of the
       value of partnerships are mixed (i.e., leveraging of partner funding/expertise
       versus the potential for conflict with or poor performance of partners).
       Nonetheless, a slim majority of EDF-funded projects were implemented in
       partnership, and the most frequently cited partner contributions included funding,
       service delivery or research.
      EDF terms and conditions: The vast majority of projects are in compliance with
       most of the terms and conditions for EDF contributions, although high rates of
       incomplete information and conflicting evidence from other sources do not allow
       the evaluators to reach a conclusion concerning compliance with five of these
       conditions.
      Environmental Protection Alternate Measures (EPAMs): EC personnel and,
       to a lesser extent, judicial respondents report very low levels of familiarity with
       the function and/or use of EPAMs. Of those respondents who are familiar with
       EPAMs, opinion was divided as to whether their use would grow or diminish in
       the future.
      Roles and responsibilities: There exists a great deal of program
       documentation outlining the roles and responsibilities of the different parties
       involved in the delivery of the EDF program. A number of respondents, however,
       expressed some confusion in this regard and felt there was a need for more
       consistency and greater oversight in the delivery of the program across regions
       (noted by EC personnel) and EDF program staff (noted by recipients).
      Performance measurement and reporting: Intended deliverables and impacts
       of the EDF program and funded projects are generally clear and performance
       data are collected for most projects with respect to activities, outputs and
       outcomes. In most regions, however, no systematic process exists for
       communicating program results to senior managers and decision-makers.




Environment Canada                                                                        iii
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


c) Success

The EDF is widely perceived by recipients and Environment Canada personnel to be
successful in the achievement of program outcomes—a finding confirmed by
administrative files demonstrating clear achievement of project deliverables and
outcomes in a majority of cases. Evidence from key informants and project files
suggests that the program is most successful in contributing to restoration objectives,
and is somewhat successful in contributing to education and awareness and research
and development outcomes as well. The primary factors associated with the
achievement of different outcomes concern the resources, number of projects and
number of applications targeting the priority area.

The existence of clear documentation pertaining to the role recipients are required to
play in the financial oversight of funded projects, as well as the high rates of projects
conforming to audit and payment authority specifications, suggests the program is being
administered with good financial oversight and accountability. While generally clear and
well understood, the transparency of the funding process could be enhanced through
more consistent staff support and better communications about court awards and
funding decisions between Enforcement and EDF personnel.

Resources to support the administration of the program overall are not tracked in any
systematic fashion, although regions are tracking the number and value of court awards
and contribution agreements. Past research suggests that overall financial
accountability and transparency would be enhanced through the development of an
information management system, which is already under development.

There has been no apparent progress in increasing awareness of the program among
the legal community and potential applicants or in increasing the court‘s use of the EDF
as a sentencing tool. Although there has been some effort to make information related
to EDF projects in the Atlantic region public, this information was incomplete and out-of-
date and no similar information was found for other regions.

Efforts have been made to expand the use of EDF in the context of federal legislation
controlled by other departments. Although implicated departments (i.e., DFO and
Transport Canada) are involved in the technical review of project applications,
information transfer between federal partners could be improved by sharing more
information on final funding decisions.

The EDF program employs a number of controls and processes to ensure the quality
and feasibility of project proposals. Evidence suggests also that guidance
documentation and program staff support are contributing to the acquisition of proposal
and planning skills by community groups, as evidenced by the high rates with which
projects achieve intended deliverables, the consistency of proposals with EDF
objectives, the use of partnerships, and the ability of projects to leverage partner
resources.

d) Cost-effectiveness

Overall, the evidence suggests that the EDF is cost-effective. Projects have been
successful at leveraging funds from other sources and some limited file review evidence


Environment Canada                                                                        iv
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


suggests that individual projects are more likely than not to be evaluated as cost-
effective. At the program level, no other federal programs exist to manage specified-
purpose funds to carry out environmental restoration activities and overall administrative
spending is low, particularly in light of the national scope and specificity of court
conditions related to funded projects.

Considering the low average dollar value of many EDF contribution agreements, the
evaluators also explored the merits of introducing a minimum threshold value for project
funding in order to increase overall program efficiency. The evaluation evidence is
inconclusive concerning the advantages of such a minimum value for EDF contribution
agreements. While cost-efficiencies might be realized by imposing a minimum value for
EDF-funded projects, doing so could impair small projects whose requirements for
restoration or other environmental contributions could be lower than a potential award
threshold.

RECOMMENDATIONS

Six recommendations were developed for the ES Board based on the evaluation
findings and conclusions.

1.   An allocation of dedicated staff and budget to support the ongoing
     management and administration of the EDF program should be considered.
     Uneven regional implementation of the EDF program can be traced, at least in part,
     to the absence of dedicated resources to support its implementation, and this in turn
     may affect the consistency of support to recipients. Further, resource constraints
     are felt to represent a key barrier to the proper promotion of the program among
     potential applicants, thus limiting the pool of potential funding recipients able to
     carry out court requests through the program.

2.   Standardized tools and program documentation for managers and funding
     recipients should be updated to ensure consistent program implementation.
     In updating these tools, consideration should be given to:
       clarifying partnership requirements for EDF-funded projects;
       the development of standard training and resources for managers that clearly
        outline program governance (i.e., accountability to courts versus Department),
        clarify the roles and responsibilities of program staff and enforcement personnel
        and enhance the consistency of program support to EDF project proponents; and
       the development of standard indicators to guide managers and project
        proponents in measuring project success.

3.   Means of streamlining the program application process and forms should be
     explored to reduce the burden on recipients and enhance consistency with other
     community partnership programs. To this end, the EDF could consider research
     (surveys, focus groups, etc.) with recipients to explore ways in which the application
     form could be improved. Alternatively, a comparison with the application forms for
     similar funding programs (e.g., EcoAction) may yield information to guide such
     changes (e.g., word limits and question simplification).

4.   A communications plan should be developed and implemented to promote
     the EDF among potential recipients, the judiciary and enforcement officers.


Environment Canada                                                                        v
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


     The communications plan should consider the development of standardized
     communications materials that can be modified, as appropriate, to address regional
     circumstances, as well as the development of national communication activities to
     enhance efficiency and consistency of messaging nationally. Efforts to promote the
     program among potential recipients would likely yield higher quality and a greater
     variety of project proposals. As well, communications efforts targeted toward judges
     and, to a lesser extent, enforcement personnel may increase the number of awards
     and other contributions to the EDF and help to reduce the number of court-
     prescribed funding restrictions.

5.   A centralized national information management system should be developed
     to track financial and performance information for EDF contribution agreements
     across regions. The development and implementation of such a system is essential
     for proper measurement, evaluation and reporting of program performance.

6.   A formal process for annual EDF performance measurement and reporting
     should be implemented. No consistent national process exists for reporting on the
     program‘s impacts and effectiveness. In particular, such a reporting process is
     necessary to monitor progress toward full national implementation and the timely
     disbursement of funding in proportion to program awards. As a first step in
     developing this process, the program logic model should be revised to better reflect
     intended outcomes and more clearly distinguish outcomes from activities and
     standard corporate functions (e.g., financial oversight).

MANAGEMENT RESPONSE

The ES Board accepts the evaluation and its recommendations, and the EDF program
has provided a plan to implement the following management actions in response to the
evaluation recommendations within the context of EDF program renewal.

Note to Reader
Bill C-16, the Environmental Enforcement Act, received Royal Assent on June 18th,
2009. Among other things, this Bill amends six EC statutes and three Parks Canada
statutes to direct all fines to the EDF. Once these amendments come into force new
minimum fines will start at $5,000 for individuals and $25,000 for corporations and
maximum fines will increase to $1 million for individuals and $6 million for corporations.
The amendments also allow for a portion of the fines to be used for administering the
EDF Program.

It is anticipated that these amendments could significantly increase the monetary value
and number of awards directed to the EDF and these changes will be factored into the
renewal of the program.

1.   An allocation of dedicated staff and budget to support the ongoing
     management and administration of the EDF program should be considered.
     Uneven regional implementation of the EDF program can be traced, at least in part,
     to the absence of dedicated resources to support its implementation, and this in
     turn may affect the consistency of support to recipients. Further, resource
     constraints are felt to represent a key barrier to the proper promotion of the



Environment Canada                                                                           vi
Audit and Evaluation Branch                 Evaluation of the Environmental Damages Fund


     program among potential applicants, thus limiting the pool of potential funding
     recipients able to carry out court requests through the program.

The ES Board agrees with this recommendation.

Under the Environmental Enforcement Act (Bill C-16), which is expected to come into
force in 2011–2012, the program will have the option to use funds directed to the EDF
for administrative purposes.

The business case to support program renewal will be requesting administrative
resources, including dedicated staff and operational and management costs. The
request for program renewal will be presented in September 2009.

Securing resources for the EDF will ensure that each region has dedicated staff to
deliver the program. This in turn will help to bring national consistency in the delivery of
the program and ensure that proper support is available to clients. These resources will
enable staff to effectively promote the EDF program to targeted audiences and diversify
the pool of potential funding recipients.

    Timeline                       Deliverable                       Responsible Party
 September 2009      Present the EDF business case and               Director, Outreach
                     documentation related to program
                     renewal.


2.   Standardized tools and program documentation for managers and funding
     recipients should be updated to ensure consistent program implementation.
     In updating these tools, consideration should be given to:
       clarifying partnership requirements for EDF-funded projects; program
          documentation is not clear about whether partnerships are a requirement for
          EDF-funded projects and assessments of the value of partnerships are mixed;
       the development of standard training and resources for managers that clearly
        outline program governance (i.e., accountability to courts versus Department),
        clarify the roles and responsibilities of program staff and enforcement personnel
        and enhance the consistency of program support to EDF project proponents; and
       the development of standard indicators to guide managers and project
        proponents in the measurement of project success.

The ES Board agrees with this recommendation.

As part of the program renewal package, the program is recommending the approval of
the EDF Management Framework and the Funding Agreement Terms and Conditions
for the Environmental Damages Fund Program. The management framework provides
an overview of how the program works, the program logic model, performance
measurement, and reporting and risk management strategies. The funding agreement
terms and conditions set forth the terms and conditions under which the EDF program
will provide funding for projects.

The program renewal package outlines a number of commitments to streamline
administrative processes to ensure consistency across all regions:


Environment Canada                                                                        vii
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund



i)     Development of operational guidelines to outline program governance, roles and
       responsibilities of program staff and clarify partnership requirements for the EDF
       program, as partnerships are considered an advantage but not essential.
ii)    Development of regional management plans that define top priorities for potential
       EDF projects and enable the program to direct funding to local environmental
       priorities that represent the best use of funds in response to environmental damage
       or damage-to-wildlife incidents. Regional management plans will be prepared in
       consultation with EC regional management teams and include other government
       departments as required. The plans will be approved by the RDG.
iii)   Development of standard project indicators to guide managers and project
       proponents in measuring project success. These indicators will contribute to the
       evaluation of program results.

    Timeline                        Deliverable                     Responsible Party
 September 2009        Present the EDF business case and        Director, Outreach
                       documentation related to program
                       renewal
 November 2009         Develop program operational              Director, Outreach
                       guidelines
 Fall 2009             Develop regional management plans        Regional Directors General
 Fall 2009             Conduct project indicators workshop      Director, Outreach
                       to develop standard project indicators


3.     Means of streamlining the program application process and forms should be
       explored to reduce the burden on recipients and enhance consistency with other
       community partnership programs. To this end, the EDF could consider research
       (surveys, focus groups, etc.) with recipients to explore ways in which the application
       form could be improved. Alternatively, a comparison with the application forms for
       similar funding programs (e.g., EcoAction) may yield information to guide such
       changes (e.g., word limits and question simplification).

The ES Board agrees with this recommendation.

The EDF program will take a more directed approach to soliciting applications for
projects. As outlined in the response to the second recommendation, the program will
develop regional management plans that define top priorities for potential EDF projects
and enable the program to direct funding to local environmental priorities that represent
the best use of funds in response to environmental damage or damage to wildlife.
Project proposals will be solicited on an on-going basis as EDF awards are received.

As a part of the EDF program renewal, the program has committed to streamlining the
EDF administrative processes, including reducing the burden of application and
reporting processes on funding applicants. The EDF program will participate in the
Departmental Action Plan to Reform the Administration of Grants and Contributions,
which has the goal of simplifying and streamlining application processes and improving
forms for grants and contributions (G&Cs), while strengthening accountability. It is
important to note that the scope and timelines of the commitments in relation to the



Environment Canada                                                                        viii
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


Departmental Action Plan to Reform the Administration of Grants and Contributions are
beyond the program‘s control.

     Timeline                        Deliverable                          Responsible
                                                                             Party
     Fall 2009          Develop Regional Management Plans              Regional Directors
                                                                            General
 Fall/winter      Review application and reporting processes           Director, Outreach
 2009–2010
 2011–2012        Adopt applications and reporting processes           Director, Outreach
                  resulting from G&C reform

4.   A communications plan should be developed and implemented to promote
     the EDF among potential recipients, the judiciary and enforcement officers.
     The communications plan should consider the development of standardized
     communications materials that can be modified, as appropriate, to address regional
     circumstances, as well as the development of national communications activities to
     enhance efficiency and consistency of messaging nationally. Efforts to promote the
     program among potential recipients would likely yield higher quality and a greater
     variety of project proposals. As well, communications efforts targeted toward judges
     and, to a lesser extent, enforcement personnel may increase the number of awards
     and other contributions to the EDF and help to reduce the number of court-
     prescribed funding restrictions.

The ES Board agrees with this recommendation.

As part of the program renewal strategy, the program has committed to developing a
promotional strategy that targets judges and prosecutors, enforcement officials from
other federal government departments and potential funding recipients. In addition, a
national communications plan will be created to ensure national consistency in efforts to
promote the EDF program.

Overview of the EDF Promotional Strategy

Judges and Prosecutors
    The program will work with the Department‘s Legislative Affairs Branch to
     disseminate information to the judicial community, increase judges‘ awareness of
     the EDF and explain how it can be used as an effective and innovative mechanism
     for sentencing. The Legislative Affairs Branch has a contribution agreement with
     the National Judicial Institute, an arms-length third-party organization dedicated to
     the education of judges, to develop materials related to environmental law aimed
     at raising judges‘ awareness of these issues.
    Develop a Web page geared towards the legal community to help promote the
     EDF among Crown prosecutors and defence lawyers.
    Work with the National Environmental Prosecutions Coordinator at the Department
     of Justice to disseminate information on the EDF to Crown prosecutors.




Environment Canada                                                                      ix
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


Enforcement
    Work with EC‘s Enforcement Program, both at the national and regional levels, to
     maintain good working relationships and determine what information, training and
     tools would best serve their needs.
    Update the EDF fact sheet and brochure to capture current program information
     and ensure its effectiveness as a tool for enforcement officers to promote the
     program among prosecutors.
    Develop an outreach plan for other federal departments whose enforcement
     officers and legislation can be used to direct funds to the EDF. Tools could include
     fact sheets, brochures, presentations and training modules.

Potential Recipients
    Undertake targeted regional promotion among key groups that could potentially
     undertake EDF projects.
    Implement improvements to the EDF Web site
    Participate in the Community Action Programs for the Environment (CAPE) Web
     portal, which offers a single entry point to all EC community funding programs.

     Timeline                         Deliverable                     Responsible Party

 Fall 2009          Implement national EDF communications plan        Director, Outreach
 Summer 2009        Develop EDF promotional strategy                  Director, Outreach
 July 2009          Participate in the CAPE Web portal                Director, Outreach
 August 2009        Develop and distribute EDF fact sheet             Director, Outreach
 Fall 2009          Disseminate EDF letter to prosecutors by the      Director, Outreach
                    Department of Justice National Environmental
                    Prosecutions Coordinator
 Fall 2009          Complete Web site improvements                    Director, Outreach
 Winter/spring      Develop enforcement training materials and        Director, Outreach
 2010               tools
 Spring/summer      Develop information in conjunction with the       Director, Outreach
 2010               Judicial Outreach Program for the National
                    Judicial Institute
 Fall/winter        Develop EDF brochure                              Director, Outreach
 2009–2010


5.   A centralized national information management system should be developed
     to track financial and performance information on EDF contribution agreements
     across the regions. The development and implementation of such a system is
     essential for proper measurement, evaluation and reporting of program
     performance.

The ES Board agrees with this recommendation.

As a part of the Department‘s G&C reform initiative, an online application and
information management system is being developed for its G&C programs. This system
will improve program efficiency, enhance alignment with departmental priorities and
improve the ability to report collectively on the results of departmental funding programs.



Environment Canada                                                                         x
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


This system is scheduled to be in operation by 2011–2012, after which the EDF
program will adopt it to collect program data.

In the meantime, the program acknowledges that a consistent and effective mechanism
for collecting and tracking program information and results is required. In April 2009, the
program implemented a Management Information System (MIS), cloned from the
existing EcoAction Community Funding Program MIS. The MIS allows program staff to
track project information including recipient information, funding agreements, project
funding, and project results. Planned modifications and improvements will allow the
program to track the source of monetary contributions, the legislation used and whether
EC enforcement officials were involved. The system will facilitate reporting and allow
project managers to track communications with funding recipients. The use of this tool
will ensure consistency of data collection across the regions and significantly improve
the program‘s ability to track and report results. It has been launched in all EC regions
and is currently being populated with historical as well as current project information.
This system will be used until the new G&C online system is up and running.

 Timeline                            Deliverable                    Responsible Party

 Fall 2009              Complete implementation of MIS            Director, Outreach
 2011–2012              Adopt the departmental G&C                Director, Outreach
                        information management system


6.   A formal process for annual EDF performance measurement and reporting
     should be implemented. No consistent national process exists for reporting on the
     program‘s impacts and effectiveness. In particular, such a reporting process is
     necessary to monitor program progress toward full national implementation and the
     timely disbursement of funding in proportion to program awards. As a first step in
     developing this process, the program logic model should be revised to better reflect
     intended outcomes and more clearly distinguish outcomes from activities and
     standard corporate functions (e.g., financial oversight).

The ES Board agrees with this recommendation.

The program recognizes that a formal reporting process is required to reflect the
success and progress of the EDF program. In response to this need, a revised logic
model, performance measurement framework and reporting strategy have been
developed and included in the Management Framework as part of the program renewal.
The program has also committed to developing project-specific indicators to assist in the
effective measurement of project results.

Program results will be collected by NCR program staff on an annual basis and
presented to the ES Board. The information will capture the number of projects that
were funded and the results achieved, including environmental impacts. The program
will also demonstrate its administrative effectiveness with respect to national
implementation as well as the timely allocation of funding.




Environment Canada                                                                       xi
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


A departmental performance measurement framework has also been developed and the
EDF has identified three expected results and three corresponding indicators against
which it will report on an annual basis in the Departmental Performance Report.

The expected results are:
       Recognition by judges of the value of the Environmental Damages Fund in
          helping communities recover from environmental damage and damage-to-
          wildlife incidents.
       Engagement of communities directly affected by environmental damage and
          damage-to-wildlife incidents.
       Natural environments affected by environmental damage are restored.

The indictors that will be used to measure these results are:
        Annual number of court awards directing payment to the Environmental
           Damages Fund
        Number of projects completed
        Proportion of area restored.

As previously mentioned the program has developed an electronic management
information system that will be used to track program results and help facilitate reporting
and ensure consistency in reporting practices across all regions. The system will be
used to roll up annual results and for program evaluation purposes.

 Timeline                             Deliverable                     Responsible Party

 September 2009     Present the EDF business case and                 Director, Outreach
                    documentation related to program renewal
 Fall 2009          Conduct project indicators workshop to            Director, Outreach
                    develop standard project indicators
 Fall 2009          Complete implementation of MIS                    Director, Outreach




Environment Canada                                                                      xii
1.0 INTRODUCTION
The Evaluation Division of the Audit and Evaluation Branch of Environment Canada
commissioned COMPAS Inc. to conduct an evaluation of Environment Canada‘s
Environmental Damages Fund (EDF) program. This project was selected for inclusion in
Environment Canada‘s Risk-Based Audit and Evaluation Plan 2008–2011, approved by
the Departmental Evaluation Committee in May 2008, to fulfil a requirement for an
evaluation prior to the renewal of the terms and conditions of the EDF program. The
previous evaluation was completed in 2002.

This document presents the findings and recommendations of the evaluation.1 It is
organized as follows: background, including the origins and administrative setting of the
Environmental Damages Fund in Section 2; evaluation design, including issues and
sources of evidence in Section 3; findings, including sub-sections devoted to relevance,
design and delivery, terms and conditions for EDF contribution agreements, success,
cost-effectiveness and Environmental Protection Alternate Measures, the negotiated
settlements known as EPAMs, in Section 4. Sections 5 and 6 respectively present the
conclusions and recommendations stemming from the evaluation.


2.0 BACKGROUND
The overarching goal of the Environmental Damages Fund (EDF) is to improve the
quality of Canada‘s environment. This is accomplished through the financial support of
projects that restore environments damaged by spills of hazardous substances,
discharges of industrial and municipal wastes, and the disruption of fragile habitats by
human activities, as well as studies and conservation actions with the special purpose of
protecting and conserving migratory birds and species at risk.

COMPAS Research Inc. was contracted to undertake an evaluation in concert with the
Evaluation Division of the Audit and Evaluation Branch of Environment Canada.
COMPAS performed the evaluation with oversight from an evaluation committee that
consisted of representatives of the Evaluation Division of the Audit and Evaluation
Branch and the EDF program.




1
This report was co-written by COMPAS Inc. and Environment Canada‘s Evaluation Division,
 Audit and Evaluation Branch.


Environment Canada                                                                        1
2.1     Environmental Damages Fund (EDF): Origins, Functions
        and Evaluation

2.1.1 Origin and Legal Setting

The accidental release or deliberate discharge of environmental contaminants such as
oil, chemical products, and industrial wastes can have a variety of adverse effects on the
biophysical components of the environment. When pollution incidents occur resulting in
environmental damage, courts may use a number of pieces of federal environmental
legislation to obtain compensation from the responsible party for damages incurred.
The compensatory awards received as a result of prosecutions may be directed to the
Environmental Damages Fund for the purpose of rehabilitating or restoring damaged
environments.

In 1993, Environment Canada was requested to work with the Treasury Board of
Canada Secretariat to develop guidelines for the management of court orders, awards
or other financial compensation for damage to the environment. In response to this
request, the Environmental Damages Fund was first established in 1995 to provide a
general framework and guidelines for the management of monies stemming from the
prosecution of environmental infractions under federal law in cases where the courts
                                                                     2
have stipulated that the awards are to be used for specific purposes (see Annex 2 for a
summary of EDF guidelines for contributions). Through the EDF, sub-accounts are
established for individual awards, with the funds being dispersed as appropriate to
support the specified purpose for which the award was originally made. The funds can
be dispersed either through contributions or contractual payments that are consistent
with the purpose to which the funds are required to be applied.3

In addition to court awards, funds can be diverted to the EDF through the use of
Environmental Protection Alternate Measures (EPAMs). A 2007 process analysis of the
EDF program describes EPAMs as:
    ―an alternative to court prosecution for a violation of the Canadian Environmental
    Protection Act, 1999. EPAMs divert the accused away from the court process
    after charges have been made by allowing for the negotiated settlement of
    certain offences between an accused and the Crown after charges are laid but
    before a prosecution is conducted. The accused must consent to negotiate and
    must accept responsibility for the offences. Under the EPAM process an
    agreement is filed with the court that outlines the agreement penalties. Although




2
  The EDF is a specified-purpose account, which is an account that is opened in the general
  ledger to ensure that funds are used only for the purposes for which they were received and that
  allows managers to better control and manage these funds (Treasury Board of Canada
  Secretariat. 1995. Policy on Specified Purpose Accounts. Ottawa.)
3
  Environment Canada. Undated. Environmental Damages Fund (EDF) Applicant’s Guide. Ottawa.


Environment Canada                                                                              2
    EPAMs have resulted in funds to the Environmental Damages Funds, the
    majority of the funds continue to result from court-directed contributions.‖4

The purpose of any contribution to the EDF is to achieve remediation of pollution-related
damages to the natural environment in a cost-effective way, in accordance with
conditions specified by the courts or other compensation regimes. Funding to the EDF
may come as a result of court awards or out-of-court settlements, voluntary
contributions, and contributions from national and international liability funds, such as
the Ship-source Oil Pollution Fund5 and the International Oil Pollution Fund.6 Court
awards are directed to the EDF through various legislative acts, including the Fisheries
Act; the Canada Shipping Act, 2001; the Migratory Birds Convention Act, 1994; and the
Canadian Environmental Protection Act, 1999.7


2.1.2 Program Delivery

The goals and objectives of the EDF are achieved through financial support to
appropriate restoration projects or alternative activities that are proposed by eligible
organizations. To receive funding, project proponents must demonstrate that the project
is ―technically sound, feasible, and cost-effective‖ and that the proponents themselves
possess ―the organizational capacity, skills and knowledge necessary to successfully
                                                 8
implement and carry out the planned activities.‖

Activities and outputs related to the delivery of the program fall within one of four
categories:

       Communications and promotion: including the preparation of communication
        plans, promotional products, the EDF Web site and media alerts for project
        approvals;
       Partnerships: involving the coordination of departmental and interdepartmental
        technical review teams;
       Proposal planning and delivery: which concerns the provision of expertise,
        tools and advice to recipients; and




4
  PWGSC Government Consulting Services. 2007. Environmental Damages Fund Process
  Analysis. Ottawa.
5
  ―The SSOP Fund is the Canadian insurance fund established for the purpose of ensuring the
  payment of claims for marine oil pollution that originates from ships. The system is designed to
  cover the risk of non-payment by the ship owner who is responsible for pollution. In addition, it
  covers claims for damage and clean-up costs where the identity of the ship that caused the
  discharge of oil cannot be established - i.e., mystery spill.‖
  (http://www.ssopfund.gc.ca/english/faq.asp, accessed June 2009).
6
  Environment Canada. 2008. The Environmental Damages Fund: A Program Overview.
  Presentation deck. Ottawa.
7
  Ibid.
8
  Environment Canada. 2005. Environmental Damages Fund Results-based Management and
  Accountability Framework. Ottawa.


Environment Canada                                                                                    3
        Project delivery: including project approvals, financial tracking and
         management of contributions, the development of contribution agreements, and
         auditing and field monitoring.

EDF activities and outputs contribute to the successive attainment of short and medium-
term outcomes, which in turn are logically associated with the achievement of three
long-term outcomes:

      Biodiversity is conserved and protected;
      Sustainable use and management of natural capital is ensured; and
      Risk to the environment posed by toxic and harmful substances is reduced.

A detailed logic model of the EDF program, which provides a visual representation of the
linkages between the program‘s activities and outputs and the achievement of its
                                   9
outcomes, is presented in Annex 1.


2.1.3 Eligible Organizations and Priority Projects

Organizations that are eligible to receive EDF funding include non-governmental
organizations (e.g., community-based environmental groups), Aboriginal groups,
                                                                        10
universities, and provincial, territorial and municipal governments. EDF funding is used
to support projects and initiatives that fall within one of the four following priority areas:11

        Restoration: Activities that restore the same type, quality and value of natural
         resource lost in the locale in which the incident occurred or in another locale, or
         that restore the same type of natural resource, but of different quality in the
         locale where the incident occurred.
        Environmental Quality Improvement: Activities that improve or enhance
         different natural resources at the locale in which the incident occurred, or that
         improve or enhance environments previously degraded by pollutant discharges
         or other human-induced alterations (e.g., tidal barriers, dike lands).
        Research and Development relating to Damage Assessment and
         Restoration: Involving the assessment or research of environmental damage in
         support of restoration planning (including the long-term effects and cumulative
         impacts of pollutant discharges), increased understanding of limiting factors for
         migratory bird populations or the development of environmental damage



9
   Ibid.
10
   The EDF Applicant’s Guide also stipulates a number of types of activities that are not eligible for
    EDF funding, such as existing activities; mandates by law or other levels of government;
    recreation, tourism, or beautification; formal curriculum material preparation; annual or regular
    events/campaigns; conferences and workshops; core funding; infrastructure, particularly as it
    relates to municipal, provincial, or federal government program areas; incentive or rebate
    programs; lobbying or advocacy activities; containment and clean-up of environmental spills;
    restoration of contaminated sites; and efforts outside of Canada (Environment Canada.
    Undated. Environmental Damages Fund (EDF) Applicant’s Guide. page 3. Ottawa.).
11
   Environment Canada. Undated. Environmental Damages Fund (EDF) Applicant’s Guide. page
    4. Ottawa.


Environment Canada                                                                                  4
          assessment and restoration methods (including techniques for the valuation of
          damage, restoration alternatives, environmental management plans, etc.).
         Education and Awareness: Activities that promote education related to
          environmental damage restoration (including training for the assessment and
          restoration of damage), increased awareness and compliance with
          environmental regulations, or community capacity-building and environmental
          stewardship in support of environmental damage restoration.


2.1.4 National Implementation of the EDF

At the time of a 2002 evaluation of the EDF, Atlantic Canada was the only region where
the program was fully operational. As such, the 2002 evaluation focused only on this
region. The four principal recommendations stemming from the 2002 evaluation aimed
at encouraging broader and more consistent implementation of the EDF were:

         The development of an education strategy and campaign aimed at prosecutors,
          the Canadian Bar Association, and judges,
         Encouraging regional directors general to channel monetary penalties to the
          EDF, irrespective of the statute under which they were assessed,
         To seek a formal legal opinion on the extent to which the EDF could be utilized
          under the Fisheries Act, the Canada Shipping Act, the Migratory Birds
          Convention Act, 1994, and the Species at Risk Act, and
         That the EDF be monitored nationally within the Environmental Protection
          Service by the director general of national programs and regionally by regional
          directors general to ensure that it continues to operate as a unique, accountable,
          transparent and effective tool.

In the years following the 2002 evaluation, results of follow-up exercises to ascertain
progress toward achieving these management actions revealed that management
actions related to the first two recommendations had been implemented. Management
actions related to the latter two recommendations, however, are not yet fully
implemented and so are considered in the context of the current evaluation. In
particular, while the EDF has clarified the statutes under which prosecution may make
use of the provisions of the Environmental Damages Fund, it is unclear how or even if
this information is being communicated to the Department of Justice‘s Regional Office
Group Heads for Prosecution for dissemination to all prosecutors within their areas of
responsibility. As well, the management response to the fourth recommendation also
noted the need to further investigate and clarify applications to Environmental Protection
Alternate Measures (EPAMs), but it remains unclear whether or not and to what extent
this action may have been taken.12




12
     Environment Canada. 2009. 2008 Request for Update on Management Response to Address
     Evaluation Recommendations. Internal document. Ottawa.


Environment Canada                                                                          5
There exist no national or regional resources dedicated to the delivery of the EDF in the
Department‘s regions. The EDF is delivered by the Community Funding program, which
has responsibility for the delivery of several other funding programs including EcoAction,
Ecosystems Initiatives and various other education and engagement activities.13
Resources to manage and deliver the EDF are drawn from A-base funding allocated for
the delivery of these other community-based funding programs. Consequently, the EDF
has been rolled out to differing extents in the other Environment Canada regions (Pacific
and Yukon, Prairie and Northern, Ontario, and Quebec) as not all regions have had the
same capacity and ability to support the program‘s administration. The absence of
dedicated resources for program administration has also meant that administrative costs
for program delivery have not been tracked in a consistent fashion.

Furthermore, implementation of the EDF may have been delayed somewhat in Ontario
Region as a result of the existence and use of a competing program. Until 2007, when
imposing fines for transgression of the Canadian Environmental Protection Act, 1999
(CEPA 1999) in Ontario, the courts could use the Environment Canada Courts Awards
Project (ECCAP), a separate mechanism for making awards that was available only in
Ontario Region. Under ECCAP, EC enforcement officers in Ontario would recommend
to the court an appropriate beneficiary of an award. The award went directly to the
recipient organization,14 which was not subject to the oversight possible under awards
transmitted through the EDF.


2.2     EDF Administration

EDF activities are undertaken under the direction of the Ecosystem Sustainability Board.
Within the departmental Program Activity Architecture, EDF is under the sub-activity
Outreach Program and sub-sub-activity Community-based Funding. Funding
agreements are established under the terms and conditions of a class grants and
contributions authority Contributions to support Environment and Sustainable
Development Initiatives. It is designed to be a national program, delivered regionally
across the Department, with representatives in all five regions of the country.




13
   The Community Funding program also had responsibility for delivering the One-Tonne
   Challenge program within the timeframe investigated in the current evaluation. This program
   was cancelled in 2005–2006.
14
   Morrison, S. 2007. Environment Canada Courts Awards (Pilot) Project: Progress Report.
   Powerpoint presentation.


Environment Canada                                                                               6
2.2.1 Roles and Responsibilities

Applications for EDF funding are reviewed for their scientific and technical merit by
experts from Environment Canada, and in some cases, experts from other government
departments. Funding recommendations are made by the EDF regional program
managers to the regional directors general of Environment Canada.

Over the time period examined for the current evaluation, departmental activities were
organized using a results management structure (RMS) divided into: Outcome Project
Subcomponents (OPSCs), the smallest element of the results structure where short-
term results are articulated; Outcome Project Plans (OPPs), which embody medium-
term results statements to which OPSCs were attached; and higher order outcomes
represented by Outcome Project Groups (OPGs), comprised of groupings of OPPs. All
levels of the RMS were aligned with the Department‘s strategic outcomes.

Formal responsibility for program administration and accountability rests with the
Minister of the Environment, as detailed in Table 2.1. The OPG lead (i.e., director
general) and regional directors general bear responsibility for broad program oversight
of such things as project solicitation, funding approval and allocation, and overall
program performance. Mid-level accountabilities for national implementation,
performance reporting, oversight of audits and evaluations, and guidance for program
management, delivery and promotion rests with the relevant OPP and OPSC leads (i.e.,
director and manager, respectively). The day-to-day operations of the fund are managed
by regional program managers. Regional program managers are also responsible for
coordinating regional technical review working groups, evaluating proposals, tracking
awards and disbursements, promoting regional awareness, and other operational tasks.




Environment Canada                                                                       7
                                  Table 2.1
              Program Administration: Roles and Responsibilities
      Program authority                                Responsibilities
 Minister of the Environment   Program accountability
 OPG lead and regional          Oversee the establishment of project solicitation, review,
 directors general                approval, funding allocation, reporting processes, and the
                                  setting of general terms and conditions of contributions.
                                Sign federal contribution agreements with provinces on
                                  behalf of the Department.
                                Approve project funding disbursements.
                                Ensure overall program performance.
 OPP lead 1C3D and OPSC         Ensure national consistency in program implementation.
 lead L266                      Monitor and report annually on program results.
                                Maintain EDF contributions and disbursement data base.
                                Oversee program auditing and evaluation activities related to
                                  conditions of TB approval.
                                Coordinate management response to program
                                  evaluation/audit findings.
                                Provide guidance and advice on program management and
                                  delivery.
                                Promote national awareness of the EDF.
 Regional program managers      Manage the day-to-day fund operations.
                                Establish and coordinate regional technical review working
                                  groups.
                                Track financial awards and disbursements.
                                Receive funding applications and provide advice on project
                                  proposal development.
                                Conduct proposal reviews and evaluations.
                                Undertake public consultations when required.
                                Recommend funding allocations to RDG.
                                Develop funding agreements.
                                Monitor approved projects to ensure accountability and
                                  results.
                                Develop communication plans for project announcements.
                                Promote regional EDF awareness and use by the legal
                                  community, recipient groups, and other federal government
                                  and nongovernment stakeholders.
Source: Environment Canada. 2005. Environmental Damages Fund Results-based
Management and Accountability Framework, Table 2. Ottawa.



2.2.2 Monetary Contributions to Date and Prospective Financial Resources

Table 2.2 displays financial data (court-funded awards and disseminated contributions)
nationally and by region for each of the nine years of the program‘s operation. This
distribution demonstrates an uneven regional pattern of both awards directed to the EDF
and contribution agreements disbursed through the program. As discussed in Section
4.2, both the number of awards (i.e., as a result of program promotion in the courts) and
contribution agreements are contingent to some extent on the resources available for
regional administration of the program.


Environment Canada                                                                           8
The other especially noteworthy pattern in Table 2.2 is the evidence of greater funds
assigned to EDF than contributions made by the EDF. The total value of court awards
and other funds directed to the EDF to date amounts to approximately $4.1 million,
compared to approximately $3.6 million in contributions disbursed by the EDF.

Future monetary contributions from court awards and other sources cannot be predicted
with certainty. Awards may nonetheless grow to the extent that the EDF becomes better
promoted to the legal community and more fully utilized by the courts. As well, recent
amendments (Bill-C15) to the Migratory Birds Convention Act, 1994 (MBCA 1994) call
for fines and penalties resulting from prosecutions of offences under the Act to be
credited to the EDF. In addition, Section 5.1 of the Act with respect to the pollution of
waters or areas frequented by migratory birds has been strengthened. As such, fines
received under this section, although less common, have the potential to be of a
sizeable amount, up to a maximum of $1 million. In the ten years before the
amendments came into effect, only one award, in the amount of $5,000 was made.
Since the amendments came into effect four years ago, 11 awards totalling $90,143
have been directed to the EDF, resulting in nine projects across Canada.

The Policy on Specified Purpose Accounts stipulates that ―Administrative charges may
be imposed on administered funds (for example, trust funds, endowments, joint project
funds) when the legislation or other enabling authority allows it.‖ However, the legislation
under which court awards and other funds can be directed to the EDF do not specifically
authorize the cost of administration to be charged to the account, and so the costs for
the administration of the EDF are ―charged to a departmental appropriation.‖15




15
     Treasury Board. 1995. Policy on Specified Purpose Accounts, Section 6, sub-section (o).
     Ottawa.


Environment Canada                                                                             9
Audit and Evaluation Branch                                                                                Evaluation of the Environmental Damages Fund



                          Table 2.2: Number and Value of EDF Awards and Contributions by Region
Region      1998        1999        2000       2001          2002        2003         2004        2005            2006        2007       2008        Total
                                                             Number of Court Awards by Region
Atlantic      1          1           9          11             6           6           10             2             5          13          2          66
Quebec        0          1           0           0             0           0           2              2             2           1          2          10
Ontario       0          0           0           1             1           1           0              0             1           1          3           8
PNR           0          0           1           0             4           2           4              3             1           4          2          21
PYR           0          1           1           0             0           2           2              4             4           1          2          17
Total         1          3           11         12            11           11          18             11           13          20         11          122
                                                              Value of Court Awards by Region
Atlantic   $15,000     $10,000    $157,000   $144,300      $131,000      $3,700     $194,000     $27,500         $146,000    $213,354   $50,000    $1,091,854
Quebec       $0        $42,500       $0         $0            $0           $0       $310,079    $115,000         $500,000    $80,000     $5,100    $1,052,679
Ontario      $0          $0          $0       $40,000       $2,200      $15,000        $0             $0         $22,500     $39,000    $23,000    $141,700
PNR          $0          $0       $66,700       $0         $150,000     $160,000    $106,500    $140,000         $150,000    $228,600   $28,000    $1,029,800
PYR          $0        $45,000    $23,000       $0            $0        $109,000    $204,500     $68,400         $25,000     $235,753   $46,295    $756,948
Total      $15,000     $97,500    $246,700   $184,300       283,200     $287,700    $815,079    $350,900         $843,500    $796,707   $152,395   $4,072,981
                                                        Number of Contribution Agreements by Region
Atlantic      1          2           13         14             5           3           14             0             8          13          7          80
Quebec        0          1           0           0             0           0           4              0             6           0          0          11
Ontario       0          0           0           1             1           0           0              0             0           2          3           7
PNR           0          0           0           0             0           2           3              2             2           0          1          10
PYR           0          1           2           0             0           2           2              3             2          11          5          28
Total         1          4           15         15             6           7           23             5            18          26         16          136
                                                         Value of Contribution Agreements by Region
Atlantic   $15,000     $10,000    $163,000   $209,470       $74,500     $16,000     $168,609          $0         $173,000    $119,074   $10,000    $958,653
Quebec       $0        $42,500       $0         $0            $0           $0       $347,079          $0         $820,000      $0         $0       $1,209,579
Ontario      $0          $0          $0       $40,000       $2,200         $0          $0             $0           $0        $37,500    $39,000    $118,700
PNR          $0          $0          $0         $0            $0        $39,800     $160,100     $24,000         $160,000      $0       $10,000    $393,900
PYR          $0        $45,000    $23,000       $0            $0        $109,000    $204,500    $195,000         $25,800     $226,275   $46,295    $874,870
Total      $15,000     $97,500    $186,000   $249,470       $76,700     $164,800    $880,288    $219,000        $1,178,800   $382,849   $105,295   $3,555,702
Source: Environment Canada. 2009. Environmental Damages Fund administrative data. Ottawa.



Environment Canada                                                                                                                                           10
Audit and Evaluation Branch                   Evaluation of the Environmental Damages Fund




3.0 EVALUATION DESIGN

3.1 General Purpose

The evaluation assessed progress in achieving the Department's expected
environmental outcomes for the EDF, with specific emphasis on developments since the
last evaluation. The timeframe for the evaluation focuses on the period between August
2002, when the last evaluation took place, and March 2008.

The study explored the relevance, design and delivery, success and cost-effectiveness
of the EDF program. Specifically, the evaluation sought to determine whether the
program is:
      Aligned with and contributing to federal government priorities, and whether they
        address actual needs (relevance);
      Achieving or on track to achieving their intended outcomes (success);
      Using the most appropriate and efficient means to achieve their outcomes (cost-
        effectiveness/alternatives); and
      Designed and delivered in the best possible way (design and delivery).16

Furthermore, the evaluation explored the extent to which 13 terms and conditions for the
EDF contribution agreements (see Annex 2), established from the outset of the EDF
program, had been satisfied.17

To address these questions, the evaluation involved a variety of data-collection
approaches, including a document review, a review of contribution agreement files and
key informant interviews. The review of EDF project files is the sole source of data for
determining if most of the EDF terms and conditions for contribution agreements had
been met. As well, the document review was the primary source of evidence used to
address issues related to the relevance of the program, as this issue was fully explored
in the 2002 evaluation and the 2007 process analysis.18 The relevance-related
conclusions stemming from these two studies remain pertinent because there has been
no change in the Government of Canada‘s constitutional role on matters of the
environment or in EDF‘s role relative to the courts. Nonetheless, some qualitative
evidence from key informant interviews was gathered to further corroborate these
conclusions.




16
   The design and fieldwork for the EDF evaluation was carried out in the 2008–2009 fiscal year,
   prior to implementation of the new Treasury Board Policy on Evaluation (http://www.tbs-
   sct.gc.ca/pol/doc-eng.aspx?id=15024). The current evaluation reflects those issues outlined in
   the 2001 evaluation policy that was in effect at the time this evaluation was conducted.
17
   Environment Canada. 2005. Environmental Damages Fund Results-based Management and
   Accountability Framework, Appendix A. Ottawa.
18
   PWGSC Government Consulting Services. 2007. Environmental Damages Fund Process
   Analysis. Ottawa.


Environment Canada                                                                              11
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund



Annex 3 presents the full series of evaluation questions, enumerates sources of
evidence, and identifies relevant challenges considered by the evaluation team in
investigating each question.

3.2 Evaluation Approach and Methodology
Three sources of evidence were necessary for the evaluation: background documents,
administrative and financial files for projects, and key informant interviews. Each of
these data-collection approaches is described in turn below.

3.2.1 Document Review

A primary source of information for the evaluation involved a review of existing
documentation. The review included:

        Performance documents (e.g., Results-based Management and Accountability
         Frameworks [RMAFs]);
        Policy documents (Treasury Board submissions, Memoranda to Cabinet,
         departmental documents);
        Corporate planning documents (e.g., guides, terms and conditions); and
        Audit and evaluation reports and other research.

The document review responded to a number of evaluation issues primarily related to
the EDF‘s ongoing relevance and design and delivery. A bibliography of documents
included in the review appears in Annex 4.

3.2.2 Review of Administrative and Financial Files

Administrative and financial files from EDF-funded projects were reviewed to explore
issues related to the success and design and delivery (i.e., performance measurement,
partnerships and accountability) of the EDF, as well as to assess the degree to which
EDF agreements satisfy EDF terms and conditions for contribution agreements.

The files from a sample of 35 EDF-funded projects were analyzed using a customized
file review guide to ensure the accuracy and consistency of the review (see Annex 5).
The sample of 35 projects was selected to reflect: (a) projects from all regions, with
oversampling for regions with fewer projects,19 (b) the diversity of purposes and subjects
of funded projects, and (c) the diversity of recipients that received EDF funding.

Overall, a large majority of project files were found to contain information to address
evaluation questions related to nearly all evaluation questions addressed through this
methodology. Only one in four (23%) project files contained any information on




19
  Oversampling provides a more precise and reliable understanding of the characteristics of
 smaller sub-groups in instances where their occurrence in the population would not yield an
 adequate number of cases employing a stratified random sampling approach.


Environment Canada                                                                             12
Audit and Evaluation Branch                        Evaluation of the Environmental Damages Fund


unintended outcomes, although this is not surprising considering that this information is
not required nor is it necessarily relevant (i.e., when no such outcomes exist) for all
projects.

With regard to information demonstrating compliance with the terms and conditions of
the EDF contribution agreements, large majorities of project files provided information
on recipient eligibility/qualifications, project eligibility (i.e., maximum value, allowable
expenditures), use of partnerships, and proposal review (see Table 3.1). Less evidence
was available for those terms and conditions that may not be applicable to all projects,
such as consistency with court conditions and ownership of equipment purchases. As
well, little evidence was found concerning project cost-effectiveness or payment
invoices.

            Table 3.1: Existence of Information from Project Records
            by Evaluation Question and EDF Contribution Guidelines

                        EVALUATION QUESTION                                          % OF PROJECTS
     1. Does the project identify clear deliverables and expected
                                                                                             100
         results?
     2. Is performance data collected against program activities?                             94
     3. Is performance data collected against program outcomes?                               83
     4. Program outcome targeted by the EDF-funded project                                   100
     5. Evidence of intended outcome achievement                                              83
     6. Is there appropriate accountability framework (e.g., for multi-
                                                                                             100
         stakeholder agreements)?
     7. Is the project delivered in partnership with any other individuals,
                                                                                              74
         groups or organizations?
     8. Have there been any unintended (positive or negative)
                                                                                              23
         outcomes?
    TERMS AND CONDITIONS OF CONTRIBUTION AGREEMENTS
     1. Recipients must be eligible to receive the monies                                     97
     2. The contribution is consistent with application and decision                           6
     3. The project meets court conditions                                                     6
     4. The project is cost-effective                                                         29
     5. The recipient possesses the skills necessary to deliver the
                                                                                             100
         project
     6. Projects undertaken build on partnerships                                             74
     7. Proposals are evaluated by program managers / are signed off
                                                                                            91/83
         by RDG*
     8. The maximum amount payable is under $5 million                                       100
     9. Awards cover allowable expenditures (not basic costs)**                               89
     10. Payment methods are on invoice and with statement of
                                                                                              29
         expenditures
     11. Equipment purchased belongs to the project                                           51
     12. There exists a right to audit each project                                           91
     13. There exists a payment authority                                                     94
*This criterion appears as a single item in the EDF terms and conditions, but was assessed separately for
  the evaluation.
**The prohibited basic costs are those costs that are necessary for maintaining the operation of the
  recipient organization.




Environment Canada                                                                                      13
Audit and Evaluation Branch                   Evaluation of the Environmental Damages Fund


3.2.3 Key Informant Interviews

Key informant interviews were used to gather in-depth information on all the evaluation
questions and to supplement information collected through the document and file
review. A total of 116 key informant interviews were conducted with representatives from
the following groups:

        46 Environment Canada staff;
        40 funding recipients;
        6 individuals at Justice Canada and in the justice system;20
        9 academics with environmental expertise; and
        15 potential recipients among universities, local governments, community
         organizations, and First Nations.

Customized key informant interview guides were developed in consultation with the
project authority for each respondent group to ensure the guides reflected their level of
familiarity with the program (e.g., past vs. potential applicants) and occupational
perspectives (e.g., EC personnel vs. university-based experts) (see Annex 6). For
example, detailed design and delivery-related questions were asked of EC personnel
but not of potential funding recipients or key informants in the court and justice segment.
Each interview guide was programmed and delivered using a Computer-assisted
Telephone Interviewing (CATI) system that allowed for real-time data collection.
Interviews lasted approximately 45 minutes each.

The sample frames for EC personnel, funding recipients and justice system respondents
                                                                            21
were supplied by the EDF program. The sample frame for potential applicants and
academic experts was developed by COMPAS. Potential applicants included
community-based environmental groups, Aboriginal groups, universities, and municipal
governments.

Given the busy schedules of individuals in professional or executive occupations,
potential respondents from these cohorts were often unavailable to take part in
interviews during the timeframe for evaluation fieldwork. For this reason, the lists of
potential key informants available from administrative files were not large enough to
complete an adequate number of interviews in the court and justice segment. To
enhance the size of the justice-segment sample frame, the researchers also employed a
referral (or ―snowball‖) method. Using this approach, respondents from this group were
asked for the names and contact information of other potential key informants involved
in the justice system and familiar with the EDF program.




20
   Respondents in this segment consist of four prosecutors, one judge, and one legal counsel to a
   federal department.
21
   Potential applicants were identified as eligible for program funding according to the program‘s
    terms and conditions.


Environment Canada                                                                              14
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


3.2.4 Challenges and Limitations

The evaluation project involved a large number of key informants (116) and a large
number of administrative files related to each of 35 EDF-funded projects. Any study of
this magnitude will necessarily encounter some challenges that may or may not be
possible to remedy. The following is a description of challenges and limitations of the
current research, as well as steps taken to address them.

1) The identification of external experts.
A potential challenge for any evaluation arises from the tendency for experts and
expertise to be concentrated within the program being evaluated. In order to satisfy a
need for independent experts, COMPAS developed a list of professors whose
specialization in environmental matters would provide them with some degree of
expertise in the program or its subject matter. Key informants at universities who fulfilled
the need for independent expertise were drawn from teaching and research faculties.
Key informants at universities who fulfilled the requirement to consult potential
applicants were drawn from senior university administrators.

2) The absence of a centralized information management system.
The major challenge associated with project records is a sometimes high rate of
incomplete records, as shown in Table 3.1. All project records were lodged in regional
offices without duplicates assembled at a central location. This geographic dispersal of
record-keeping made it impossible to verify if more complete project records were
available than those transmitted to the researchers under short timeframes. Given the
incompleteness of the records provided, there remains some question about whether or
not the findings may be underestimating compliance with EDF terms and conditions
(e.g., compliance with court conditions) as well as EDF‘s success in achieving various
evaluation goals (e.g., related to deliverables and impacts).

3) Limited information to assess cost-effectiveness.
Limited information related to financial and human resource expenditures, possibly the
result of a lack of dedicated resources for the administration of the EDF, meant a
thorough assessment of the cost-effectiveness of the program was not possible.
Although key informants were able to provide qualitative feedback on the cost-
effectiveness of the program, very little administrative data exist to support these views.

4) The absence of a comparison group for funded projects.
Court restrictions and limited program promotion among potential applicants meant that
an adequate pool of unsuccessful applicants was not available to develop a comparison
group against which to assess the success of EDF-funded projects. In this manner then,
it is not possible to determine conclusively whether progress toward the realization of
program objectives is uniquely attributable to the EDF.




Environment Canada                                                                        15
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund




4.0 FINDINGS
Findings for the following four main evaluation issues are presented serially: relevance,
design and delivery, success, and cost-effectiveness. A discussion of project
compliance with EDF terms and conditions for contribution agreements follows.

A rating is also provided for each evaluation question. The ratings are based on a
judgment of whether or not the findings indicate that:

      The intended outcomes or goals have been achieved or met—labelled as
       Achieved;
      Considerable progress has been made to meet the intended outcomes or goals,
       but attention is still needed—labelled as Progress Made, Attention Needed; or
      Little progress has been made to meet the intended outcome and attention is
       needed on a priority basis—labelled as Little Progress, Priority for Attention; and
      The N/A symbol identifies items where a rating is not applicable.

In addition, a tilde (~) is used to denote instances where outcome achievement ratings
are based solely on subjective evidence. A summary of ratings for the evaluation issues
and questions is presented in Annex 7.


4.1 Relevance

   Overall Findings:
The EDF is highly relevant in terms of supporting federal priorities related to
environmental enforcement and departmental priorities related to natural
resource rehabilitation, environmental quality improvement and conservation,
research and development, and education and awareness. The vast majority of
key informants feel the EDF serves an important role and does not duplicate other
existing programming. As well, the evidence suggests the program is the only
federal mechanism of its kind for managing court awards and financial penalties
arising from federal environmental enforcement activities, is expected to play an
expanded role in the management of court awards prosecuted under different
federal statutes, and supports the court’s use of creative sentencing in
environmental damage cases.




Environment Canada                                                                      16
Audit and Evaluation Branch                      Evaluation of the Environmental Damages Fund




       Evaluation Issue                       Indicator(s)                Methods       Rating
1. Is there a legitimate and          Evidence of similar internal      Document     Achieved
  necessary role for government        or external programs               review
  in this program area or             Demonstration of the              Interviews
  activity?                            utility/rationale for program/
                                       funding mechanism
  Is the program connected
   with societal/environmental        Views of program staff                          Achieved
   needs?
    Does the program theory                                                           Achieved
     (activities, instruments,
     objectives, design) address
     the societal need identified?

Recent federal budgets have invested considerably in environmental stewardship and
enforcement of environmental laws. In the 2007 budget, the Government invested
$110 million for more effective implementation of the Species at Risk Act and $22 million
to improve the enforcement of environmental protection laws.22 Similarly, the 2008
federal budget committed $21 million to enhance the effectiveness of environmental law
enforcement; $12 million for law enforcement within national parks; $62 million for the
health of the oceans, water pollution prevention, and improved surveillance and
enforcement along Canada‘s coasts; and $15 million for protected areas in the
Northwest Territories.23

The EDF is an important mechanism for the management of court awards and financial
penalties arising from federal enforcement activities. The EDF is specifically identified in
amendments to the Migratory Birds Convention Act, 1994 (MBCA 1994) as the
mechanism to be used in managing fines and penalties resulting from prosecutions of
offences under the Act. While not identified by name, the EDF can also be used for the
management of court awards and penalties meted out through a variety of other
legislation, including the Fisheries Act, the Canada Shipping Act, 2001, the Species at
Risk Act, and the Canadian Environmental Protection Act, 1999. Furthermore, the
Government has signalled its intent to expand the use of the EDF by considering
amendments to six Environment Canada statutes24 and amending three Parks Canada
statutes25 to enable awards and fines to be directed to the EDF.

The Environmental Damages Fund is the only federal mechanism of its kind for the
management of court awards and financial penalties resulting from environmental



22
   Department of Finance. 2007. The Budget Plan 2007: Aspire to a Stronger, Safer, Better
   Canada. Ottawa.
23
   Department of Finance. 2008. The Budget Plan 2008: Responsible Leadership. Ottawa.
24
   The Environment Canada statutes include the Canadian Environmental Protection Act, the
   Canada Wildlife Act, the Migratory Birds Convention Act, the Wild Animal and Plant Protection
   and Regulation of International and Interprovincial Trade Act, the Antarctic Environmental
   Protection Act, and the International River Improvements Act.
25
   The Parks Canada statutes include the Canada National Parks Act, the Canada National
   Marine Conservation Areas Act, and the Saguenay–St. Lawrence Marine Park Act.


Environment Canada                                                                             17
Audit and Evaluation Branch                    Evaluation of the Environmental Damages Fund


damage events that have been prosecuted under these and other legislative
authorities.26 What distinguishes the Environmental Damages Fund from other
programs that support third parties in implementing environmental projects, such as
grants and contributions programs, is its role as a tool to support the use of creative
sentencing27 by the courts in environmental damage cases. Creative sentencing allows
the courts to specify how funds derived from fines and penalties for environmental
infractions are to be used for beneficial environmental purposes.28 Although the creative
sentencing provisions of some federal legislation allow the courts to order monies to be
paid directly to local community groups to help reverse the effects of environmental
damage, these funds are not managed with either the oversight or the stringent
                                               29
conditions on their use imposed by the EDF. In particular, the EDF ensures that:

        The funds directed to it are managed in a fully accountable manner;
        Dispensed funds are overseen by Environment Canada personnel with an
         interest in remediation rather than in the court process;
        Funded projects are somehow related to the environmental harm incurred; and
        Projects are implemented in partnership with interested parties.30

The creation of the EDF offers the courts the option of using this national program as an
intermediary capable of overseeing and auditing the work of organizations engaged in
using EDF funds for their specified purposes, as well as finding geographically niche
organizations suited to carrying out a court‘s geographically specific requirements.31

Almost all key informants consulted believe that the EDF serves a valuable societal
need. Most feel the EDF does not duplicate the functions of other organizations and is
                                           32
uniquely placed to carry out its functions. Furthermore, the EDF priorities of natural
resource rehabilitation; environmental quality improvement and conservation; research




26
   HDP Group Inc. 2002. Environmental Damages Fund Evaluation. Ottawa.
27
   The Environmental Law Centre defines creative sentencing as the option for judges ―…to take
   an innovative approach to sentencing by ordering that funds be dedicated to certain projects
   like research, education or improvements in industry standards‖.
   (http://www.elc.ab.ca/Content_Files/Files/NewsBriefs/CreativeSentencing-V18-2.pdf, accessed
   June 2009).
28
   Creative sentencing might also include requiring polluters to undergo environmental audits or
   perform community service (Wruck, J. 2004. The Federal Environmental Damages Fund. The
   Advocate, Vol. 62-2.)
29
   Wruck, J. 2004. The Federal Environmental Damages Fund. The Advocate, Vol. 62-2, page
   228.
30
   HDP Group Inc. 2002. Environmental Damages Fund Evaluation. Ottawa.
31
   These points appear in the interviews with EC key informants and in Wruck, J. 2004. The
   Federal Environmental Damages Fund. The Advocate, Vol. 62-2., passim and Environment
   Canada. 2005. Environmental Damages Fund Results-based Management and Accountability
   Framework, page 4. Ottawa.
32
   The one exception to this largely uniform viewpoint is the few key informants, primarily EC
   personnel, who advocate assigning the EDF‘s entire role to either the judiciary or the provinces.
   These key informants, however, provided no insights as to how the courts or provinces would
   acquire the authority or administrative capacity to deliver such a program, nor how the
   assumption of these responsibilities would impact on the courts‘ neutral adjudicative role.


Environment Canada                                                                               18
Audit and Evaluation Branch                    Evaluation of the Environmental Damages Fund


and development; and education and awareness are well aligned with the Department‘s
strategic outcome for the restoration and conservation of Canada‘s natural environment.

Only two key informant respondents noted vaguely similar programs which operate at
the provincial level. One judicial key informant suggested that the EDF is ―roughly
analogous‖ to the British Columbia Habitat Conservation Trust Fund, although this fund
operates exclusively within the province and is funded and supported by outdoors
sportspeople (e.g., hunters and anglers) rather than the courts. Similarly, one program
respondent mentioned Ontario‘s Community Environment Fund, which was launched in
2007 to support restoration projects using penalties exacted by the Ontario Ministry of
the Environment (not the courts) for industrial wastewater spills, but not for other types
of environmental damage incidents. Environment Canada‘s Ontario Region also
managed the Environment Canada Courts Awards (Pilot) Project until 2007. This
program, however, was limited to offences prosecuted under CEPA 1999, with penalties
                                             33
disbursed directly to the award beneficiary.


4.2 Design and Delivery

The following section of the report presents evaluation evidence related to various
aspects of the EDF program‘s design and delivery. The section begins with an overview
of evidence related to EDF‘s general delivery, including administrative resourcing and
program awareness among key stakeholder groups.

Overall Findings:
The EDF is generally perceived to be delivered as designed, although low familiarity with
the program and a lack of supporting documentation for its use in the courts may reduce
its application when prosecuting environmental infractions.

The absence of dedicated resources for the EDF program has resulted in less than full
implementation of the program in most Environment Canada regions, with the exception
of Atlantic Canada. The solution to uneven regional implementation is generally felt to
involve an increase in human resources devoted to the program, which should improve
promotion in the courts and among potential applicants, accessibility of staff to support
funding recipients, and the ability of program staff to respond to court-imposed funding
conditions.

Awareness of the EDF is low among both the courts and potential applicants to the
program, and this appears to be largely the result of a lack of dedicated resources for
program delivery. Low program awareness is thought to limit the number of court
awards being diverted to the program, lead to a higher rate of court-imposed conditions,
and result in fewer applications for EDF funding. Possible strategies for raising
awareness among the courts include training for prosecutors, communications activities
targeting judges, and engaging enforcement officers in efforts to educate the judiciary.




33
     Morrison, Stacey. 2007. Environment Canada Courts Awards (Pilot) Project: Progress Report
     Powerpoint presentation.


Environment Canada                                                                               19
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund



The evaluation also found the following with respect to various specific aspects of the
EDF‘s delivery:

      The application process: is generally perceived to be appropriate, although
       some funding recipients and EC personnel feel it is burdensome. A comparison
       of the EDF application form with other similar program forms suggests there may
       be some limited room to streamline it. EDF program personnel are perceived to
       be quite helpful in navigating the application process, but uneven accessibility
       and expertise of staff likely resulting from a lack of dedicated resources may
       dampen the effectiveness of support to program applicants.
      Partnership requirements: Program documentation is not clear about whether
       partnerships are a requirement for EDF-funded projects and assessments of the
       value of partnerships are mixed (i.e., leveraging of partner funding/expertise
       versus the potential for conflict with or poor performance of partners).
       Nonetheless, a slim majority of EDF-funded projects were implemented in
       partnership, and the most frequently cited partner contributions included funding,
       service delivery or research.
      EDF terms and conditions: The vast majority of project files are in compliance
       with most of the terms and conditions for EDF contributions, although high rates
       of incomplete information and conflicting evidence from other sources do not
       allow the evaluators to reach a conclusion concerning compliance with five of
       these conditions.
      Environmental Protection Award Measures (EPAMs): EC personnel and, to a
       lesser extent, judicial respondents report very low levels of familiarity with the
       function and/or use of EPAMs. Of those respondents who are familiar with
       EPAMs, opinion was divided as to whether their use would grow or diminish in
       the future.
      Roles and responsibilities: There exists a great deal of program
       documentation outlining the roles and responsibilities of the different parties
       involved in the delivery of the EDF program. A number of respondents, however,
       expressed some confusion in this regard and felt there was a need for more
       consistency and oversight in the delivery of the program across regions (noted
       by EC personnel) and EDF program staff (noted by recipients).
      Performance measurement and reporting: Intended deliverables and impacts
       of the EDF program and funded projects are generally clear and performance
       data are collected for most projects with respect to activities, outputs and
       outcomes. In most regions, however, no systematic process exists for
       communicating program results to senior managers and decision-makers.




Environment Canada                                                                        20
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


4.2.1 Program Delivery


   Evaluation Issue                      Indicator(s)               Methods        Rating
Is the program delivered as     Demonstration of whether the    Document      Little
designed?                        program/project was              review        Progress,
                                 implemented as designed         File review   Priority for
                                Views of program staff and      Interviews    Attention
                                 recipients


Key informants from Environment Canada and the justice and courts segment were
asked to rate on a 7 point scale the extent to which they perceive the EDF program is
being delivered as designed. As shown in Figure 4.1, four of six key informants from the
justice and court segment were strongly of the view (i.e., responding with a 6 or 7 on a
seven-point scale) that the EDF is being delivered as designed, compared to only one
who felt the program is delivered as designed to only a moderate extent (i.e.,
responding with a 3, 4 or 5). In contrast, Environment Canada staff were divided as to
whether the program is being delivered as designed to a large (43% responding with a 6
or 7) or moderate extent (40% responding with a 3, 4 or 5). Almost no key informants in
either group felt the EDF is not functioning as it was intended (i.e., responding with a 1
or 2 on a seven-point scale).




Environment Canada                                                                        21
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


The six key informants in the justice and court segment were also asked to describe
how the EDF‘s use by the courts has been consistent or inconsistent with the program‘s
original design. The majority of these respondents expressed some uncertainty of the
program‘s original design and how it was to be used, and so had difficulty in assessing
whether it was being delivered accordingly. Based on what these respondents do
understand of the program, most interviewees felt that the courts used the program as
designed, particularly in terms of specifying conditions for how penalties are to be used
to address the environmental damage. Some respondents suggested, however, that the
absence of specific examples of how the funds can be and have been used means the
EDF is less likely to be well received by the courts as a sentencing option and that the
courts are left to their own devices to invent ways of applying the program.

Many Environment Canada key informants volunteered the view that insufficient staff
support and stringent court conditions, especially geographically tight restrictions, are
the biggest challenges to delivering the program as designed. In some regions,
insufficient staffing is seen by many of these key informants as impeding promotion of
the program among potential fund applicants, with the result that the assignment of
contributions does not keep pace with the receipt of funds from the courts.

Summary: The EDF is generally perceived to be designed as delivered, although
low familiarity with the program and a lack of supporting documentation for its
use in the courts may reduce its application when prosecuting environmental
infractions. The primary barrier to delivering the EDF was perceived by
Environment Canada staff to be insufficient staff support, which has also resulted
in poor program promotion among potential recipients.

       4.2.1.1 Administrative Resources and EDF Development across Canada
There exist no national or regional resources dedicated to the delivery of the EDF. The
EDF is one of several programs (including EcoAction and other education and
engagement activities) delivered by the Community Partnerships program at
Environment Canada. In the absence of dedicated resources, the program‘s
management is a shared responsibility among all 31 headquarter and regional
Community Partnerships staff. According to administrative information, it is estimated
that collectively, the Community Partnerships personnel devote the equivalent of four
full-time staff to administering the program. This amounts to an average assignment of
one-eighth of each Community Partnership staff person.

EC employees generally feel that the absence of dedicated resources has tended to
result in less than full program implementation and uneven national delivery. In
particular, while approximately $4.1 million in court awards and penalties have been
diverted to the EDF over the nine years of the program‘s operation, only $3.6 million has
been allocated to support EDF projects over the same period (see Table 2.2). The one
exception has been the virtually full delivery of the program in Atlantic Region, even
though in 2008–2009, 54% of recorded pollution incidences (1,611 of 2,958 events) in
Environment Canada‘s NEMISIS data base on environmental emergencies occurred in




Environment Canada                                                                          22
Audit and Evaluation Branch                    Evaluation of the Environmental Damages Fund


the Atlantic region.34 Although not addressed directly through interviews, EC
respondents commented variously that the Atlantic region‘s success in implementing the
EDF is attributable to its resources (i.e., experienced staff, Web site and toll-free phone
number) and established relationships with partners and organizations external to the
program (e.g., community groups, EC enforcement officers and other departments). In
particular, these relationships facilitate the identification of eligible and qualified project
proponents, and enhance the likelihood that an informed judiciary will direct court
awards and penalties to the EDF.

The predominant recommendation among EC respondents to address the issue of
uneven regional implementation was an increase in human resources assigned to the
program, believing that insufficient personnel in key regions was the root cause of the
program‘s underdevelopment in these areas. Insufficient personnel in key regions
hampered the program‘s ability to promote the EDF in the courts and hence increase
the number of awards to the program, in their view. Insufficient personnel also
hampered EDF‘s ability, in the view of some key informants, to promote itself among the
population of potential applicants and hence expand the pool of potential beneficiaries
able to carry out court requests.

EC personnel and past EDF beneficiaries alike also reported problems of accessibility to
EDF personnel for both applicants and approved beneficiaries. Both groups of key
informants reported navigational problems with the Web site and delays in processing
applications and paperwork. It is also possible that gaps in the records for some EDF
project files, as outlined in Section 4.2.2.3 on compliance with program terms and
conditions, are due in part to the absence of dedicated resources.

Another characteristic of the program which may place further strains on these limited
resources involves the imposition of stringent court conditions. Court-imposed conditions
on the types of projects that can be supported through court awards and penalties
diverted to the EDF can require significant effort on the part of program staff to find
potential recipients who can satisfy the court‘s requirements and who also have the
organizational and technical capabilities necessary for the tasks. To meet court-
determined requirements, EDF personnel are sometimes required to search for potential
funding recipients. As such, the EDF may have a greater need for administrative
capacity than a program of its size would normally require.

Summary: With the exception of Atlantic Canada, the absence of dedicated
resources for the EDF program has resulted in less than full implementation of
the program in most regions, such that the pace at which funds are being diverted
to the program through the courts outstrips the program’s ability to allocate
funds to EDF projects. More resources and established relationships with
stakeholders are thought to underlie the Atlantic region’s successful
implementation of the program. The solution to uneven regional implementation
is generally thought to involve an increase in human resources devoted to the



34
     Environment Canada. 2009. Number of Spill/Releases, by Fiscal year, by Region, National
     Emergencies and Enforcement Management Information System and Intelligence System.
     Administrative data.


Environment Canada                                                                             23
Audit and Evaluation Branch                      Evaluation of the Environmental Damages Fund


program, which should improve promotion in the courts and among potential
applicants, accessibility of staff to support funding recipients and the ability of
program staff to respond to court-imposed funding conditions.

       4.2.1.2 External Awareness
Evidence from key informants reveals that awareness of the EDF is low among potential
EDF applicants and the courts (see Section 4.3.1 for a detailed discussion of the
achievement of communications and program promotion outcomes). Previous
research35 suggests that low awareness of the program may have several negative
consequences for its effectiveness. The most obvious drawback is fewer court awards
being diverted to the EDF, such that the application of these funds to address
environmental damage is not overseen in an accountable and neutral fashion. As well,
some key informants among funding recipients, judicial respondents and EC personnel
suggested that excessive court-imposed restrictions on awards and penalties diverted to
the EDF may be attributable to low awareness of the program. In particular, some
individuals in the small judicial segment of key informants noted that all sides in the
courtroom appear to share a common mistrust of government and hence an inclination
to impose restrictions. Respondents among these various groups note that if
communication to the judiciary were framed in such a way that crown attorneys and
judges alike understood the impacts of stringent court conditions, a supplementary
benefit could be a reduction in court-mandated restrictions that make EDF projects
difficult to implement. Some EC key informants further note that this could in turn lower
administrative costs and reduce delays, as the absence of stringent conditions would
mean projects could be initiated without searching out or even assembling recipient
organizations to carry out the court-mandated task.

Low awareness among potential beneficiaries may also explain the low rate of proposals
and consequent high rate (more than 90%) of acceptance of proposals. Increased
awareness of EDF among potential funding recipients could lead to an enlarged pool of
submissions and hence the possibility of support for more projects. This would lead to a
greater balance between EDF funding from awards, approximately $4.1 million in total
across the nine years of the program, and EDF spending on projects, approximately
$3.6 million over the same period (see Table 2.2). It remains unclear, however, if
heightened program awareness among potential recipients would result in a greater
number and quality of proposals to the program, as the conditions imposed on many
court awards necessarily limits the number of potential recipient organizations equipped
to meet these conditions.

Key informants in the judiciary suggest several potential strategies by which EC could
increase awareness of the EDF in the courts. One strategy involves working with
prosecutors to develop and provide opportunities for training in environmental
legislation. Other suggestions from several key informants in the judiciary include
separate communications campaigns to crown attorneys and judges, as well as
communicating to judges by means of a newly created workshop or at annual or semi-
annual meetings and retreats attended by judges.36 One key informant spoke of a chain



35
     HDP Group Inc. 2002. Environmental Damages Fund Evaluation. Ottawa.
36
     Presentations at meetings and retreats attended by judges are already given in Alberta.


Environment Canada                                                                             24
Audit and Evaluation Branch                       Evaluation of the Environmental Damages Fund


of education efforts beginning with enforcement officers: ―We've talked to the
enforcement people about the need to encourage prosecutors to recommend to judges
that they not be too rigid in the conditions attached to the money.‖

Summary: Awareness of the EDF is low among both the courts and potential
applicants to the program, and this appears to be largely the result of a lack of
dedicated resources for program delivery. Low program awareness is thought to
limit the number of court awards being diverted to the program, lead to a higher
rate of court-imposed conditions, and result in fewer applications for EDF
funding. Possible strategies for improving awareness among the courts include
training for prosecutors, communications activities targeting judges, and
engaging enforcement officers in efforts to educate the judiciary.


4.2.2 Administration of EDF Projects

This section explores various aspects of the management and oversight of EDF-funded
projects, including the application process, compliance with EDF terms and conditions
for contribution agreements and the use of partnerships to support funded projects. The
potential to apply a minimum project value standard for EDF-funded projects was also
explored.

      4.2.2.1 The Application Process
EC personnel and funding recipients tend to perceive the application process and the
application form favourably, with many recipients noting that it is no more onerous than
applications for other funding programs and fairly straightforward. A few funding
recipients and EC personnel, however, consider the application tedious and repetitive,
and disproportionate to the size of the EDF contributions: ―A lot of administration for not
a lot of money.‖

A comparison of the EDF and EcoAction Community Funding program application forms
suggests that the EDF application might be considered marginally more burdensome at
most. For example, EDF applicants are asked in two separate places to describe how
                               37
the environment would benefit. The EDF application also requires slightly more writing
than the EcoAction process. The latter includes a checklist of potential evaluation
indicators while the former requires applicants to describe their evaluation indicators in
essay form.

EC personnel and funding recipients tended to characterize EC Community
Partnerships employees who manage the EDF as being helpful and solicitous. EDF staff
assistance is sometimes seen as essential. ―If I hadn‘t helped the applicants out,‖
volunteered one EC key informant, ―they would still be trying to navigate the
bureaucratic paperwork.‖ However, a few key informants note that the uneven




37
     In the ‗Project Details‘ section (page 3) of the application form, the EDF asks applicants to
     ―Identify what your project will accomplish and how the environment and community will
     benefit.‖ In the ‗Evaluation‘ section (page 4) of the application form, the EDF asks applicants,
     ―How will the environment benefit from this project?‖


Environment Canada                                                                                      25
Audit and Evaluation Branch                      Evaluation of the Environmental Damages Fund


accessibility and expertise of personnel assigned to the EDF are other factors which
may contribute to the burdensomeness of the process. Some funding recipients
emphasize their frustration trying to reach EC staff and elicit useful answers to their
questions. As outlined in Section 4.2.1 above, it is likely that the variable quality of
support to recipients is the result of a lack of dedicated resources for EDF program
delivery and the consequent shared responsibility for program administration among
Community Partnerships staff.

Summary: The application process is generally perceived to be appropriate,
although some funding recipients and EC personnel feel it is burdensome. A
comparison of the EDF application form with similar forms of other programs
suggests there may be some limited room to streamline it. EDF program
personnel are perceived to be quite helpful in navigating the application process,
but their uneven accessibility and expertise, likely resulting from a lack of
dedicated resources, may dampen the effectiveness of support to applicants.

    4.2.2.2    Partnerships
Terms and conditions for EDF contribution agreements require that projects ―build on
partnerships,‖ but EDF program documentation is ambiguous on the issue. Neither the
EDF Applicant’s Guide38 nor any other literature treats partnerships as an explicit
requirement. The evaluation point structure allots up to 14 percentage points for
partnerships, thereby communicating by inference that partnerships are desirable but
not mandatory.

Feedback from key informants is divided about the value of such partnerships. Funding
recipients and EC personnel tend to be favourably disposed to the idea of partnerships
while academics and potential applicants tend to view partnerships unfavourably. Key
informant advocates of partnerships perceive the benefits of partnerships as
contributing more resources, more technical know-how, and a greater chance of leaving
behind capacity in the community after the project has been completed. The main
disadvantages are thought to be the difficulties in finding suitable partners, a potential
for conflict or poor performance among partners with fewer resources and less
experience, and the artificiality of some apparent partnership relationships. As one
academic expert pointed out, ―A lot of grantsmanship [the art of successful applications]
goes into giving the appearance of leveraging funds and in many cases that's not really
happening.‖

The review of administrative files found that just more than half (54%) of the sample of
projects examined provided evidence of partnership support, either in terms of the
identity of partner organizations or partners‘ contributions to EDF-funded projects.
Partners identified in project files tend to include a wide range of organizations, including
environmental organizations, municipalities, businesses, and community organizations,
and to a somewhat lesser extent post-secondary educational institutions (see Table




38
     The EDF Applicant’s Guide says that ―funding is available for projects which…demonstrate
     broad community support (e.g. both cash and in-kind partnerships)‖ and calls for ―details on
     other funding sources which include cash and in-kind partner support.‖


Environment Canada                                                                                  26
Audit and Evaluation Branch                    Evaluation of the Environmental Damages Fund


4.1).39 Of these partners, more than half (58%) contribute funding to the project, while
nearly half (47%) assist in service delivery and one-third (32%) contribute research. For
one in five projects (22%) that identified partners, there was either no statement of the
partners‘ contribution (11%) or the partners‘ contribution was unclear from the
documentation supplied (11%).

                                         Table 4.1:
                            Profile of EDF Project Partnerships
 Type of Partner                                                     % of sample (n=35)*
 Environmental organization                                                    34
 Municipality                                                                  23
 Business                                                                      23
 Community organization (excl. environmental organization)                     23
 Post-secondary educational institution                                        14
 No partners                                                                   20
 Insufficient information                                                      26
 Nature of Partnership                                             % of partnered projects
                                                                             (n=19)
  Funding                                                                      58
  Assistance in service delivery                                               47
  Research                                                                     32
  Unclear description of partners‘ obligations                                 11
  No statement of their obligations                                            11
* Sum of partnership categories may exceed 100%, as multiple partners and types of partnership
  arrangements are possible.

Summary: Program documentation is not clear about whether partnerships are a
requirement for EDF-funded projects, as suggested by the terms and conditions
for EDF contribution agreements. Key informant assessments of the value of
partnerships are mixed. The key strengths of partnerships are perceived to be the
leveraging of partner funding/expertise and project sustainability, while the
weaknesses are felt to involve difficulties in finding suitable partners and the
potential for conflict with or poor performance of partners. Nonetheless, a slim
majority of EDF-funded projects were implemented in partnership, and the most
frequently cited partner contributions included funding, service delivery or
research.

      4.2.2.3 Compliance with Terms and Conditions for EDF Contribution
               Agreements
The 2002 EDF evaluation found that for the Atlantic region, the program and EDF
projects were being administered in compliance with all EDF terms and conditions for
contribution agreements.40 Now that implementation of the program is further evolved in
the other regions than was the case in 2002, the current evaluation sought to explore
the extent to which funded projects across all regions are in compliance with these
terms and conditions.



39
   For reasons of inherent overlap, Aboriginal organizations appear within the other categories in
   the classification displayed in Table 4.1.
40
   HDP Group Inc. 2002. Environmental Damages Fund Evaluation. Ottawa.


Environment Canada                                                                               27
Audit and Evaluation Branch                   Evaluation of the Environmental Damages Fund



Table 4.2 presents data from the administrative files to assess project compliance with
each of the 13 EDF terms and conditions. In several instances, the administrative files
contained too little information to verify whether the conditions were satisfied; in none
was there evidence of noncompliance.

Overall, information from the file review demonstrates compliance with EDF terms and
conditions in the following areas, where a very high percentage of project files were
found to be compliant:

         Recipient eligibility (97%) and qualifications (100%);
         Project eligibility criteria related to maximum project value (100%) and
          allowable expenditures (89%);
         Financial oversight in the form of the right to audit (91%) and the existence of a
          payment authority (94%); and
         Proposal review, including evaluation by the program managers (91%) and
          sign-off by the regional director general (RDG) (83%).

For the remaining five terms and conditions (described in bulleted text below; also see
Table 4.2), the high rates of incomplete information in the files, when considered in
combination with evidence from other sources, do not allow researchers to reach a
conclusion concerning compliance with the EDF guidelines. In particular, the absence of
a centralized information management system (see Section 3.2.5 Challenges and
Limitations) suggests that it is possible that the high rate of missing information reflects
the poor accessibility of this information given current data management practices rather
than noncompliance with these conditions. For these five guidelines, the file review
found that:

         Project files contained no information pertaining to whether the contribution was
          consistent with the application and decision or met the court’s conditions
          in 94% of the sampled cases. Considering that these conditions are central to
          the review of project proposals and seminal to the development of the
          contribution agreements (i.e., the contribution agreement is based on the
          application and the application must meet court conditions to be approved), the
          absence of project information concerning these agreement conditions may
          simply reflect a lack of apparent need to document this information. As well,
          there was no suggestion from key informants that projects were ever at odds
          with court conditions and a few noted that the EDF program had no obligation to
          report back to the court upon the completion of project with an assessment of
          whether the projects satisfied court conditions.
         While there was evidence to assess compliance with the cost-effectiveness for
          26% of sampled files, this information was missing in 71% of cases. This is most
          likely due to an absence of any formal requirement for individual cost-
          effectiveness analyses to be conducted in any program documentation.41 As



41
     While a number of documents make passing reference to the need for projects to be cost-
     effective (Environment Canada. Undated. Environmental Damages Fund Applicant’s Guide.
     Ottawa; Environment Canada. 2006. Environmental Damages Fund Grants and Contributions


Environment Canada                                                                         28
Audit and Evaluation Branch                   Evaluation of the Environmental Damages Fund


        well, there is a widespread belief among key informants that projects are cost-
        effective.
       The high rate of missing information (71%) related to invoicing (i.e.,
        payment methods on invoice with a statement of expenditures) is
        surprising considering that the Recipient’s Handbook devotes an entire
                                         42
        chapter to ―Project Accounting.‖ The handbook stipulates that payments
        should be made by cheque and that when bills are paid, recipients should
        ensure that a ―proper invoice was received‖ and that they ―write “Paid” by
        the cheque number on [their] record of all paid bills.‖ Once again, this
        rate of missing information may reflect data accessibility rather than
        noncompliance.
       Half (49%) the files provide insufficient information to confidently assess
        whether any equipment was purchased as part of the project.




   Approval Form. Ottawa; Environment Canada. 2000. Introducing the Environmental Damages
   Fund, program brochure.), there is no formal requirement for project proponents to
   demonstrate the project‘s cost-effectiveness when applying for EDF funding. The closest any
   program documentation makes to an assessment of cost-effectiveness is the EDF project
   Technical Review Evaluation Form, where an aggregate rating (out of 5) of the project‘s budget
   considers cost-effectiveness in combination with the budget detail and reasonableness of costs
   (Environment Canada. 2005. Results-based Management Accountability Framework, Appendix
   C. Ottawa).
42
   Environment Canada. 2007. Environmental Damages Fund Recipient’s Handbook. Ottawa.


Environment Canada                                                                             29
Audit and Evaluation Branch                         Evaluation of the Environmental Damages Fund


                                   Table 4.2:
   Project Compliance with Terms and Conditions for Contribution Agreements

                                            COMPLIANCE RATING (%)
       TERMS AND                                                                            OVERALL
       CONDITIONS                                                    NO                      RATING
                                 FULL        PART         NON                    N/A
                                                                    INFO.
1 Recipients must be
eligible to receive the            97          0           0           3          0        Compliant
monies
2 The contribution is
                                                                                              No
consistent with application         3          3           0          94          0
                                                                                           conclusion
and decision
3 The project meets the                                                                       No
                                    3          3           0          94          0
court‘s conditions                                                                         conclusion
4 The project is cost-                                                                        No
                                   20          6           3          71          0
effective                                                                                  conclusion
5 The recipient possesses
the skills necessary to            74          26          0           0          0        Compliant
deliver the project
6 Projects undertaken build
                                   54          0           20         26          0        Compliant
on partnerships
7(a) Proposals are
evaluated by program               83          9           0           9          0        Compliant
managers
7(b) Proposals are signed
                                   83          0           0          17          0        Compliant
off by RDG
8 A ceiling of $5 million per
                                  100          0           0           0          0        Compliant
award
9 Awards cover allowable
expenditures (not basic            89          0           3           9          0        Compliant
costs)*
10 Payment methods are
                                                                                              No
on invoice and with                9           0           20         71          0
                                                                                           conclusion
statement of expenditures
11 Equipment purchased                                                                        No
                                  52*          0           0          49          0
belongs to the project                                                                     conclusion
12 There exists a right to
                                   91          0           0           9          0        Compliant
audit each project
13 There exists a payment
                                   94          0           0           6          0        Compliant
authority
* In roughly 90% of cases for which ownership of equipment purchases was identified, the contribution
  agreement states that ―the ownership of any equipment purchased by the Recipient with funds provided by
  the Minister under this agreement shall rest with the Recipient.‖ This was interpreted to mean that the
  equipment purchased belongs to the project.




Environment Canada                                                                                    30
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund


Summary: The EDF is subject to 13 terms and conditions of operation.43 The
review of project files confirms that the vast majority of projects are in
compliance with most of the terms and conditions for EDF contributions. For five
terms and conditions, the high rates of incomplete information, as well as
evidence from other sources that suggest the conditions may have been met, do
not allow the evaluators to reach a conclusion concerning compliance.

4.2.3 Environmental Protection Alternate Measures (EPAMs)

As previously discussed in Section 2 of the present report, Environmental Protection
Alternate Measures (or EPAMs) allow for the negotiated settlement of certain offences
between an accused and the Crown after charges are laid for a violation of the
Canadian Environmental Protection Act, 1999 (CEPA 1999).44

Key informants from the judicial segment and from EC45 were asked if the use of EPAM-
negotiated settlements was being encouraged and if their frequency of use would likely
rise or fall in the future. EC respondents most frequently reported that they did not know
and so could not comment on the promotion or expected use of EPAMs, either now or in
the future. Some also indicated that they do not know what an EPAM is, which may not
be surprising since they are rarely used and only then for offences prosecuted under
CEPA 1999.

A lack of familiarity with EPAMs was also evident among key informants in the judicial
segment, some of whom remarked that they had never been involved with an EPAM
settlement or were unaware of Environment Canada‘s connection to such activities.
Even one judicial key informant with prior experience using EPAMs was uncertain about
the current promotion of EPAMs or if prosecutors were encouraging their use.

Of the few key informants who were knowledgeable enough to comment on the use of
EPAMs, opinion was divided about whether the frequency of their use would grow or
diminish in the future. One EC respondent noted that with more enforcement of
environmental laws, more charges will likely be laid and so their use would likely rise,
while others felt that their use would diminish given that the process is perceived to be
―large and unmanageable‖ or that pollution events are on the decline.

Most judicial respondents were equivocal about whether the use of EPAMs will increase.
A couple of these respondents noted that any increase in their use will likely be limited
by the possibility of EPAMs being perceived as a means for offenders to buy their way
out of breaking environmental laws. Other factors that judicial respondents think may
result in a decreased use of EPAMs include:




43
   Environment Canada. 2005. Environmental Damages Fund Results-based Management and
   Accountability Framework - Draft 7, Appendix A. Ottawa.
44
   PWGSC Government Consulting Services. 2007. Environmental Damages Fund Process
   Analysis. Ottawa.
45
   Not all EC respondents were asked for their views on EPAMs, as this question was included in
   the questionnaire only after fieldwork had begun.


Environment Canada                                                                            31
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund


           a greater focus on detection/investigation of serious environmental crime,
            such that the number of prosecutions for which it would be appropriate to use
            EPAMs (i.e., minor or less serious infractions) does not increase or declines;
            and
           the cumbersome nature of EPAMs, which suggests they are unlikely to be
            chosen when looking for the flexibility of alternative measures.

Alternatively, some judicial respondents suggested that EPAMs might be used more in
light of increasing public pressure to deal with environmental offences and to
demonstrate some beneficial outcome in the communities where the offense took place,
as well as if heightened enforcement of environmental laws occurs for both serious and
less serious environmental infractions where the use of EPAMs may be more
appropriate.

Summary: EC personnel and, to a lesser extent, the judicial respondents report
very low levels of awareness of the function and/or use of EPAMs. Of those
respondents who are familiar with EPAMs, opinion was divided as to whether
their use would grow or diminish in the future. While increased enforcement
activity and pressures to deal with environmental offences suggest their use may
rise, the cumbersomeness of the EPAM process and the possibility that the
number of cases for which EPAMs can be appropriately applied does not increase
are equally compelling reasons to predict that their use may diminish.


4.2.4 Accountability


   Evaluation Issue                         Indicator(s)             Methods        Rating
Who is accountable for the          Defined and known program     Document     Progress
program?                             management structure for       review       made,
  Are the roles and                 program                       Interviews   Attention
    responsibilities of all         Views of program staff and                  needed
    groups involved clear?           recipients
  Is there an appropriate
    accountability framework
    (e.g., for multi-stakeholder
    agreements)?

A number of EDF program documents outline the roles and responsibilities of the
various parties involved in the administration of the program and delivery of individual
projects:

       An EDF governance framework identifies the lead position and/or agency
        responsible for each stage of the oversight and management of pollution
        incidents from which penalties flow to the EDF, from the incident notification and
        damage assessment to EDF administration, project implementation and




Environment Canada                                                                         32
Audit and Evaluation Branch                     Evaluation of the Environmental Damages Fund


         completion. The framework also includes a description of activities that occur at
         each stage of the process.46
        The EDF Results-based Management Accountability Framework outlines the
         roles and responsibilities for administration of the program within Environment
         Canada, including Ministerial, Regional Director General and OPG Lead (i.e.,
         Director General) accountabilities and responsibilities.47
        The application guide details the roles and responsibilities of different groups
         involved in the evaluation of program applications and the expected time to
                                   48
         process the application.
        The Recipient’s Handbook outlines the roles and responsibilities of funding
         recipients vis-a-vis the program, including details related to project accounting,
         reporting, communications activities and project closure.49
        An EDF communications plan, which includes an internal communications
         strategy involving InfoLane updates, one-on-one meetings, a national EDF
         meeting, lunch-and-learn presentations, and education/information sessions.
         One purpose of the strategy is to ―explain [the Fund‘s] purpose, the role of each
                                                                      50
         region in administering the Fund and general operations‖.

Most EDF managers and federal respondents report that the roles and responsibilities of
various players involved in the EDF are clear, most often noting that there is good
documentation to outline expectations and roles, which are also actively communicated
to affected staff and partners. Nonetheless, roughly three in ten respondents from this
group felt roles and responsibilities are not clear, or expressed some reservations in this
regard. These respondents most often felt there is need for more consistency and
oversight in the way the program is delivered across regions,51 noting also that efforts to
provide national program oversight and coordination are relatively new. These
respondents also commented on the need for better communication between
enforcement officers and EDF staff to track what funds are flowing to the program and
seek input on funding decisions, training for program officers, and more information
about project selection, approvals and funding processes.

Similarly, while the majority of funding recipients felt their roles and responsibilities and
those of the various partners involved in the delivery of the EDF program were clear,




46
   Environment Canada. 2006. Environmental Damages Fund Framework. Ottawa.
47
   Environment Canada. 2008. Environmental Damages Fund Results–Based Management
  Accountability Framework. Ottawa.
48
   Environment Canada. undated. Environmental Damages Fund Applicant’s Guide. Ottawa.
49
   Environment Canada. 2007. Environmental Damages Fund Recipient Handbook. Ottawa.
50
   Environment Canada. 2007. Communications Plan for the National Promotion of the
   Environmental Damages Fund (EDF) Program 2007 – 2009. Ottawa.
51
   For example, a 2007 process review of the EDF recommended that the program establish a
   clear separation of duties when it was found that certain tasks, particularly those related to the
   diversion of funds from court awards to the EDF, could be performed by a number of different
   organizations and varied from region to region. Specifically, while offenders mail cheques to the
   courts in some regions (Pacific and Atlantic), enforcement officers take receipt of cheques in
   Quebec and in Ontario, it is sent to the Regional Director General (RDG) (PWGSC
   Government Consulting Services. 2007. Environmental Damages Fund Process Analysis.
   Ottawa).


Environment Canada                                                                                33
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund


they were not always sure about the roles and responsibilities of personnel assigned to
the EDF. Some recipients expressed doubt about the roles of staff because of the
uneven accessibility of EDF-assigned personnel or the perception that EDF-assigned
personnel were not equally familiar with the program: ―Different people had different
answers to some of my questions.‖ According to other recipients, EDF-assigned
personnel seemed confused about whether their first obligation was to the courts or to
the Department.

At the level of individual EDF projects, the review of project files demonstrates that the
roles and responsibilities of proponents and partners are typically very well articulated.
All files were found to include an accountability framework, the vast majority of which
included a very clear (83%) or moderately clear (14%) description of the roles and
responsibilities of project stakeholders.

Summary: A great deal of program documentation exists which outlines the roles
and responsibilities of different parties involved in the delivery of the EDF
program, and these are perceived to be clear for most EC personnel and funding
recipients as well. Nonetheless, a number of respondents expressed some
confusion in this area. In particular, respondents from both groups suggested a
need for more consistency and oversight in the delivery of the program across
regions (noted by EC personnel) and EDF program staff (noted by recipients).


4.2.5 Performance Measurement and Reporting


    Evaluation Issue                        Indicator(s)             Methods       Rating
Does the program identify clear    Demonstration of the          Document       Progress
deliverables and expected           program‘s or project‘s         review          made,
results?                            expected deliverables and     File review    Attention
   Is performance data
                                    results                       Interviews      needed
                                   Views of program staff and
    collected against program
                                    recipients
    activities/outcomes?
   If so, is collected
    information used to inform
    senior management/
    decision-makers?



Both administrative and key informant data suggest that the program‘s intended
deliverables and intended impacts are fairly clear. Key informants among EC personnel
and funding recipients appear to have a generally positive assessment of the clarity of
program deliverables and impacts, as two-thirds of recipients (66%) and nearly half
(46%) of program personnel reported that these were very clear (i.e., responding with a
6 or 7 on a seven-point scale). There is room to improve the clarity of deliverables and
results, however, as more than one-third of both recipients (35%) and EC personnel
(35%) feel the program‘s deliverables and expected results are only somewhat clear
(i.e., responding with a 3, 4 or 5 on a seven-point scale) (see Figure 4.2).



Environment Canada                                                                           34
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund




At the project level, analysis of administrative data shows that projects identify clear
deliverables and expected impacts in 97% of cases. Likewise, feedback from program
personnel suggests that the large majority feel performance indicators are developed
and clearly identified for EDF-funded projects. Several respondents, however, suggest
that the process of identifying indicators for EDF projects may not be as evolved as for
more established programs, such as EcoAction, where standard indicators have been
developed. One respondent also suggested that the identification of performance
indicators may present more of a challenge for projects involving education and
awareness, as the results of these projects are ―a little less tangible.‖

In terms of the collection of performance data for EDF-funded projects, key informant
and administrative data both provide evidence that this is being done to a large extent.
The review of administrative files found evidence that performance data were collected
for activities and outputs in 95% of project files reviewed and, to a lesser extent, for
outcomes in 83% of cases. Similarly, more than four in five funding recipients (83%)
reported that performance data was collected for their projects.

While the program is systematic in eliciting performance data on deliverables and
outcomes, Environment Canada key informants suggest that both types of performance
data are only transmitted irregularly or informally to senior management. In Atlantic
Canada, where the upward transmission of performance data seems relatively formal,
all the information is passed on in a report to the RDG of Atlantic Region. Elsewhere,
transmission of information to management is typically informal and may not include
information on small projects. In these regions, reporting of performance information


Environment Canada                                                                     35
Audit and Evaluation Branch                 Evaluation of the Environmental Damages Fund


may take the form of presentations or briefings that are sporadically made to the
Minister or senior management. Some EC key informants expect the flow of
performance data to senior management to become more formal and regular as a
consequence of the installation of a dedicated information management system, which
is currently under development.

Summary: Most EC personnel and recipients feel the intended deliverables and
impacts of the EDF program are at least somewhat clear, while project
deliverables and impacts are clearly identified in the vast majority of project files.
Strong evidence emerges from both key informants and administrative files that
performance data are collected for virtually all projects with respect to activities
and outputs and are collected for most but not all projects with respect to
outcomes. In most regions, no systematic process exists for communicating
program results to senior managers and decision-makers.


4.2.6 Strengths and Challenges

         4.2.6.1       Program Strengths
In the view of several Environment Canada personnel, the EDF‘s focus and design is
seen as its greatest strength, as the program allows fines to be applied directly toward
addressing the environmental damage that was done. A second commonly cited
strength among EC respondents is the perceived simplicity of the program, as it is felt to
be straightforward and easier to manage given the consistency of the EDF‘s design with
another Community Partnerships program called EcoAction. This too may explain
another key strength noted by many recipients, which is the simplicity of the application
process and well-defined deliverables.

Another key strength of the program identified by some recipients concerns the
establishment of partnerships with environmental, community and government
organizations. Similarly, several EC personnel note also that one of the program‘s
greatest strengths as it is delivered in the Atlantic region is the partnerships that have
been built within EC and with other government departments. As presented here in
Section 4.2.2.2, however, academics and potential recipients are less favourably
disposed to the idea of partnerships, believing partners are difficult to find, can create
conflict or perform poorly.

Potential recipients, academics and members of the judiciary were also asked to identify
strengths and best practices in the design and delivery of funding programs generally,
based on their experiences with programs similar to the EDF. Not surprisingly, each
group of respondents tended to identify somewhat different strengths and best
practices, as described in turn below:

      Potential recipients tend to feel a key strength and best practice related to the
       design and delivery of similar programs has been a non-competitive funding
       process, whereby the primary focus is to meet federal requirements and so
       simplify the application process. Other best practices involve issues related to:
       program oversight (i.e., minimal bureaucracy, reducing partnership
       requirements, support from advisory or steering groups, clearly communicated
       program objectives, peer review, and timely funding); the program focus (i.e.,


Environment Canada                                                                           36
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


       environmental improvement); and partnerships (e.g., provincial government
       partnership, matching funds to project).
      Respondents from the judicial segment often could not comment on strengths
       or best practices given their limited experience with similar programs, but one
       respondent felt the best practices of the EDF program are having court
       conditions (i.e., predetermined projects describing how money will be spent and
       how the environment will benefit) and supporting documents describing the
       program.
      Academics are of the view that timed reporting and accountability, peer review
       and training of individuals to carry on the work in related fields are the key
       strengths and best practices of similar programs.

        4.2.6.2       Key Challenges
In terms of program challenges, respondents from all groups identified many of the
same issues, chiefly a lack of resources for the program, court restrictions, internal
departmental processes, and engendering buy-in among key program partners. For
managers, challenges relating to resources concerned a lack of funds and staff to
support program delivery, while for recipients and potential applicants, mention was
most often made of the low levels of EDF project funding available, the need to rely on
other partners and the short timelines for funding. Both managers and recipients also
noted that it was challenging to use and access EDF resources. This latter challenge
likely stems from both the absence of resources for administration of the program, as
well as from court conditions imposed on EDF funding. Challenges related to addressing
court conditions were frequently mentioned by managers, recipients, and justice
respondents, as they caused difficulties in matching project submissions to the narrow
conditions and criteria for funding. One manager also noted that court conditions make it
challenging to plan program activities because funding for different priorities and
geographic regions varies from year to year.

Internal processes were also frequently mentioned by managers, academics, potential
applicants and recipients as posing challenges, particularly in terms of delays in
receiving funding approval and the administrative burden related to proposal
development and project reporting for proponents with limited capacity or for very small
projects where the level of effort may not be commensurate to the size of the
contribution. Some respondents also question the merits of current levels of project
reporting and management. One academic and one potential applicant argue that
project deliverables and indicators are rarely representative of a project‘s true impact on
society, while a second academic noted that a focus on project management issues can
dilute the quality of the science funded through such programs by diverting energies
away from the project itself.

Finally, several recipients and federal respondents noted difficulties related to
engendering buy-in among key program partners, and in particular to promoting the
EDF‘s use among members of the judiciary. As a remedy, one federal respondent
suggested the program could improve its reporting and promotion of program results
and success stories to all program partners. Several EC respondents also remarked that
it was often challenging to encourage Enforcement personnel in the Ontario region to
promote the EDF while the regionally specific alternative ECCAP program was still in
existence. Although the ECCAP no longer exists as an alternative, one EC respondent


Environment Canada                                                                       37
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


noted that there is still work to be done in terms of promoting buy-in to this ―new‖ EDF
program.


4.3    Success

Overall Findings:
The EDF is generally perceived by recipients and Environment Canada personnel
to be successful in the achievement of most program outcomes, a finding which
is confirmed by administrative files demonstrating a clear achievement of project
deliverables and outcomes in a majority of cases. Evidence from key informants
and project files suggest that the program is most successful in contributing to
restoration objectives, and is somewhat successful in contributing to education
and awareness and research and development outcomes as well. The primary
factors underlying the degree to which different outcomes are achieved are
thought to include the level of resources, number of projects and/or number of
applications targeting the priority area.

The existence of clear documentation pertaining to the role recipients are
required to play in the financial oversight of funded projects, as well as the high
rates of projects conforming to audit and payment authority specifications,
suggests the program is being administered with good financial oversight and
accountability. While generally clear and well understood, the transparency of the
funding process at the project level could be enhanced through more consistent
staff support and better communications about court awards and funding
decisions between Enforcement and EDF personnel.

Resources to support the administration of the program overall are not tracked in
any systematic fashion, although regions are tracking the number and value of
court awards and contribution agreements. Past research suggests that overall
financial accountability and transparency would be enhanced through the
development of an information management system, which is already under
development.

The EDF program has made no apparent progress in increasing awareness of the
program among the legal community and potential applicants to the program, or
in increasing the court’s use of the EDF as a sentencing tool. Although the
Atlantic region has made some effort to make information related to EDF projects
public, this information was incomplete and out-of-date and no similar
information was found for other regions.

Efforts have been made to expand the use of EDF in the context of federal
legislation controlled by other departments. Although implicated departments
(i.e., DFO and Transport Canada) are involved in the technical review of project
applications, information transfer between federal partners could be improved by
sharing more information pertaining to final funding decisions.

The EDF program employs a number of controls and processes to ensure the


Environment Canada                                                                         38
Audit and Evaluation Branch                    Evaluation of the Environmental Damages Fund


quality and feasibility of project proposals. Evidence suggests also that program
materials on the application process and staff support are contributing to the
acquisition of proposal and planning skills by community groups, as evidenced
by the high rates with which projects achieve intended deliverables, the
consistency of proposals with EDF objectives, the use of partnerships, and the
ability of projects to leverage partner resources.


     Evaluation Issue                         Indicator(s)               Methods        Rating
To what extent have intended         Evidence of intended output     Document        Achieved
outcomes been achieved as a           and outcome achievement          review
result of the program?               View of program staff and       File review
                                      recipients                      Interviews

When asked to rate the extent to which the intended outcomes of funded projects have
been achieved, both Environment Canada personnel and recipients in particular tend to
provide positive assessments. Three-quarters of Environment Canada staff feel the
outcomes of EDF projects have been achieved to a large (50%) or moderate extent
(26%), while fully nine in ten recipients indicate large (83%) or moderate (11%)
achievement of project outcomes (see Figure 4.3). It is noteworthy that one in five EC
respondents report that they do not know whether outcomes had been achieved, likely
reflecting their lack of direct involvement in providing support to funded projects (e.g.,
enforcement personnel).52

Environment Canada personnel most often cited personal experience with projects to
explain their positive ratings of outcome achievement. Several also note that the
program is administered according to clear terms and conditions and that project
proposals are closely reviewed and monitored once implemented, while others note that
the capacity and qualifications of project proponents, and the support and resourcing
provided to projects may underlie the achievement of project outcomes. For several
more, assessing project success was more difficult either because the program is not
fully implemented in their region, a longer timeframe is needed to observe results, there
is no standard approach to evaluating projects or the results of projects aren‘t widely
promoted.

Several project proponents explained that their project‘s success was the result of
strong support from partners, other funders and the program (i.e., flexibility), as well as
the achievement of objectives that were beyond the scope of the original project
proposal. A few respondents noted less than 100% success resulting from unforeseen
circumstances that prompted design adjustments, while several others mentioned that it
was difficult to assess the achievement of outcomes because the project was either not
yet complete or ongoing, with results expected only in the longer term.




52
     Given the small sample size for non-EDF Environment Canada personnel, results for this sub-
      group could not be reported separately.


Environment Canada                                                                             39
Audit and Evaluation Branch                     Evaluation of the Environmental Damages Fund




Results of the review of project files are consistent with the perceptions of key
informants. For nearly nine in ten (86%) project files reviewed, clear evidence of
implementation of activities and the attainment of deliverables was found. A further one
in ten (9%) files showed ―partial evidence‖ of the attainment of activities and
deliverables, while only 6% found no such evidence. Some funding recipients reported
that the requirement for performance information on deliverables is a strength of the
program.

Administrative files revealed less performance information pertaining to outcomes than
to deliverables. Projects were deemed to have achieved intended outcomes if the final
report specified that intended outcomes had been observed, such as by using an
―indicators of [impact] success‖ checklist. Roughly three-quarters of project files
reviewed contained either a very (46%) or moderately (31%) detailed53 description of
outcomes achieved, while one in four project files described achieved outcomes with
only minimal detail (6%) or provided no such evidence (17%). Careful review of the
administrative files demonstrated that awareness and education activities were less
likely than others to have outcome data. This is likely explained by the complexity and/or




53
     Evidence was deemed partial in cases where the final project report acknowledged that only
     some impacts had been achieved or where interim reports contained evidence of progress
     toward achieving intended outcomes but the final report was missing.


Environment Canada                                                                                40
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund


high cost of collecting performance information for these types of activities (e.g., attitude
surveys).

In terms of the specific outcomes to which EDF projects contribute the most, federal
respondents were asked to indicate the extent to which funded projects contributed to
each of four priority areas related to restoration, environmental quality improvement,
research and development, and education and awareness. Federal respondents were
most likely to feel the EDF has contributed to environmental restoration, with more than
half reporting that the program has contributed somewhat or a lot to this outcome. Just
fewer than half felt the same about the program‘s contribution to education and
awareness while roughly one in four felt the program contributed at least somewhat to
research and development. Relatively few felt the program contributed to environmental
quality improvement. This may not come as a surprise considering that the focus of the
EDF is typically on restoring the ―same type, quality and value of the natural resource
lost‖ rather than on environmental quality improvements targeting ―different natural
resources in the locale.‖54

In general, respondents explained the program‘s impact on one or another outcome as
due to the number of projects associated with a given outcome, as well as the amount of
funding for projects in that area. In particular, they note that the focus on one or another
priority is often linked to court conditions, as well as to the types of projects being
proposed. Some key informants also attribute the frequency of environmental
restoration projects to a preference for these projects based on their highly visible and
measureable results and high success rates. The program‘s contribution to education
and awareness and to research and development was often credited to the ease with
which these outcomes can be incorporated into activities targeting other outcomes. As
well, the relatively low costs of including the public in projects aimed at improvements in
the communities affected by the environmental damage also contributed to the
achievement of education and awareness objectives.

Although the EDF‘s overall contribution to one or another of these four priorities may be
explained by the number and/or value of contributions in the priority area, the review of
project files shows very little difference in the degree to which individual EDF projects
are successful, regardless of their focus. With few exceptions in any priority area,
project files yielded at least partial evidence of the achievement of intended activities
and outcomes, with the vast majority of these project files containing comprehensive
evidence of their achievement. Evidence of outcome achievement, however, was slightly
less likely to be found for projects targeting environmental quality improvement and
education and awareness.

Summary: The EDF is generally perceived by key informants to be successful in
the achievement of most program outcomes, a finding which is confirmed by
administrative files demonstrating a clear achievement of project deliverables and
outcomes in a majority of cases. Evidence from key informants and project files
suggests that the program is most successful in contributing to restoration




54
     Environment Canada. Undated. Environmental Damages Fund (EDF) Applicant’s Guide, page
      4. Ottawa.


Environment Canada                                                                        41
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund


objectives, and is somewhat successful in contributing to education and
awareness and research and development outcomes as well. The primary factors
underlying the degree to which different outcomes are achieved are thought to
include the level of resources, number of projects and/or number of applications
targeting the priority area.


4.3.1 Project Delivery

     Evaluation Issue                       Indicator(s)                Methods        Rating
To what extent has the program    Evidence of intended output       Document        Progress
achieved intended project          and outcome achievement            review           made,
delivery outcomes?                View of program staff and         File review     Attention
                                   recipients                        Interviews       needed
     Financial accountability
     Transparency of funding
              55
      process

Outcomes related to the delivery of funded projects concern the extent to which the
program demonstrates financial accountability and the degree to which the funding
process is transparent. At the project level, the review of project files suggests that EDF-
funded initiatives are managed in a financially accountable fashion. For over nine in ten
project files reviewed there existed an explicit statement concerning the Government‘s
right to audit the project (91%) and a payment authority was identified (94%).

The EDF Recipient Handbook outlines the roles and responsibilities of funding
recipients, including details related to project accounting, reporting, communications
                                56
activities and project closure. It is not surprising, then, that most recipients expressed
a good understanding of the funding process and in particular of the source of EDF
funds and the specific restrictions for their use. Several recipients remarked that support
received from EDF staff ensured that the funding process was clear and straightforward,
although this transparency may be dampened somewhat by the uneven accessibility
and program expertise of EDF-assigned personnel.

The detailed accounting and reporting requirements for funded projects serve to
enhance the overall transparency of the funding process, but as noted previously, many
recipients remarked that the financial and other reporting for the program seemed
excessive in light of the amount of funding received. As well, program documentation is
not clear about whether leveraging financial or other resources through partnerships is a
requirement for EDF-funded projects (see Section 4.2.2.2) and a few recipients provided




55
   The program logic model (Annex 1) also lists ―EDF restoration objectives‖ as an outcome under
   the Project Delivery activity stream. This issue is addressed in the preceding section in the
   context of findings related to the achievement of overall program outcomes and the program‘s
   contribution to environmental restoration, environmental quality improvement, research and
   development, and education and awareness.
56
   Environment Canada. 2007. Environmental Damages Fund Recipient Handbook. Ottawa.


Environment Canada                                                                            42
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


conflicting feedback concerning whether partners were or were not a requirement for
their projects.

At the program level, the absence of dedicated resources for the EDF‘s administration
means that no systematic tracking of human or financial resources to administer the
program is taking place. Nonetheless, the program has maintained detailed records by
region and calendar year of the overall number and value of court awards directed to the
EDF, as well as the number and value of contribution agreements supported through the
program. This information is tracked separately by regional offices and aggregated by
the program headquarters. While a 2007 process review concluded there was ―no
compelling reason to change the current system based on anticipated growth,‖ it was
also suggested that the development of a centralized information management system
(IMS) would improve the visibility of project information, the consistency of information
available across community funding programs, and make information related to funding
                                       57
activity levels more readily available. Work has already begun to adapt the IMS used
for the EcoAction program for use by the EDF program, but this new system is not yet
operational.

The development of an IMS might also enhance transparency and information-sharing
between the EDF program and its partners. In particular, a number of federal
respondents felt there was a need for better communication between enforcement
officers and Community Partnerships staff to track what funds are flowing to the
program and seek input on funding decisions, training for program officers, and more
information about project selection, approvals and funding processes.

Summary: The existence of clear documentation pertaining to the role recipients
are required to play in the financial oversight of funded projects, as well as the
high rates of project agreements containing audit and payment authority
statements, suggests the program is being administered with good financial
oversight and accountability. While generally clear and well understood, the
transparency of the funding process at the project level could be enhanced
through more consistent staff support.

Resources to support the administration of the program overall are not tracked in
any systematic fashion, although regions are tracking the number and value of
court awards and contribution agreements. Past research suggests that overall
financial accountability and transparency would be enhanced through the
development of an information management system, already under development.




57
     PWGSC Government Consulting Services. 2007. Environmental Damages Fund Process
     Analysis. Ottawa


Environment Canada                                                                     43
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund



4.3.2 Communications and Program Promotion

Evaluation Issue                           Indicator(s)              Methods         Rating
To what extent has the program     Evidence of intended output    Document           Little
achieved intended                   and outcome achievement         review          Progress,
communications and promotion       View of program staff and      File review     Priority for
outcomes?                           recipients                     Interviews      Attention

    Increased awareness of
     EDF by the legal
     community
    Increased use as a
     sentencing option by the
     courts resulting in
     monetary awards
    Awareness of the EDF
     among recipient
     organizations
    Public information on EDF-
     funded projects and on
     their geographic location

In order for the EDF to function properly, it is important for parties involved in the
prosecution of environmental infractions (i.e., prosecutors, investigators and
judges) to be aware of the EDF program and how it operates.58 Research
suggests that enhancing stakeholders‘ familiarity with and interest in using the
EDF requires a significant investment, but that understanding and judicial
precedents are also necessary for the program to be accepted as a vehicle to be
used in prosecuting environmental offences.59 Regional growth of the program
similar to what has been observed in Atlantic Canada is contingent on greater
                                60
promotion and use of the fund.

Previous studies have also found that ―…counsel for the polluter and even Crown
counsel and government officials were not familiar with the EDF‖ 61 and feedback from
key informants for the current evaluation suggests that awareness of the EDF has not
increased appreciably among external stakeholders in the interim. A large majority of
potential recipient organizations reported being unaware of the existence of the EDF,
while more than half of the academics interviewed acknowledged a lack of awareness.
Although all key informants in the judiciary reported being aware of the EDF, they also
emphasized that most judges and many crown attorneys are unaware of the EDF
program. According to court/judicial key informants, it appears that the courts become




58
   HDP Group Inc. 2002. Environmental Damages Fund Evaluation, pages 3, 5 and 6.
59
   Ibid, page 8.
60
   PWGSC Government Consulting Services. 2007. Environmental Damages Fund Process
   Analysis, page 11. Ottawa.
61
   Wruck, J. 2004. The Federal Environmental Damages Fund. The Advocate. Vol. 62-2, page 8.


Environment Canada                                                                           44
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


aware of the EDF on a case-by-case basis mainly from the information provided by the
crown attorneys.

These results may not be surprising if we consider that promotion of regional EDF
awareness and use by the legal community, recipient groups, and other government and
                                                                              62
nongovernment stakeholders is the responsibility of EDF regional managers. The
absence of dedicated resources for program implementation and evidence of uneven
regional implementation of the program would suggest that little promotional activity is
taking place in many EC regions. It should be noted, however, that in Atlantic Canada,
where the program has a longer history and is far better developed, EC key informants
report that efforts have been systematically undertaken to educate the judiciary.

The apparent lack of awareness of the program among the courts is consistent with
administrative evidence which suggests there has been little increase in the court‘s use
of the EDF as a sentencing tool. Since 2000, the number of court awards to the EDF
has remained steady at between 10 and 13 awards per year, with a larger number of
awards occurring in 2004 (n = 18), 2007 (n = 20) and 2008 (n = 18). While the larger
number of awards in 2007 and 2008 may be indicative of the initial stages of an
increasing trend in the number of court-awarded contributions to the EDF, the continued
absence of dedicated support to the program, uneven regional implementation and low
awareness of the program in the courts suggest that this is unlikely the case.

Finally, there is little evidence to suggest that the EDF has made progress in making
information on EDF-funded projects and their geographic location available to the public.
The only evidence of efforts to publicize this information is a reference to completed
                                                                  63
EDF projects on Environment Canada‘s Green Lane Web site. The information posted
on the EC Web site, however, only included links to projects completed in Atlantic
Region and only until 2005. As well, not all links for a given year were found to be active.

Summary: The EDF program has made no apparent progress in increasing
awareness of the program among the legal community and potential applicants to
the program, or in increasing the court’s use of the EDF as a sentencing tool.
Although Atlantic Region has made some effort to publicize information related to
EDF projects, this information was incomplete and out-of-date and no similar
information was found for other EC regions.




62
   Environment Canada. 2005. Environmental Damages Fund Results-based Management and
    Accountability Framework, table 2. Ottawa.
63
   Environment Canada. 2006. Environmental Damages Fund Funded Projects. Accessed at:
    http://www.ec.gc.ca/edf-fde/default.asp?lang=En&n=24600456-1


Environment Canada                                                                       45
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund



4.3.3 Partnerships

     Evaluation Issue                       Indicator(s)               Methods       Rating
To what extent has the program    Evidence of intended output      Document       ~Progress
achieved intended partnerships     and outcome achievement           review           made,
outcomes?                         View of program staff and        File review      some
                                   recipients                       Interviews      attention
     Interdepartmental                                                              required
      cooperation and
                           64
      information transfer

The EDF has been reasonably successful in engendering interdepartmental cooperation
and information transfer. To begin, use of the fund is not restricted to Environment
Canada legislation65 but can also be applied to the management of court awards and
penalties meted out through legislation governed by other federal departments,
                                                           66                        67
including the Department of Fisheries and Oceans (DFO) and Transport Canada.
Furthermore, the Government has signalled the expanded use of the EDF by amending
three Parks Canada statutes68 to enable awards and fines to be directed to the EDF. In
regions where the program is being consistently implemented, implicated departments
(i.e., DFO, Transport Canada, and Agriculture and AgriFood Canada) are also invited,
as appropriate according to the nature of the project to be reviewed, to be members of
technical review teams for EDF funding applications.

In terms of interactions between departments, EC respondents generally commented
that these interactions were positive. Non-EC federal respondents were divided on
whether these interactions could be improved, however, noting only that they would like
more follow-up information on final funding decisions and that it is challenging to discuss
and review proposals when the technical review team members are not all situated in
the same area.

Similarly, although most Environment Canada and non-EC federal respondents report
that the roles and responsibilities of various players involved in the EDF are clear (see
Section 4.2.4), many also note the potential for improvement in this regard. Specifically,
several remarked on the need for more consistency and oversight in the way the
program is delivered across regions and better communication between federal
stakeholders to share information on awards directed to the program and project
selection and funding.




64
   The program logic model lists ―increased court-awarded contributions to the EDF‖ as both a
   partnership outcome and a communication outcome. This result is addressed in the discussion
   of communications and program promotion outcomes in Section 4.3.1.
65
   Migratory Birds Convention Act (MBCA 1994), Species at Risk Act, and the Canadian
   Environmental Protection Act, 1999 (CEPA 1999).
66
   Fisheries Act.
67
   Canada Shipping Act, 2001
68
   The Parks Canada statutes include the Canada National Parks Act, the Canada National
   Marine Conservation Areas Act, and the Saguenay–St. Lawrence Marine Park Act.


Environment Canada                                                                          46
Audit and Evaluation Branch                    Evaluation of the Environmental Damages Fund


Summary: Efforts have been made to expand the use of EDF in the context of
federal legislation controlled by other departments. Although implicated
departments (i.e., DFO and Transport Canada) are involved in the technical review
of project applications, information transfer between federal partners could be
improved by sharing more information pertaining to final funding decisions.


4.3.4 Proposal Planning and Development

     Evaluation Issue                         Indicator(s)               Methods        Rating
To what extent has the program       Evidence of intended output     Document        Achieved
achieved intended proposal            and outcome achievement          review
planning and development             View of program staff and       File review
outcomes?                             recipients                      Interviews

      Scientific and technically
       sound, feasible and cost-
       effective restoration
       projects
      Community groups acquire
       knowledge and skills for
       proposal development and
       restoration planning

As discussed previously in Section 4.2.2.3, the available evidence suggests the EDF
has been successful in encouraging scientific, technically sound and feasible restoration
projects. The review of project files revealed a majority of funding recipients are eligible
to receive program funds (97%) and possess the necessary skills to deliver the project
(74%), both of which speak to the feasibility of the proposed initiative. As well, the file
review found firm evidence of internal reviews of project proposals to ensure their
scientific and technical merit for a large majority of proposals, as more than four in five
project files were evaluated by program managers (83%) and signed off by the RDG
(83%).69 Much less evidence, however, was found to demonstrate the cost-effectiveness
of individual project proposals, with only one in four (26%) project files containing
sufficient information to determine that the project was cost-effective. Seventy-one per
cent of project files contained too little information to assess the cost-effectiveness of
the project.

Some program documentation exists to support applicants to the programs in acquiring
knowledge and skills for proposal development. The EDF Applicant’s Guide defines
eligible and ineligible projects and outlines required information for applying to the
program, including details relating to required applicant information, the proposed
project team, a project summary, the rationale or need for the project, evaluation plans,
communications activities, a funding summary, and budget.




69
     There was insufficient information to draw a conclusion about the manager‘s review and RDG
     sign-off in 17% of project files.


Environment Canada                                                                                47
Audit and Evaluation Branch                       Evaluation of the Environmental Damages Fund


Although recipients‘ frequent mentions of a reliance on program staff to assist them with
their applications and proposal development suggests many do not yet possess
expertise in the development of proposals, this close contact with program personnel
likely reflects a great deal of information sharing and tutelage. A review of administrative
data would seem to confirm that EDF applicants are acquiring these skills, as nearly four
in five proposals to the EDF are approved (78%), and over nine in ten (92%) meet
eligibility requirements for the program. While the high project approval rates are likely
due at least in part to low awareness of the program and a corresponding lack of
competition between eligible projects, it is reasonable to conclude also that the high rate
of project eligibility is indicative of the quality of the applications received.

The review of administrative data also demonstrates that funding recipients have
developed a certain expertise in the planning of restoration and other EDF projects. Of
91 approved projects supported through the EDF since its inception, none of the final
project budgets was found to have exceeded the approved value. Further, the review of
administrative project files demonstrates that the projects are well administered in terms
of:

          The achievement of project milestones and deliverables (i.e., performance data
                                                                                       70
           are collected for activities and outputs in 95% of project files reviewed);
          The degree to which project proposals conform to EDF objectives (i.e., 97% of
           recipients of projects reviewed are eligible to receive monies);71
          The numbers of community project partnerships (i.e., 54% of projects are
           delivered in partnership); and
          The value of partner-leveraged funding (i.e., average total project costs
           [approximately $77,000] are nearly double the EDF contributions [approximately
           $36,000]).

Summary: The EDF program employs a number of controls and processes to
ensure the quality and feasibility of project proposals. Evidence suggests also
that program materials on the application process and staff support are
contributing to the acquisition of proposal and planning skills by community
groups, as evidenced by the high rates with which projects achieve intended
deliverables, the consistency of proposals with EDF objectives, the use of
partnerships, and the ability of projects to leverage partner resources.




70
     Insufficient information was available to make a determination about the remaining project files.
71
     Ibid.


Environment Canada                                                                                  48
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund



4.3.5 Unintended Outcomes

  Evaluation Issue                       Indicator(s)             Methods       Rating
Have there been any             Presence/absence of           Document          NA
unintended (positive or          unintended outcomes            review
negative) outcomes? Were any    Documented management         File review
actions taken as a result?       actions and/or lessons        Interviews
                                 learned from unintended
                                 outcomes
                                Views of program staff and
                                 recipients


In addition to the intended outcomes of EDF-funded projects, approximately half of EC
key informants identify unintended outcomes. For the most part, these respondents
report positive outcomes involving stronger intradepartmental, interdepartmental, and
intergovernmental relationships. Roughly half of the funding recipients interviewed
highlighted generally positive unintended benefits resulting from their projects. These
include unexpected sources of partnerships and financial support at the local level,
unforeseen positive impacts on community attitudes and awareness, and unanticipated
discoveries in the realm of science and technology.


4.4    Cost-effectiveness

Overall Findings:
Overall, the evidence suggests that the EDF is cost-effective. Projects have been
successful at leveraging funds from other sources and some limited file review
evidence suggests that individual projects are more likely than not to be
evaluated as cost-effective. At the program level, no other federal programs exist
to manage specified purpose funds to carry out environmental restoration
activities and overall administrative spending is low, particularly in light of the
national scope and specificity of court conditions related to funded projects.

The evidence is inconclusive concerning the advantages of introducing a
minimum threshold value for project funding in order to increase overall program
efficiency. While cost-efficiencies might be realized by imposing a minimum value
for EDF-funded projects, doing so could impair small projects whose
requirements for restoration or other environment contributions could be lower
than a potential award threshold.




Environment Canada                                                                     49
Audit and Evaluation Branch                       Evaluation of the Environmental Damages Fund




         Evaluation Issue                       Indicator(s)              Methods             Rating
Are there alternative ways of          Analysis of delivery               Document         Achieved
achieving the objectives of the         options/opportunities               review
program?                               Views of Finance and               Interviews
  If the program or activity           program staff                                           N/A
     continues, how could its
     efficiency be improved?
  Are others involved in the
     same areas of activities                                                                Achieved
     and/or share similar
     objectives? How is
     duplication avoided and
     complementarity
     achieved?
  Has the program provided            Analysis of costs of program       Document         Achieved
     value for federal dollars          compared with achievement           review
     spent?                             of program outcomes                File review
                                       Selection process for areas
                                        of importance are applied

The cost-effectiveness paradigm focuses on the following four analyses, two of which
explore the issue of cost-effectiveness at the project level and two of which examine this
issue at the program level.72

Project-level analyses of cost-effectiveness consider evidence related to:
      Leveraging – the extent to which funded projects are successful in attracting
        resources from outside EDF and EC, thereby multiplying the impact and hence
        the cost-effectiveness of EDF resources alone;
      Project expenditure costs – the extent to which the goods and services procured
        under individual projects were economical and hence cost-effective.

Program-level cost-effectiveness analyses consider evidence related to:
      Redundancy risk – the extent to which project activities could have been
       undertaken by alternative programs; and
      Administrative spending – a comparison of administrative spending to
       contribution outlays.

At the project level, evidence of leveraging is strong. In the sample of projects (n = 35)
whose files were examined, recipients reported average contributions of approximately
$36,000 as compared to average total project costs of approximately $77,000. This
roughly equates to a one-to-one ratio of EDF funding to leveraged funds and suggests




72
     For practical reasons, the evaluation approach did not include individual analyses of project files
     to assess their cost-effectiveness, and so relied on existing information. It would be prohibitive
     to undertake such an assessment in the context of the current evaluation. However, several
     funding recipients did note unexpected requirements for financial resources and the resulting
     necessity of stretching their funding.


Environment Canada                                                                                    50
Audit and Evaluation Branch                    Evaluation of the Environmental Damages Fund


that project contributions benefited from significant leveraging. With the exception of
three projects, project costs exceeded the value of EDF contributions.

In a minority of instances, the project files also contained evidence that project officers
themselves had reached a conclusion that the individual project was cost-effective. In
23% of cases, ―program officers or auditors assessed cost-effectiveness and reached a
positive conclusion‖ (20%) or there was sufficient evidence to suggest the project was
cost-effective (3%). In contrast, in only 6% of project files reviewed (n = 2) did the
evidence (3%) or a direct assessment73 (3%) suggest the project was not cost-effective.
In the vast majority of cases (71%), however, the project files contained no evidence of
efforts independent of the funding recipients to verify the cost-effectiveness of individual
projects. Although the paucity of evidence related to the cost-effectiveness of individual
projects does not allow firm conclusions to be drawn, it is nonetheless important to note
that projects were much more likely to be assessed as cost-effective than not.

At the program level, cost-effectiveness involves a consideration of redundancy risk. If a
significant redundancy risk existed that either projects could have been supported or
court awards managed by an alternative program, it would be theoretically possible that
the alternative program might have been more cost-effective. Although several G&C
programs within Environment Canada exist which might in theory have supported EDF-
                 74
funded projects, there is no evidence that projects could have been supported with
greater cost-effectiveness. Furthermore, and as discussed previously in Section 4.1, the
EDF is a unique federal mechanism for managing court awards stemming from
prosecutions under various legislative authorities and for supporting creative sentencing
by the courts in environmental damages cases. Consequently, it is reasonable to
conclude that no redundancy risks exist in the context of the EDF program.

The absence of any dedicated funding to support the EDF program and consequent
lack of any system to track administrative costs for program delivery means that it is not
possible to evaluate the program‘s overall cost-effectiveness with any certainty (see
Section 4.2 for a more complete discussion). Nonetheless, Community Partnership
program estimates of fewer than four person-years per annum for the administration of
EDF projects would appear to be cost-effective in light of the program‘s commitment to
reflecting regional realities, the diversity of environmental subjects addressed by the
awards, and the administrative costs that flow from very specific court-award conditions.

EC personnel almost universally agree that the program provides a great value for
federal dollars spent. Respondents generally note that this is because the program costs
almost nothing for the Department to deliver, while at the same time the recipients
leverage additional dollars, resources and expertise. In Atlantic Region, other federal



73
   In the one case where project officers reached a conclusion that the individual project was not
   cost-effective, the conclusion was not based on cost-effectiveness per se but rather on the
   absence of potential partners capable of supplementing the EDF contribution with additional
   financial or non-financial support.
74
   For example, one EcoAction priority related to nature is to support ―projects focusing on
   protecting wildlife and plants, and protecting and improving the habitat where they live (e.g.,
   grasslands, rivers, forests, etc.).‖ (Environment Canada. 2004. Results-based Management
   and Accountability Framework for the EcoAction Community Funding Program. Ottawa).


Environment Canada                                                                               51
Audit and Evaluation Branch                    Evaluation of the Environmental Damages Fund


departments are also now contributing fines to the EDF. When the views of respondents
are considered together with evidence suggesting that the program has been successful
in achieving outcomes (i.e., majorities of project files contain clear evidence of the
achievement of deliverables [86%] and outcomes [60%] – see Section 4.3), and the
apparently low administrative costs for delivery of the program, the evaluation evidence
suggests that the EDF provides good value for federal dollars spent.

EC personnel also made several suggestions to improve the efficiency of program
delivery. The two most prominent were a national communications strategy to promote
the program and its successes, and less paperwork for the funding recipients. Other
suggestions included the development of national guidelines for administering the EDF
and the development of better indicators to measure the program outcomes.


4.4.1 Minimum Project Size

The terms and conditions for EDF contribution agreements require that contributions
have a ceiling of $5 million and no minimum or threshold value. The principle of internal
economies of scale would call for a minimum award value, however, as the proportional
costs of administrative oversight can safely be assumed to be higher for small
projects.75 Ongoing pressure for efficiency in Government of Canada programs, the
modest administrative capacity of the program, and the administrative burdens imposed
by stringent court conditions all call for attention to be devoted to protecting and
improving efficiency.

The issue of a possible minimum or threshold size for project funding elicits mixed
reaction. EC personnel tend to oppose a threshold while academics and potential
                                                                             76
recipients support it. Existing funding recipients are divided on the matter. Opponents
of a threshold tend to perceive a potential minimum figure as a threat to small
organizations, whose restoration or other projects do not require large sums since the
amount of environmental damage is small. The size of court awards and their potential
conditions also weigh against having a threshold minimum. In the presence of a
threshold, it would be difficult for the EDF to administer court awards that have
conditions imposed on their use and that fall below the threshold size unless they can be
combined with other awards that have identical or compatible geographic and court
conditions.

Supporters of a threshold favour a minimum figure because they perceive the burden
(i.e., application, reporting) on both applicants and assessors as disproportionate to the
amount of the award. In practice, the average contribution is $22,000 over the life of the
program, and fully one-third of all awards have been under $10,000. EC personnel who
favour a threshold call for a low minimum amount of $8,000, while funding recipients




75
   Every project, however small, entails inherent assessment, processing, oversight, and audit
   costs. The proportional costs for these functions are necessarily higher for small projects.
76
   The respective ―yes/no/do not know" percentages for EC key informants are 28/61/11;
   academics, 78/0/22; potential recipients, 60/27/13; and funding recipients, 43/48/10.


Environment Canada                                                                                52
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


recommend $14,643 on average. Potential recipients and academics both call for high
minimum threshold amounts of $43,000 and $ 56,000 respectively.


5.0 CONCLUSIONS
The evaluation of Environment Canada‘s Environmental Damages Fund (EDF) has led
to the following broad conclusions about the program‘s relevance, design and delivery,
success, and cost-effectiveness:

1) The EDF is an important and unique mechanism for the management of court
   awards from the prosecution of environmental infractions. It is relevant to federal and
   departmental priorities for the enforcement and conservation of Canada‘s natural
   capital, and in the future is expected to play an expanded role in the management of
   specified-purpose awards stemming from prosecutions under federal legislation.
2) To date, program delivery has been consistent with the original purpose and design
   of the EDF. In the absence of dedicated funding to support the EDF‘s administration,
   however, delivery and implementation of the program have been significantly
   attenuated in different regions, such that the pace with which funds are being
   diverted to the program through the courts outstrips the program‘s ability to allocate
   funds to EDF projects.
3) Low awareness and promotion of the program as a result of a lack of necessary
   resources has likely impeded the use of the EDF by the courts and consequently the
   diversion of court awards and penalties to the program. As well, the number and
   possibly the quality of proposals received by the program from eligible recipients are
   affected by low awareness and promotion.
4) A number of standard tools and processes exist to ensure consistent delivery of the
   program in all regions, although there is a need to update certain tools to ensure the
   program terms of reference are clear (e.g., the need for partnerships) and to
   streamline certain processes (e.g., more focused application forms). The
   consistency of delivery and implementation of the program across regions has been
   impeded to date as a result of low resourcing as well as a lack of national oversight.
5) The program exhibits a strong tradition of identifying and collecting performance
   measures for funded projects, but no consistent approach to reporting this
   information to senior managers is in evidence for most regions.
6) The roles and responsibilities of different EDF stakeholders are well documented but
   not clearly understood by a sizeable minority of departmental personnel and
   recipients.
7) Environmental Protection Award Measures (EPAMs) are poorly understood. Those
   who report familiarity with EPAMs note that they should be applied only in limited
   circumstances (i.e., for accidental as opposed to intentional offences) and there is
   some question as to their value (e.g., perception of ―paying to pollute,‖ cumbersome
   to administer).
8) To the extent that the EDF has been administered to date, evidence suggests that
   the program has been successful in achieving most intended outcomes and that
   funded projects for the most part are producing their desired deliverables and
   results. While the program demonstrates good progress toward outcomes related to
   financial oversight, interdepartmental cooperation and the development of
   community capacity and skills, little progress is observed in increasing awareness of
   the program among the legal community and potential applicants.


Environment Canada                                                                     53
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund


9) The EDF represents good value for federal dollars spent and is a cost-effective
    means of managing court awards and supporting projects in the pursuit of activities
    related to the program‘s intended outcomes. Projects have successfully leveraged
    funds from partners, and to a lesser extent are reviewed for their cost-effectiveness.
    As well, no other federal programs exist to manage specified purpose funds
    stemming from federal prosecution of environmental offences and overall
    administrative spending is low.
10) The merits of introducing a minimum threshold value for project funding in order to
    increase overall program efficiency are unclear in light of the small size of some
    funded projects and the challenges this would create in meeting court conditions.


6.0 RECOMMENDATIONS
1.   An allocation of dedicated staff and budget to support the ongoing
     management and administration of the EDF program should be considered.
     Uneven regional implementation of the EDF program can be traced, at least in part,
     to the absence of dedicated resources to support its implementation, and this in
     turn may affect the consistency of support to recipients. Further, resource
     constraints are felt to represent a key barrier to the proper promotion of the
     program among potential applicants, thus limiting the pool of potential funding
     recipients able to carry out court requests through the program.

2.   Standardized tools and program documentation for managers and funding
     recipients should be updated to ensure consistent program implementation.
     In updating these tools, consideration should be given to:
       Clarifying partnership requirements for EDF-funded projects;
       The development of standard training and resources for managers that clearly
        outline program governance (i.e., accountability to courts versus Department),
        clarify the roles and responsibilities of program staff and enforcement personnel
        and enhance the consistency of program support to EDF project proponents; and
       The development of standard indicators to guide managers and project
        proponents in measuring project success.

3.   Means of streamlining the program application process and forms should be
     explored to reduce the burden on recipients and enhance consistency with other
     community partnership programs. To this end, the EDF could consider research
     (surveys, focus groups, etc.) with recipients to explore ways in which the application
     form could be improved. Alternatively, a comparison with the application forms for
     similar funding programs (e.g., EcoAction) may yield information to guide such
     changes (e.g., word limits and question simplification).

4.   A communications plan should be developed and implemented to promote
     the EDF among potential recipients, the judiciary and enforcement officers.
     The communications plan should consider the development of standardized
     communications materials that can be modified, as appropriate, to address regional
     circumstances, as well as the development of national communication activities to
     enhance efficiency and consistency of messaging nationally. Efforts to promote the
     program among potential recipients would likely yield higher quality and a greater


Environment Canada                                                                       54
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


     variety of project proposals. As well, communications efforts targeted toward judges
     and, to a lesser extent, Enforcement personnel may increase the number of awards
     and other contributions to the EDF and help to reduce the number of court-
     prescribed funding restrictions.

5.   A centralized national information management system should be developed
     to track financial and performance information for EDF contribution agreements
     across regions. The development and implementation of such a system is essential
     for proper measurement, evaluation and reporting of program performance.

6.   A formal process for annual EDF performance measurement and reporting
     should be implemented. No consistent national process exists for reporting on the
     program‘s impacts and effectiveness. In particular, such a reporting process is
     necessary to monitor progress toward full national implementation and the timely
     disbursement of funding in proportion to program awards. As a first step in
     developing this process, the program logic model should be revised to better reflect
     intended outcomes and more clearly distinguish outcomes from activities and
     standard corporate functions (e.g., financial oversight).


7.0 MANAGEMENT RESPONSE
This section outlines the management response to the evaluation recommendations.
The ES Board accepts the evaluation and its recommendations and the EDF program
has provided a plan to implement the following management actions in response to the
evaluation recommendations within the context the EDF program renewal.

Note to Reader
Bill C-16, the Environmental Enforcement Act, received Royal Assent on June 18th,
2009. Among other things, this Bill amends six EC statutes and three Parks Canada
statutes to direct all fines to the EDF. Once these amendments come into force new
minimum fines will start at $5,000 for individuals and $25,000 for corporations and
maximum fines will increase to $1 million for individuals and $6 million for corporations.
The amendments also allow for a portion of the fines to be used for administering the
EDF Program.

It is anticipated that these amendments could significantly increase the monetary value
and number of awards directed to the EDF and these changes will be factored into the
renewal of the program.

1.   An allocation of dedicated staff and budget to support the ongoing
     management and administration of the EDF program should be considered.
     Uneven regional implementation of the EDF program can be traced, at least in part,
     to the absence of dedicated resources to support program implementation, and this
     in turn may affect the consistency of support to recipients. Further, resource
     constraints are felt to represent a key barrier to the proper promotion of the
     program among potential applicants, thus limiting the pool of potential funding
     recipients able to carry out court requests through the program.

The ES Board agrees with this recommendation.

Environment Canada                                                                     55
Audit and Evaluation Branch                 Evaluation of the Environmental Damages Fund



Under the Environmental Enforcement Act (Bill C-16), which is expected to come into
force in 2011–2012, the program will have the option to use funds directed to the EDF
for administrative purposes.

The business case to support program renewal will be requesting administrative
resources, including dedicated staff and operational and management costs. The
request for program renewal will be presented in September 2009.

Securing resources for the EDF will ensure that each region has dedicated staff to
deliver the program. This will in turn help to bring national consistency in the delivery of
the program and ensure that proper support is available to clients. These resources will
enable staff to effectively promote the EDF program among targeted audiences and
diversify the pool of potential funding recipients.

    Timeline                       Deliverable                        Responsible Party
 September 2009      Present the EDF business case and                Director, Outreach
                     documentation related to program
                     renewal.


2.   Standardized tools and program documentation for managers and funding
     recipients should be updated to ensure consistent program implementation.
     In updating these tools, consideration should be given to:
       Clarifying partnership requirements for EDF-funded projects; program
          documentation is not clear about whether partnerships are a requirement for
          EDF-funded projects and assessments of the value of partnerships are mixed;
       The development of standard training and resources for managers that clearly
        outline program governance (i.e., accountability to courts versus Department),
        clarify the roles and responsibilities of program staff and enforcement personnel
        and enhance the consistency of program support to EDF project proponents; and
       The development of standard indicators to guide managers and project
        proponents measuring project success.

The ES Board agrees with this recommendation.

As part of the program renewal package, the program is recommending the approval of
the EDF Management Framework and the Funding Agreement Terms and Conditions
for the Environmental Damages Fund Program. The management framework provides
an overview of how the program works, the program logic model, performance
measurement, and reporting and risk management strategies. The funding agreement
terms and conditions set forth the terms and conditions under which the EDF program
will provide funding for projects.

The program renewal package outlines a number of commitments to streamline
administrative processes to ensure consistency across all regions:

i)   Development of operational guidelines to outline program governance, roles and
     responsibilities of program staff and clarify partnership requirements for the EDF
     program, as partnerships are considered an advantage but not essential..


Environment Canada                                                                         56
Audit and Evaluation Branch                  Evaluation of the Environmental Damages Fund


ii)    Development of regional management plans that define top priorities for potential
       EDF projects and enable the program to direct funding to local environmental
       priorities that represent the best use of funds in response to environmental damage
       or damage-to-wildlife incidents. Regional management plans will be prepared in
       consultation with EC regional management teams and include other government
       departments as required. The plans will be approved by the regional director
       general.
iii)   Development of standard project indicators to guide managers and project
       proponents in measuring project success. These indicators will contribute to the
       evaluation of program results.

    Timeline                        Deliverable                     Responsible Party
 September 2009        Present the EDF business case and        Director, Outreach
                       documentation related to program
                       renewal
 November 2009         Develop program operational              Director, Outreach
                       guidelines
 Fall 2009             Develop regional management plans        Regional Directors General
 Fall 2009             Conduct project indicators workshop      Director, Outreach
                       to develop standard project indicators


3.     Means of streamlining the program application process and forms should be
       explored to reduce the burden on recipients and enhance consistency with other
       Community Partnership programs. To this end, the EDF could consider research
       (surveys, focus groups, etc.) with recipients to explore ways in which the application
       form could be improved. Alternatively, a comparison with the application forms for
       similar funding programs (e.g., EcoAction) may yield information to guide such
       changes (e.g., word limits and question simplification).

The ES Board agrees with this recommendation.

The EDF program will take a more directed approach to soliciting applications for
projects. As outlined in the response to the second recommendation, the program will
develop regional management plans that define top priorities for potential EDF projects
and enable the program to direct funding to local environmental priorities that represent
the best use of funds in response to environmental damage or damage to wildlife.
Project proposals will be solicited on an on-going basis as EDF awards are received.

As a part of the EDF program renewal, the program has committed to streamlining EDF
administrative processes, including reducing the burden of application and reporting
processes on funding applicants.

The EDF program will participate in the Departmental Action Plan to Reform the
Administration of Grants and Contributions, which has the goal of simplifying and
streamlining application processes and improving forms for grants and contributions
(G&Cs), while strengthening accountability. It is important to note that the scope and
timelines of the commitments in relation to the Departmental Action Plan to Reform the
Administration of Grants and Contributions are beyond the program‘s control.



Environment Canada                                                                         57
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


     Timeline                        Deliverable                          Responsible
                                                                              Party
     Fall 2009          Develop Regional Management Plans              Regional Directors
                                                                            General
 Fall/winter      Review application and reporting processes           Director, Outreach
 2009–2010
 2011–2012        Adopt applications and reporting processes           Director, Outreach
                  resulting from G&C reform

4.   A communications plan should be developed and implemented to promote
     the EDF among potential recipients, the judiciary and enforcement officers.
     The communications plan should consider the development of standardized
     communications materials that can be modified, as appropriate, to address regional
     circumstances, as well as the development of national communication activities to
     enhance efficiency and consistency of messaging nationally. Efforts to promote the
     program among potential recipients would likely yield higher quality and a greater
     variety of project proposals. As well, communications efforts targeted toward judges
     and, to a lesser extent, Enforcement personnel may increase the number of awards
     and other contributions to the EDF and help to reduce the number of court-
     prescribed funding restrictions.

The ES Board agrees with this recommendation.

As part of the program renewal strategy, the program has committed to developing a
promotional strategy that targets judges and prosecutors, enforcement officials from
other federal government departments and potential funding recipients. In addition, a
national communications plan will be created to ensure national consistency in efforts to
promote the EDF program.

Overview of the EDF Promotional Strategy

Judges and Prosecutors
    The program will work with the Department‘s Legislative Affairs Branch to
     disseminate information to the judicial community, increase judges‘ awareness of
     the EDF and relay how it can be used as an effective and innovative mechanism
     for sentencing. The Legislative Affairs Branch has a contribution agreement with
     the National Judicial Institute, an arms-length third-party organization dedicated to
     the education of judges, to develop materials related to environmental law aimed
     at raising judges‘ awareness of these issues.
    Develop a Web page geared towards the legal community to help promote the
     EDF among Crown prosecutors and defence lawyers.
    Work with the National Environmental Prosecutions Coordinator at the Department
     of Justice to disseminate information on the EDF to Crown prosecutors.

Enforcement
    Work with EC‘s Enforcement Program both at the national and regional levels to
     maintain good working relationships and to determine what information, training
     and tools would best serve their needs.




Environment Canada                                                                      58
Audit and Evaluation Branch                 Evaluation of the Environmental Damages Fund


      Update the EDF fact sheet and brochure to capture current program information
       and ensure its effectiveness as a tool for enforcement officers to promote the
       program among prosecutors.
      Develop an outreach plan for other federal departments whose enforcement
       officers and legislation can be used to direct funds to the EDF. Tools could include
       fact sheets, brochures, presentations and training modules.

Potential Recipients
    Undertake targeted regional promotion among key groups that could potentially
     undertake EDF projects.
    Implement improvements to the EDF Web site
    Participate in the Community Action Programs for the Environment (CAPE) Web
     portal, which offers a single entry point to all EC community funding programs.

      Timeline                         Deliverable                     Responsible Party

 Fall 2009           Implement national EDF communications plan        Director, Outreach
 Summer 2009         Develop EDF promotional strategy                  Director, Outreach
 July 2009           Participate in the CAPE Web portal                Director, Outreach
 August 2009         Develop and distribute EDF fact sheet             Director, Outreach
 Fall 2009           Disseminate EDF letter to prosecutors by the      Director, Outreach
                     Department of Justice National Environmental
                     Prosecutions Coordinator
 Fall 2009           Complete Web site improvements                    Director, Outreach
 Winter/spring       Develop enforcement training materials and        Director, Outreach
 2010                tools
 Spring/summer       Develop information in conjunction with the       Director, Outreach
 2010                Judicial Outreach Program for the National
                     Judicial Institute
 Fall/winter         Develop EDF brochure                              Director, Outreach
 2009–2010


5.   A centralized national information management system should be developed
     to track financial and performance information for EDF contribution agreements
     across the regions. The development and implementation of such a system is
     essential for proper measurement, evaluation and reporting of program
     performance.

The ES Board agrees with this recommendation.

As a part of the departmental G&C reform initiative, an online application and
information management system is being developed for its G&C programs. This system
will improve program efficiency, enhance alignment with departmental priorities and
improve the ability to report collectively on the results of departmental funding programs.
This system is scheduled to be in operation by 2011–2012, after which the EDF
program will adopt it to collect program data.

In the meantime, the program acknowledges that a consistent and effective mechanism
for collecting and tracking program information and results is required. In April 2009, the

Environment Canada                                                                       59
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


program implemented a Management Information System (MIS), cloned from the
existing EcoAction Community Funding Program MIS. The MIS allows program staff to
track project information, including recipient information, funding agreements, project
funding, and project results. Planned modifications and improvements will allow the
program to track the source of monetary contributions, the legislation used and whether
EC enforcement officials were involved. The system will facilitate reporting and allow
project managers to track communications with funding recipients. The use of this tool
will ensure consistency of data collection across the regions and significantly improve
the program‘s ability to track and report results. This system has been launched in all
EC regions and is currently being populated with historical as well as current project
information. This system will be used until the new G&C online system is up and
running.

      Timeline                      Deliverable                   Responsible Party

 Fall 2009             Complete implementation of MIS           Director, Outreach
 2011–2012             Adopt the departmental G&C               Director, Outreach
                       information management system


6.   A formal process for annual EDF performance measurement and reporting
     should be implemented. No consistent national process exists for reporting on the
     program‘s impacts and effectiveness. In particular, such a reporting process is
     necessary to monitor program progress toward full national implementation and the
     timely disbursement of program funding in proportion to program awards. As a first
     step in developing this process, the program logic model should be revised to better
     reflect intended program outcomes and more clearly distinguish outcomes from
     program activities and standard corporate functions (e.g., financial oversight).

The ES Board agrees with this recommendation.

The program recognizes that a formal reporting process is required to reflect the
success and progress of the EDF program. In response to this need, a revised program
logic model, performance measurement framework and reporting strategy have been
developed and included in the Management Framework as part of the program renewal.
The program has also committed to developing project-specific indicators to assist in the
effective measurement of project results.

Program results will be collected by NCR program staff on an annual basis and
presented to the ES Board. The information will capture the number of projects that
were funded and the results achieved, including environmental impacts. The program
will also demonstrate its administrative effectiveness with respect to national
implementation of the program as well as the timely allocation of funding.

A departmental performance measurement framework has also been developed and the
EDF has identified three expected results and three corresponding indicators against
which it will report on an annual basis in the Departmental Performance Report.

The expected results are:



Environment Canada                                                                    60
Audit and Evaluation Branch              Evaluation of the Environmental Damages Fund


            Recognition by judges of the value of the Environmental Damages Fund in
             helping communities recover from environmental damage and damage-to-
             wildlife incidents.
            Engagement of communities directly affected by environmental damage and
             damage-to-wildlife incidents.
            Natural environments affected by environmental damage are restored.

The indictors that will be used to measure these results are:
        Annual number of court awards directing payment to the Environmental
           Damages Fund
        Number of projects completed
        Proportion of area restored.

As previously mentioned the program has developed an electronic Management
Information System that will be used to track program results and help facilitate
reporting and ensure consistency in reporting practices across all regions. The system
will be used to roll up annual results and for program evaluation purposes.

    Timeline                         Deliverable                  Responsible Party

 September 2009      Present the EDF business case and            Director, Outreach
                     documentation related to program renewal
 Fall 2009           Conduct project indicators workshop to       Director, Outreach
                     develop standard project indicators
 Fall 2009           Complete implementation of MIS               Director, Outreach




Environment Canada                                                                 61
Audit and Evaluation Branch     Evaluation of the Environmental Damages Fund




                              ANNEXES




Environment Canada                                                       63
Audit and Evaluation Branch                                                                                                    Evaluation of the Environmental Damages Fund



                                                            Annex 1: EDF Program Logic Model
                              Communications                                                                Proposal Planning and
                                                                          Partnerships                                                                 Project Delivery
                            Program Promotion                                                                     Development
 Activities    Preparation of EDF project communication      Coordination of departmental/             Provision of knowledge,              Administration of project approval
               plans                                         interdepartmental technical review        expertise, informational tools and   process
               Preparation of EDF informational and          teams                                     advice                               Financial tracking and management of
               promotional products                                                                                                         fund contributions
               Development and maintenance of EDF                                                                                           Preparation of contribution
               Web site                                                                                                                     agreements
               Preparation of media alerts for project                                                                                      Project auditing and in-field monitoring
               approvals
 Reach         Recipient community: environmental            EC branches and divisions (EPB,           Recipient community                  Recipient community
               community, Aboriginal groups, universities    CWS, MSC, CDR)
               and provincial, territorial and municipal     Other federal departments (Habitat
               levels of government                          Management DFO, Marine Safety TC,
                Legal community: Judges, crown               PWGSC)
               prosecutors, DOJ, EC Enforcement and
               other federal departments
               General public
 Outputs       EDF communications and promotional            Review and evaluation of project          Information and advice to            Administrative and technical reviews
               products (brochures, fact sheets, EDF Web     proposals.                                recipient community related to       Project approvals
               site, public presentations, workshops);       Provision of interdepartmental agency     fund specifications, proposal        Contribution agreements signed
               media briefing material for project           expertise and knowledge.                  development strategy, and federal    Restoration projects funded
               approvals                                     Damage compensation increased             partnerships and financial
                                                             through utilization of environmental      planning
                                                             legislation (CEPA 1999, FA, MBCA
                                                             1994, CSA).
 Outcome 1     Increased awareness of EDF by the legal       Scientific and technically sound,         Scientific and technically sound,    Financial accountability
               community and use as sentencing option        feasible and cost-effective restoration   feasible and cost-effective          Transparency of funding process
               by the courts resulting in monetary awards    projects                                  restoration projects

 Outcome 2     Awareness of the Fund among recipient         Increased court-awarded contributions     Community groups acquire             EDF restoration objectives are
               organizations                                 to the EDF                                knowledge and skills for proposal    achieved
                                                                                                       development and restoration
                                                                                                       planning
 Outcome 3     Public information on EDF-funded projects     Interdepartmental cooperation and
               and on their geographic location              information transfer

 Long-term     Biodiversity is conserved and protected
 Outcome       Sustainable use and management of natural capital is ensured
               Risk to the environment posed by toxic and harmful substances is reduced


 Key Results   Canada‘s natural capital is restored, conserved and enhanced
               The environment is protected from the effects of pollution and waste
   Source: Environment Canada. 2008. EDF Evaluation Plan - Draft 6. Ottawa.


Environment Canada                                                                                                                                                                     64
Audit and Evaluation Branch                      Evaluation of the Environmental Damages Fund



         Annex 2: Summary of Terms and Conditions for EDF
                     Contribution Agreements



1. Recipients must be eligible to receive the monies
2. The contribution is consistent with application and decision

3. The project meets the court‘s conditions
4. The project is cost-effective
5. The recipient possesses the skills necessary to deliver project
6. Projects undertaken build on partnerships
7. Proposals are evaluated by program managers and signed off by the RDG
8. The maximum amount payable is under $5 million
9. Awards cover allowable expenditures (not basic costs)
10. Payment methods are on invoice and with statement of expenditures
11. Equipment purchased belongs to the project
12. There exists a right to audit each project
13. There exists a payment authority




Environment Canada                                                                        65
Audit and Evaluation Branch                           Evaluation of the Environmental Damages Fund



                Annex 3: Evaluation Issues and Data Sources
     Issue                  Evaluation Question                      Documents          Interviews     Observations
                1.   Is there a legitimate and necessary role
                     for government in this program area or                                No        Not contested;
                     activity?                                   2002 Evaluation;                    already well
                                                                 2008 RMAF/RBAF;                     documented,
                2.   Is the program connected with
Relevance                                                                                  No        however, it will
                     societal/environmental needs?               2007 Process Audit;                 be re-stated from
                3.   Does the program theory (activities,        Administrative Files                existing
                     instruments, objectives, design) address                              No        documentation
                     the societal need identified?
                4.   Does the program identify clear
                                                                                           Yes
                     deliverables and expected results?
                5.   Is the program delivered as designed?       RMAF/RBAF                 Yes
                6.   Is performance data collected against       EDF data base and
                     program activities/outcomes? If so, is      financials                Yes
Design and           collected information used to inform
                     senior management/decision-makers?          Minutes of mgmt
Delivery                                                         meetings using PM
                                                                 data
                7.   Who is accountable for the program?         EDF WG summary
                     Are the roles and responsibilities of all   reports since 2006
                     groups involved clear? Is there an                                    Yes
                     appropriate accountability framework
                     (e.g., for multi stakeholder agreements)?
                                                                 Process Audit 2007
                                                                 RMAF/RBAF
                8.   What are the best practices and lessons
                                                                 Program review            Yes
                     learned from this program?
                                                                 workshop summary
                                                                 2002
                                                                 Administrative file
                9.   To what extent have intended outcomes       reviews Technical
                     been achieved as a result of the            survey reviews            Yes
                     program?
                                                                 EDF WG reports
                                                                 Technical survey
                                                                 reviews
Success
                                                                 EDF WG summary
                10. Have there been any unintended               reports since 2006        Yes
                    (positive or negative) outcomes? Were
                    any actions taken as a result?               Program review
                                                                 workshop
                                                                 summaries
                                                                 Recipient surveys
                11. Are there alternative ways of achieving
                                                                                           Yes
                    the objectives of the program?
                12. Are others involved in the same areas of
                    activities and/or share similar
                                                                                           Yes
                    objectives? How is duplication avoided
                    and complementarity achieved?
Cost-
effectiveness                                                    EDF data base and
                13. If the program or activity continues, how    financials                          Financial
                                                                                           Yes
                    could its efficiency be improved?                                                analysis

                                                                 EDF data base and
                14. Has the program provided value for           financials                          Financial
                                                                                           Yes
                    federal dollars spent?                                                           analysis




Environment Canada                                                                                               66
Audit and Evaluation Branch             Evaluation of the Environmental Damages Fund



                              Annex 4: Bibliography

Department of Finance. 2007. The Budget Plan 2007: Aspire to a Stronger, Safer, Better
Canada. Accessed at: http://www.budget.gc.ca/2007/pdf/bp2007e.pdf

Department of Finance. 2008. The Budget Plan 2008, Responsible Leadership.
Accessed at: http://www.budget.gc.ca/2008/pdf/plan-eng.pdf

Environment Canada. 2000. Introducing the Environmental Damages Fund. Ottawa.

Environment Canada. 2002. Environmental Damages Fund Recommendations from
EDF National Workshop. Ottawa.

Environment Canada. 2004. Results-based Management and Accountability Framework
for the EcoAction Community Funding Program. Ottawa

Environment Canada. 2005. Environmental Damages Fund Results-based Management
and Accountability Framework. Ottawa.

Environment Canada. 2006. Environmental Damages Fund Framework. Ottawa.

Environment Canada. 2006. Environmental Damages Fund Grants and Contributions
Approval Form. Ottawa.

Environment Canada. 2007. Communications Plan for the National Promotion of the
Environmental Damages Fund (EDF) Program 2007–2009. Ottawa.

Environment Canada. 2007. Environmental Damages Fund Recipient’s Handbook.
Ottawa.

Environment Canada. 2008. Environmental Damages Fund: A Program Overview.
Ottawa.

Environment Canada. 2008. Environmental Damages Fund Results-based Management
and Accountability Framework. Ottawa.

Environment Canada. 2009. 2008 Request for Update on Management Response to
Address Evaluation Recommendations. Ottawa.

Environment Canada. 2009. Environmental Damages Fund Administrative Data.
Ottawa.

Environment Canada. 2009. Environmental Damages Fund Application Form. Accessed
at: http://www.ec.gc.ca/edf-fde/3901F8D5-B32D-4929-9FE5-
AA1BE9180A11/EDF_Applicants_Guide.doc

Environment Canada. 2009. Environmental Damages Fund Final Narrative Evaluation
Report. Access at: http://www.ec.gc.ca/edf-fde/BC9910A4-5735-43EF-AFA3-
59D0770857CE/EDF_Final_Narrative_Report.doc


Environment Canada                                                                 67
Audit and Evaluation Branch              Evaluation of the Environmental Damages Fund



Environment Canada. 2009. Environmental Damages Fund In-Kind Contribution Report.
Access at: http://www.ec.gc.ca/edf-fde/BC9910A4-5735-43EF-AFA3-
59D0770857CE/EDF_In-Kind_Contribution_Record.doc

Environment Canada. 2009. Environmental Damages Fund Quarterly Narrative
Evaluation Report. Access at: http://www.ec.gc.ca/edf-fde/BC9910A4-5735-43EF-AFA3-
59D0770857CE/EDF_Quarterly_Narrative_Report.doc

Environment Canada. 2009. Environmental Damages Fund Sample Financial Audit.
Access at: http://www.ec.gc.ca/edf-fde/BC9910A4-5735-43EF-AFA3-
59D0770857CE/EDF_Example_of_a_Financial_Audit.doc

Environment Canada. 2009. Number of Spill/Releases, by Fiscal year, by Region,
National Emergencies and Enforcement Management Information System and
Intelligence System. Administrative data.

Environment Canada. Undated. Environmental Damages Fund Applicant’s Guide.
Access at: http://www.ec.gc.ca/edf-fde/3901F8D5-B32D-4929-9FE5-
AA1BE9180A11/EDF_Applicants_Guide.doc

Environment Canada. Undated. Environmental Damages Fund Framework for
Environmental Damages Assessment and Restoration. Ottawa.

Environmental Law Centre. 2003. News Brief, Creative Sentencing Part I – Overview.
Volume 18, No. 2. Accessed at:
http://www.elc.ab.ca/Content_Files/Files/NewsBriefs/CreativeSentencing-V18-2.pdf

Government of Canada. 2009. Office of the Administrator of the Ship-source Oil
Pollution Fund, Frequently Asked Questions. Accessed at:
(http://www.ssopfund.gc.ca/english/faq.asp

HDP Group Inc. 2002. Environmental Damages Fund Evaluation. Ottawa.

Morrison, S. 2007. Environment Canada Courts Awards (Pilot) Project: Progress Report.

PWGSC – Public Works and Government Services Canada. 2007. Environmental
Damages Fund Process Analysis. Ottawa.

Treasury Board of Canada Secretariat. 1995. Environmental Damages Fund Guidelines,
Appendix A. Ottawa.

Treasury Board of Canada Secretariat. 1995. Policy on Specified Purpose Accounts.
Accessed at: http://www.tbs-sct.gc.ca/pol/doc-
eng.aspx?evttoo=X&id=12249&section=text

Wruck, J. 2004. The Federal Environmental Damages Fund. The Advocate. Vol. 62-2.




Environment Canada                                                                   68
Audit and Evaluation Branch                      Evaluation of the Environmental Damages Fund




       Annex 5: Administrative (Project) File Assessment Guide
   Project Attributes and
                                                                Scale
    Evaluation Criteria
Project Characteristics
Agreement number                As shown
Project title                   As shown
Region                          As shown
                                1 = single year
Whether the project is single
                                2 = multi-year
or multi-year one
                                9 = insufficient information to judge
Effective date (start date)
                                (dd/mm/yy)
(dd/mm/yy)
End date (dd/mm/yy)             (dd/mm/yy)
Amount of EC agreement
                                In thousands of dollars
funding
Total project cost (final
                                In thousands of dollars
financial reporting)
Evaluation Issue
                                1 = no mention of deliverables/outputs
Does the project identify       2 = description of deliverables/outputs or their relations to
clear deliverables and          promised results/impacts/outcomes are not entirely clear
expected results?               3 = clear description of deliverables/outputs and their relationship
                                to the promised results/impacts/outcomes
                                1 = no evidence of measuring or gauging activities/deliverables
                                2 = partial evidence of implementation of project activities and the
                                attainment of expected deliverables
                                3 = clear evidence of implementation of project activities and the
                                attainment of expected deliverables
Is performance data
collected against program
activities/outcomes?            1 = no evidence of measuring or gauging
                                results/impacts/outcomes
                                2 = partial evidence that results/impacts/outcomes were
                                measured or gauged
                                3 = clear evidence that results/impacts/outcomes were measured
                                or gauged
                                1 = no mention of partners
                                2 = poor – stakeholders listed, with minimal detail on roles and
Is there appropriate            responsibilities
accountability framework
(e.g. for multistakeholder      3 = moderate – stakeholders listed, with moderate detail on roles
agreements)?                    and responsibilities
                                4 = very clear – stakeholders‘ roles and responsibilities fully
                                explained



Environment Canada                                                                                 69
Audit and Evaluation Branch                      Evaluation of the Environmental Damages Fund


   Project Attributes and
                                                                  Scale
    Evaluation Criteria
                                 Record number
                                 999 = insufficient information to judge
                                 1 = municipality
                                 2 = community organization (excl. environmental organization)
                                 3 = environmental organization
                                 4 = post-secondary educational institution
Is the project delivered in      5 = business
partnership with any other       8 = other
individuals, groups or           9 = insufficient information
organizations?
                                 TREAT AS MULTIPLE RESPONSE, i.e. SEVERAL ANSWERS
                                 ARE TO BE ALLOWED
                                 1 = no statement of their obligations
                                 2 = confused description of stakeholders‘ obligations
                                 3 = research
                                 4 = assistance in service delivery
                                 5 = funding
                                 1 = poor – Outcomes are difficult to discern, minimal detail
Have intended project            2 = moderate – Outcomes described/mentioned, moderate detail
outcomes been achieved?          3 = very clear – Outcomes successes described in detail
                                 9 = no outcomes listed
                                 1 = mention of negative unintended outcomes
Have there been any              2 = mention of unintended outcomes that are neutral or un-
unintended (positive or          assessed
negative) outcomes?              3 = mention of positive unintended outcomes
                                 9 = no mention of unintended outcomes
     EDF Contribution
       Guidelines
                                 1 = clear demonstration of recipients‘ ineligibility relative to
                                 program eligibility criteria
Recipients must be eligible to
                                 2 = clear demonstration of recipients‘ eligibility relative to program
receive the monies
                                 eligibility criteria
                                 9 = insufficient information provided
                                 1 = little or no consistency with application and court decision
                                 2 = consistent with some but not most aspects of application and
                                 court decision
The contribution is consistent   3 = consistent with most aspects of application and court decision
with application and decision
                                 4 = totally consistent with court decision and intended use or
                                 application of funds
                                 9 = insufficient or no information provided
The project meets the court‘s    1 = the project does not meet the court‘s conditions
conditions                       2 = the project meets some conditions



Environment Canada                                                                                  70
Audit and Evaluation Branch                      Evaluation of the Environmental Damages Fund


   Project Attributes and
                                                                 Scale
    Evaluation Criteria
                                  3 = the project meets most conditions
                                  4 = the project meets all conditions
                                  9 = insufficient or no information provided on court conditions
                                  1 = program officers or auditors assessed cost-effectiveness and
                                  reached a NEGATIVE conclusion
                                  2 = no direct reference to cost-effectiveness but files contain
                                  enough information to reach a general conclusion that project
                                  plausibly NOT cost-effective
                                  3 = no direct reference to cost-effectiveness but files contain
The project is cost-effective     enough information to reach general conclusion that project was
                                  plausibly cost-effective
                                  4 = program officers or auditors assessed cost-effectiveness and
                                  reached positive conclusion
                                  9 = setting aside the issue that administrative and staff costs are
                                  not accounted for across all projects, this particular project
                                  provides too little information to determine its cost-effectiveness.
                                  1 = file contains little evidence that the recipient has adequate
                                  formal skills and adequate experience for project delivery
Recipient possesses the           2 = recipient provides moderate evidence of formal skills or
skills necessary to deliver the   experience relevant to project delivery
project                           3 = recipient demonstrates formal skills and experience relevant
                                  to project delivery
                                  9 = insufficient information to reach a conclusion
                                  1 = no evidence of proposal evaluation
                                  2 = program managers appear to have evaluated the proposal but
                                  evidence is informal
Proposals are evaluated by
program managers and              3 = program managers have formally evaluated the proposal
signed off by RDG                 1 = no apparent RDG sign-off
                                  2 = an apparent RDG sign-off


                                  1 = amount exceeded $5 million
The maximum amount
                                  2 = amount was under $5 million
payable is under $5 million
                                  9 = amount unclear
Awards cover allowable
expenditures (not basic
costs). The prohibited basic      1 = amount covered basic costs as well
costs are those costs that are    2 = amount covered allowable expenditures only
necessary for maintaining the     9 = insufficient information to judge
operation of the recipient
organization.
                                  1 = most or all payments are lacking an invoice or statement of
Payment methods are on            expenditures
invoice and with statement of     2 = most payment methods are on invoice and with statement of
expenditures                      expenditures
                                  3 = all payment methods are on invoice and with statement of


Environment Canada                                                                                    71
Audit and Evaluation Branch                    Evaluation of the Environmental Damages Fund


   Project Attributes and
                                                                 Scale
    Evaluation Criteria
                                expenditures
                                9 = insufficient information
                                1 = does not belong to project
Equipment purchased
                                2 = belongs to project
belongs to the project
                                9 = insufficient information
There exists a right to audit   1 = no statement of audit right in contract
each project                    2 = audit right stated in contract
There exists a payment          1 = a payment authority is not indicated
authority                       2 = a payment authority is indicated




Environment Canada                                                                      72
   Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund



                             Annex 6: Interview Guides
                                 Federal Respondent’s Guide
Hello, my name is _____ _____ and I‘m phoning from COMPAS Research. We‘re currently
conducting interviews with Environment Canada personnel as part of an evaluation of the
Environmental Damages Fund. Is there a day and time when you might be available to take
part in an interview?

   A. IF THE KEY INFORMANT WANTS THE INTERVIEW NOW: Thank you for agreeing
       to the interview. We generally audio record our interviews to ensure our discussion
       is documented accurately. Would this be ok? In either instance, the interview would
       be confidential and no comment would be linked to you. UNPROMPTED

        Yes, agrees to taping
        Agrees to interview but no taping
        Reconsiders and does not agree to interview.

       IF THE INTERVIEW IS SCHEDULED FOR ANOTHER TIME, (a) WE WILL
          TRANSMIT A COPY OF THE QUESTIONNAIRE AHEAD OF TIME TO THE
          PROSPECTIVE KEY INFORMANT AND (b) THE RESUMPTION OF THE
          CONVERSATION WOULD BEGIN HERE: Hello, it‘s _____ _____ phoning from
          COMPAS Research for the interview with respect to an evaluation of the
          Environmental Damages Fund. Thank you for agreeing to the interview. We
          generally audio record our interviews to ensure our discussion is documented
          accurately. Would this be ok? In either instance, the interview would be
          confidential and no comment would be linked to you.
       UNPROMPTED

        Yes, agrees to taping
        Agrees to interview but no taping
        Reconsiders and does not agree to interview.

   Background

   1. Please briefly describe your responsibilities within the Environmental Damages Fund
      and how long you have been involved with it.

   Design and Delivery

   2. Using a seven-point scale where 1 means ―not at all‖ and 7 means ―to a great
      extent,‖ to what extent do you feel the EDF program is delivered as designed?

   3. In what ways is the EDF‘s delivery consistent or inconsistent with the program‘s
      original design? PROBE: Please explain and provide examples.


   4. Overall, how well is the program being delivered?

       PROMPT AS NECESSARY: And how about:


   Environment Canada                                                                    73
Audit and Evaluation Branch                 Evaluation of the Environmental Damages Fund


     the application process?
     external communications around the program?
     program support to applicants?

5. Using a seven-point scale where 1 means ―not at all clear‖ and 7 means ―very clear,‖
   to what extent do you feel the EDF program‘s deliverables and expected results are
   clear?

     What are the reasons that led you to this score?
     Generally speaking, are performance indicators developed and clearly identified
      for projects funded through the EDF program?

6. For EDF to attract worthwhile applications, is there a minimum size contribution that
   it should consider? IF YES: What would that minimum amount be and why?

7. What means are used to promote the EDF to eligible recipient groups?

     Have these efforts been effective in increasing awareness of EDF contribution
      opportunities among Aboriginal groups, NGOs, provinces, municipalities, and
      universities?

8. As you know, the program encourages recipients to work with partners. In your view,
   what is the effect of these partnerships on EDF-funded projects?

     Are these partnership arrangements working well?

9. Is performance data being collected and reported against EDF activities/outputs and
   outcomes? Please provide examples.

     Is the available performance information accurate and reliable? Timely? Are
      there any gaps?
     Is collected information used to inform senior management/decision-makers? If
      so, how is this done?

10. Are the roles and responsibilities of the various parties involved in the delivery of the
   EDF program clear? IF NOT, PROBE: Which aspects of the EDF are not well
   understood? Please explain.

11. As you likely know, EPAMs are negotiated settlements as an alternative to court
   prosecution for a violation of the Canadian Environmental Protection Act. Are efforts
   being made to encourage EPAM awards to be directed to the EDF?

     Is the use of EPAMs likely to grow or diminish in the future? If so, why?

12. Although CEPA is the federal statute under which financial assignments to the EDF
   are generally made, it is also possible for the EDF to be used in relation to infractions
   under the Fisheries Act, the Canada Shipping Act, the Migratory Birds Convention
   Act, and the Species at Risk Act. In your view, are any efforts being made to
   encourage the courts to use these other laws to direct payments to the EDF? Please
   explain.


Environment Canada                                                                         74
Audit and Evaluation Branch              Evaluation of the Environmental Damages Fund



13. What have been the strengths and best practices in the design and delivery of the
   program? What has worked well?
     What have been the challenges or limitations? What has not worked well?

Success

14. On a seven-point scale where 1 means ―largely unachieved‖ and 7 means ―largely
   achieved,‖ to what extent do you feel that the intended outcomes of funded projects
   have been achieved?

     What are the reasons that led you to this score?

15. And to what extent has the EDF program overall contributed to:

      environmental restoration? PROMPT FOR DETAILS (e.g. ―Please describe or
       explain.‖)
      environmental quality improvement
      to what extent has it contributed to research and development? PROMPT FOR
       DETAILS (e.g., ―Please describe or explain.‖)
      How much has it contributed to education and awareness? PROMPT FOR
       DETAILS (e.g., ―Please describe or explain.‖)

16. What factors have had the greatest impact on the achievement of program
   outcomes? PROBE FOR EXAMPLES AND EXPLANATIONS

17. Has the EDF had any unintended or unexpected outcomes, either positive or
   negative? If yes, please describe.

     Were any actions taken to address these unintended outcomes? If yes, please
      specify.

Cost-effectiveness and Alternatives

18. In your view, is the EDF program‘s current approach the most efficient and cost-
    effective way of accomplishing its objectives? Please explain.

     If not, what improvements could be made to the design and delivery of the
      program to make it more efficient and cost-effective?

19. Are you aware of any alternative programs or approaches in other federal
   departments/agencies or in other jurisdictions within or outside Canada that are
   comparable to the EDF?
    IF YES:
     Are any efforts made to avoid duplication and facilitate the complementarity of
       the programs? Please describe.

     Are there any gaps in the activities and reach of the programs? IF YES: Is there
      anything EC‘s EDF could do to help address these gaps?



Environment Canada                                                                      75
Audit and Evaluation Branch              Evaluation of the Environmental Damages Fund


20.    Considering the program‘s overall relevance, impacts and cost-effectiveness, to
   what degree do you feel the EDF provides value for dollars spent? Is the EDF able to
   achieve its outcomes at the lowest possible cost? Please explain.

21.    As we near the conclusion, are there any other thoughts that you would like to
   share with respect to evaluating the Environmental Damages Fund?

THANK AND CONCLUDE




Environment Canada                                                                      76
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


                              Funding Recipient’s Guide

Hello, my name is _____ _____ and I‘m phoning from COMPAS Research. As part of
an evaluation of the Government of Canada‘s Environmental Damages Fund, we‘re
conducting interviews with individuals whose organizations have received support from
the Environmental Damages Fund. Is there a day and time when you might be available
to take part in an interview?

IF NECESSARY SAY: ―Recently, an e-mail was sent out from Shelley Borys at
Environment Canada advising you that COMPAS would be calling you to take part in an
evaluation of the EDF program.‖

A. IF THE KEY INFORMANT WANTS THE INTERVIEW NOW: Thank you for agreeing
   to the interview. We generally audio record our interviews to ensure our discussion
   is documented accurately. Would this be ok? In either instance, the interview would
   be confidential and no comment would be linked to you. UNPROMPTED

      Yes, agrees to taping
      Agrees to interview but no taping
      Reconsiders and does not agree to interview.

B. IF THE INTERVIEW IS SCHEDULED FOR ANOTHER TIME, (a) WE WILL
   TRANSMIT A COPY OF THE QUESTIONNAIRE AHEAD OF TIME TO THE
   PROSPECTIVE KEY INFORMANT AND (b) THE RESUMPTION OF THE
   CONVERSATION WOULD BEGIN HERE: Hello, it‘s _____ _____ phoning from
   COMPAS Research for the interview with respect to an evaluation of the
   Environmental Damages Fund. Thank you for agreeing to the interview. We
   generally audio record our interviews to ensure our discussion is documented
   accurately. Would this be ok? In either instance, the interview would be confidential
   and no comment would be linked to you. UNPROMPTED

      Yes, agrees to taping
      Agrees to interview but no taping
      Reconsiders and does not agree to interview

Background

   1. Please briefly describe your role in the environmental project for which [NAME
      OF ORGANIZATION] received support from the Environmental Damages Fund.

   2. Was your EDF-supported project mainly concerned with

          environmental restoration
          environmental quality improvement
          research and development or
          education and awareness?


Design and Delivery


Environment Canada                                                                     77
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund



   3. Thinking of the information you received about the EDF program, what made the
      EDF an attractive source of funding for your organization?

   4. Was there anything about the EDF that made it less attractive as a source of
      funding?

   5. Based on what you know about how the EDF is meant to operate, to what extent
      do you feel the EDF program is delivered as designed?

   6. In what ways is the EDF‘s delivery consistent or inconsistent with the program‘s
      original design? PROBE: Please explain and provide examples.

   7. Overall, how well is the program being delivered?
      PROMPT AS NECESSARY: And how about:

          the application process?
          communications around the program?
          program support to applicants?

   8. Are the roles and responsibilities of the various parties involved in the delivery of
      the EDF program clear? IF NOT, PROBE: Which aspects of the EDF are not well
      understood? Please explain.

   9. As you know, the program encourages recipients to work with partners. In your
      experience, what is the effect of these partnerships on EDF-funded projects?

          Did your partnership arrangement work well?

   10. For EDF to attract worthwhile applications, is there a minimum level of financial
       support to applicants that it should consider? IF YES: What would that minimum
       amount be and why?

   11. Thinking of your project, would you say that the amount of funding was

          a lot more than needed
          somewhat more than needed
          about right
          somewhat less than needed
          a lot less than needed?

   12. Using a seven-point scale where 1 means not at all clear‖ and 7 ―very clear,‖ to
       what extent do you feel that the EDF program‘s deliverables and expected
       results are clear?

          What are the reasons that led you to this score?

   13. Was performance data collected for your EDF-funded project? Was this
       information reported against EDF activities/outputs and outcomes? Please
       provide examples.


Environment Canada                                                                        78
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund



           Are there any gaps in the information collected?
           How could the information related to your project‘s performance be
            improved?

   14. What have been the strengths and best practices in the design and delivery of
       the program? What has worked well?

       What have been the challenges or limitations? What has not worked well?
Success

   15. On a seven-point scale where 1 means ―largely unachieved‖ and 7 ―largely
       achieved,‖ to what extent do you feel that the intended outcomes of your funded
       projects were achieved?

           What are the reasons that led you to this score?

   16. Did your project have any unintended or unexpected outcomes, either positive or
       negative? If yes, please describe.

           Were any actions taken to address these unintended outcomes? If yes,
            please specify.

Cost-effectiveness and Alternatives

   17. Are you aware of any alternative programs or approaches in other federal
       departments/agencies or in other jurisdictions within or outside Canada that are
       comparable to the EDF?

   IF YES

           Based on what you know of the EDF, what aspects of these other programs
            are most similar to the EDF? What is most unique about them? Please
            describe.
           Are there any gaps in the activities and reach of these programs? IF YES: Is
            there anything EC‘s EDF could do to help address these gaps?

   18. As we near the conclusion, are there any other thoughts that you would like to
       share with respect to evaluating the Environmental Damages Fund?

THANK AND CONCLUDE




Environment Canada                                                                      79
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


                              Potential Recipient’s Guide

Hello, my name is _____ _____ and I‘m phoning from COMPAS Research. We‘re doing
an evaluation of the Government of Canada‘s Environmental Damages Fund. As part of
the evaluation, we‘re conducting interviews with municipalities, community
organizations, environmental organizations, universities, and First Nations‘s
organizations because these are the groups that are entitled to apply for project support
from the Environmental Damages Fund. Is there a day and time when you might be
available to take part in an interview?

A. IF THE KEY INFORMANT WANTS THE INTERVIEW NOW: Thank you for agreeing
   to the interview. We generally audio record our interviews to ensure our discussion
   is documented accurately. Would this be ok? In either instance, the interview would
   be confidential and no comment would be linked to you. UNPROMPTED

      Yes, agrees to taping
      Agrees to interview but no taping
      Reconsiders and does not agree to interview.

B. IF THE INTERVIEW IS SCHEDULED FOR ANOTHER TIME, (a) WE WILL
   TRANSMIT A COPY OF THE QUESTIONNAIRE AHEAD OF TIME TO THE
   PROSPECTIVE KEY INFORMANT AND (b) THE RESUMPTION OF THE
   CONVERSATION WOULD BEGIN HERE: Hello, it‘s _____ _____ phoning from
   COMPAS Research for the interview with respect to an evaluation of the
   Environmental Damages Fund. Thank you for agreeing to the interview. We
   generally audio record our interviews to ensure our discussion is documented
   accurately. Would this be ok? In either instance, the interview would be confidential
   and no comment would be linked to you. UNPROMPTED

      Yes, agrees to taping
      Agrees to interview but no taping
      Reconsiders and does not agree to interview.

Background

   1. Please briefly describe your role in any environmental project that [NAME OF
      ORGANIZATION] has undertaken or might undertake in the future.

Design and Delivery

   2. The Environmental Damages Fund, administered by Environment Canada,
      makes contributions to local government, community organizations,
      environmental organizations, universities and First Nations for environmental
      restoration, environmental quality improvement, research and development, and
      education and awareness projects using court awards raised from penalties on
      polluters. Have you heard of the fund before?

   3. Have you or your organization ever applied to the EDF for support? OPTIONAL
      PROMPT


Environment Canada                                                                     80
Audit and Evaluation Branch              Evaluation of the Environmental Damages Fund



          yes
          no
          DNK

   4. ASK ONLY IF HEARD OF EDF IN Q2: Considering what you know about the
      EDF, what might make the EDF an attractive source of funding for your
      organization?

   5. ASK ONLY IF HEARD OF EDF IN Q2: Is there anything about the EDF that
      might make it a less attractive source of funding?

   6. To ensure the highest quality of funded projects, the program needs to be
      promoted among [SPECIFY LOCAL GOVERNMENT, COMMUNITY
      ORGANIZATIONS, ENVIRONMENTAL ORGANIZATIONS, UNIVERSITIES,
      AND FIRST NATIONS ACCORDING TO THE TYPE OF ORGANIZATION OF
      THE RESPONDENT]. What would be the most effective way of doing so?

   7. For EDF to attract worthwhile applications, is there a minimum level of financial
      support to applicants that it should consider? IF YES: What would that minimum
      amount be and why?

   8. The EDF program encourages applicants to have partners. In your view, what
      effect would the requirement for partners have on organizations like your own if
      you were to apply for funding through this program?

   9. Based on your own or your organization‘s experience applying for grants or
      contributions of any kind, what have been the strengths and best practices in the
      design and delivery of this type of program?

          What have been the challenges or limitations? What has not worked well?

Cost-effectiveness and Alternatives

   10. Are you aware of any alternative programs or approaches in other federal
       departments/agencies or in other jurisdictions within or outside Canada that are
       comparable to the Environmental Damages Fund?

   11. Based on what you know of the EDF, what aspects of these other programs are
       most similar to the EDF? What is most unique about them? Please describe.
   12. Are there any gaps in the activities and reach of these programs? IF YES: Is
       there anything EC‘s EDF could do to help address these gaps?

   13. As we near the conclusion, are there any other thoughts that you would like to
       share with respect to the Environmental Damages Fund?

THANK AND CONCLUDE




Environment Canada                                                                      81
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


                                   Academic’s Guide

Hello, my name is _____ _____ and I‘m phoning from COMPAS Research. We‘re
currently conducting brief interviews with environmental experts as part of an evaluation
of the Environmental Damages Fund. The environmental experts we're consulting do
not need specific knowledge of the Environmental Damages Fund. Is there a day and
time when you might be available to take part in an interview?

A. IF THE KEY INFORMANT WANTS THE INTERVIEW NOW: Thank you for agreeing
   to the interview. We generally audio record our interviews to ensure our discussion
   is documented accurately. Would this be ok? In either instance, the interview would
   be confidential and no comment would be linked to you. UNPROMPTED

      Yes, agrees to taping
      Agrees to interview but no taping
      Reconsiders and does not agree to interview.

B. IF THE INTERVIEW IS SCHEDULED FOR ANOTHER TIME, (a) WE WILL
   TRANSMIT A COPY OF THE QUESTIONNAIRE AHEAD OF TIME TO THE
   PROSPECTIVE KEY INFORMANT AND (b) THE RESUMPTION OF THE
   CONVERSATION WOULD BEGIN HERE: Hello, it‘s _____ _____ phoning from
   COMPAS Research for the interview with respect to an evaluation of the
   Environmental Damages Fund. Thank you for agreeing to the interview. We
   generally audio record our interviews to ensure our discussion is documented
   accurately. Would this be ok? In either instance, the interview would be confidential
   and no comment would be linked to you. UNPROMPTED

      Yes agrees to taping
      Agrees to interview but no taping
      Reconsiders and does not agree to interview.

Background

   1. Please briefly describe your role within any environmental project that [NAME OF
      RESPONDENT’S ORGANIZATION] has undertaken or might undertake in the
      future.

Design and Delivery

   2. The Environmental Damages Fund, administered by Environment Canada,
      makes grants to local government, community organizations, environmental
      organizations, universities and First Nations for environmental restoration,
      research and development, and education and awareness projects using court
      awards raised from penalties on polluters. Have you heard of the fund before?

   3. Have you or your organization ever applied to the EDF?

          yes
          no


Environment Canada                                                                     82
Audit and Evaluation Branch              Evaluation of the Environmental Damages Fund


       DNK
   4. To ensure the highest quality of funded projects, the program needs to be
      promoted among [SPECIFY LOCAL GOVERNMENT, COMMUNITY
      ORGANIZATIONS, ENVIRONMENTAL ORGANIZATIONS, UNIVERSITIES,
      AND FIRST NATIONS ACCORDING TO THE TYPE OF ORGANIZATION OF
      THE RESPONDENT]. What would be the most effective way of doing so?

   5. For EDF to attract worthwhile applications, is there a minimum level of financial
      support to applicants that it should consider? IF YES: What would that minimum
      amount be and why?

   6. The EDF program encourages recipients to have partners. In your view, what
      effect is the requirement for partners likely to have on projects? Is this a
      desirable requirement?"

   7. Based on your own or your organization‘s experience applying for grants or
      contributions of any kind, what have been the strengths and best practices in the
      design and delivery of this type of program?

          What have been the challenges or limitations? What has not worked well?

Success

   8. EDF project grants contribute to environmental restoration, research and
      development, and education and awareness. On the basis of your broad
      knowledge, would you expect a fund like the Environmental Damages Fund to be
      more effective than otherwise or less effective than otherwise in respect of one
      or other of these three purposes? OPTIONAL PROMPT: As mentioned, these
      three purposes are environmental restoration, research and development, and
      education and awareness.

Cost-effectiveness and Alternatives

   9. Are you aware of any alternative programs or approaches in other federal
      departments/agencies or in other jurisdictions within or outside Canada that are
      comparable to the EDF?

          Based on what you know of the EDF, what aspects of these other programs
           are most similar to the EDF? What is most unique about them? Please
           describe.

          Are there any gaps in the activities and reach of these programs? IF YES: Is
           there anything EC‘s EDF could do to help address these gaps?

   10. As we near the conclusion, are there any other thoughts that you would like to
       share with respect to the Environmental Damages Fund?

THANK AND CONCLUDE




Environment Canada                                                                      83
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund


                               Judicial Segment’s Guide

Hello, my name is _____ _____ and I‘m phoning from COMPAS Research. We‘re
currently conducting interviews on behalf of Environment Canada with individuals in the
judicial or court systems as part of an evaluation of the Environmental Damages Fund.
Is there a day and time when you might be available to take part in an interview?
[IF NECESSARY SAY] ―Recently an e-mail was sent out from Shelley Borys at
Environment Canada advising you that COMPAS would be calling you to take part in
evaluation of the EDF program.‖

A. IF THE KEY INFORMANT WANTS THE INTERVIEW NOW: Thank you for agreeing
   to the interview. We generally audio record our interviews to ensure our discussion
   is documented accurately. Would this be ok? In either instance, the interview would
   be confidential and no comment would be linked to you. UNPROMPTED

      Yes, agrees to taping
      Agrees to interview but no taping
      Reconsiders and does not agree to interview.

B. IF THE INTERVIEW IS SCHEDULED FOR ANOTHER TIME, (a) WE WILL
   TRANSMIT A COPY OF THE QUESTIONNAIRE AHEAD OF TIME TO THE
   PROSPECTIVE KEY INFORMANT AND (b) THE RESUMPTION OF THE
   CONVERSATION WOULD BEGIN HERE: Hello, it‘s _____ _____ phoning from
   COMPAS Research for the interview with respect to an evaluation of the
   Environmental Damages Fund. Thank you for agreeing to the interview. We
   generally audio record our interviews to ensure our discussion is documented
   accurately. Would this be ok? In either instance, the interview would be confidential
   and no comment would be linked to you. UNPROMPTED

      Yes, agrees to taping
      Agrees to interview but no taping
      Reconsiders and does not agree to interview.

Background

   1. Please briefly describe any direct or indirect involvement you have had with legal
      activity in environmental areas in general and also with the Environmental
      Damages Fund.

Design and Delivery

   2. Have you ever heard of the Environmental Damages Fund?

   3. The Environmental Damages Fund, administered by Environment Canada,
      makes contributions to local government, community organizations,
      environmental organizations, universities and First Nations for environmental
      restoration, environmental quality improvement, research and development, and
      education and awareness projects using court awards raised from penalties on
      polluters. How familiar are the courts with the fund insofar as you can tell?


Environment Canada                                                                     84
Audit and Evaluation Branch                Evaluation of the Environmental Damages Fund



          How do the courts generally hear about this type of program?

   4. How might Environment Canada increase awareness of the EDF in the courts?

          And what do you feel could be done to promote the use of the EDF in the
           courts?

   5. As you likely know, EPAMs are negotiated settlements as an alternative to court
      prosecution for a violation of the Canadian Environmental Protection Act. Are
      efforts being made to encourage EPAM awards to be directed to the EDF?

          Is the use of EPAMs likely to grow or diminish in the future? Why?

   6. Although CEPA is the federal statute under which financial assignments to the
      EDF are generally made, it is also possible for the EDF to be used in relation to
      infractions under the Fisheries Act, the Canada Shipping Act, the Migratory Birds
      Convention Act, and the Species at Risk Act. In your view, are any efforts being
      made to encourage the courts to use these other laws to direct payments to the
      EDF? Please explain.

   7. Using a seven-point scale where 1 means ―not at all‖ and 7 means ―to a great
      extent,‖ to what extent do you feel the courts are using the EDF program as it
      was intended to be used?

          In what ways is the EDF‘s use by the courts consistent or inconsistent with
           the programs original design? PROBE Please explain and provide examples.

   8. What have been the strengths and best practices in the design and delivery of
      the EDF program through the courts?

       What has worked well? What aspects have been less successful?
Success

   9. On a seven-point scale where 1 means ―largely unachieved‖ and 7 ―largely
      achieved,‖ to what extent do you feel that the Environmental Damages Fund
      contributes to environmental restoration, environmental quality improvement,
      research and development, and education and awareness?

          What are the factors that lead you to give it this score?

Cost-effectiveness and Alternatives

   10. Are you aware of any alternative programs or approaches in other federal
       departments/agencies or in other jurisdictions within or outside Canada that are
       comparable to the EDF?

          Based on what you know of the EDF, what aspects of these other programs
           are most similar to the EDF? What is most unique about them? Please
           describe.


Environment Canada                                                                     85
Audit and Evaluation Branch              Evaluation of the Environmental Damages Fund


          Are there any gaps in the activities and reach of these programs? IF YES: Is
           there anything EC‘s EDF could do to help address these gaps?

   11. As we near the conclusion, are there any other thoughts that you would like to
       share with respect to evaluating the Environmental Damages Fund?

   12. Do you know of any individuals in the judicial or court systems who could
      provide feedback on the Environmental Damages Fund and its used by the
      courts? PLEASE ELICIT NAMES, TITLES, AND TELEPHONE NUMBERS TO
      THE EXTENT THAT THE KEY INFORMANT FEELS COMFORTABLE MAKING
      SUGGESTIONS.

THANK AND CONCLUDE




Environment Canada                                                                      86
Audit and Evaluation Branch               Evaluation of the Environmental Damages Fund



                         Annex 7: Summary of Findings

                                                  Progress        Little
                                                    Made,      Progress,        Not
     Evaluation Question (EQ)         Achieved
                                                  Attention    Priority for   Applicable
                                                   Needed       Attention
  Relevance:
 EQ1 Is there a legitimate and
      necessary role for
      government in this program          
      area or activity?
 EQ2 Is the program connected
      with societal/environmental         
      needs?
 EQ3 Does the program theory
      (activities, instruments,
      objectives, design) address         
      the societal need identified?
  Design and Delivery:
 EQ4 Identification of deliverables
      and results                                      
 EQ5 Delivered as designed                                           
 EQ6 Performance measurement
      and reporting                                    
 EQ7 Accountability                                    
 EQ8 Best practices and lessons
      learned
Success:
 EQ9 Achievement of program
      outcomes                            
     a) project delivery outcomes                      
     b) communications and
     promotion                                                       
     c) partnerships                                   ~
     d) proposal planning and
     development                          
 EQ10 Unintended outcomes                                                           
Cost-Effectiveness:
 EQ11 Alternatives                        
 EQ12 Duplication/
      complementarity                     
 EQ13 Improved efficiency                                                           
 EQ14 Value for federal dollars           



Environment Canada                                                                      87

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:12/25/2011
language:
pages:102