Get documents about "Planning and Implementing Program Evaluations within the
Document Sample
PLANNING AND IMPLEMENTING PROGRAM EVALUATIONS
WITHIN THE OFFICE OF SOLID WASTE
Program evaluation can be an important and objective and formal assessment of the
useful tool for program management. This results, impact, or effects of a program or
paper lays out a strategy for planning and policy.” Program evaluations also may
conducting program evaluations for the RCRA examine the implementation (processes
program. This strategy includes and/or results) and policies of the programs.
A program evaluation will draw conclusions
# definitions and descriptions of about the effectiveness of the design,
program evaluation, implementation, and/or impacts of a program.
# a discussion of the rationale for
conducting evaluations, and a “Program” here includes the usual
summary and potential applications connotation of “a set of staff activities with a
for evaluation, and defined goal or purpose;” this is what GPRA
# proposed procedures for identifying terms “activities” to achieve goals and
candidate programs for evaluation objectives. The term “program” may also
and for conducting program include a policy initiative, an investment
evaluations in the RCRA program. project, or even a change in procedures.
Purpose of Program Evaluation
I BACKGROUND AND DEFINITIONS
Programs are often developed and
implemented with a linear process that looks
Program evaluation is defined 1 as “an like this:
1
Government Performance and Results Act
of 1993: Report of the Committee on Governmental
Affairs, United States Senate, to Provide for the
Establishment, Testing, and Evaluation of Strategic
Planning and Performance Measurement in the
Federal Government, and for Other Purposes, p. 32.
1
“Implementation” is considered to be the last intended outcomes, including diverting
phase of the project. When implementation wastes from landfills and reducing carbon
is over, the project is considered to be emissions?
complete. If the process ends there,
however, important questions are never Process evaluations examine the operations
answered: What were the results? Was the and procedures of the program. How has the
program effective? How did the program program been planned and implemented?
work? What entities have taken part, what are their
roles? How do the implementors and
Evaluation is the step that provides this stakeholders interact, and how do these
information feedback. Evaluation is therefore interactions affect the operation of the
a crucial (though often overlooked) function program? For example, the GPRA team is
of program management. By providing a considering a process evaluation focusing on
mechanism to allow management to monitor the Part 258 criteria for municipal landfills.
implementation of programs, program What has been the process for implementing
evaluation is a means to the regulations after promulgation by EPA?
What has worked, and what hasn’t? Process
# measure the success of a program evaluations might focus on headquarters
# communicate the strengths and activities (e.g., the process for listing
advantages of programs hazardous wastes) or on Regional activities
# identify potential improvements (e.g, the permitting process, or enforcement).
# redirect resources to ensure effective
use of staff time and funds. Formative evaluations are used in the earliest
stages of a program, during the design phase
Using program evaluation as a diagnostic tool or early in the implementation phase. These
can help improve our processes, make our evaluations examine the intent of the
work more efficient, and improve our ability program, and help assess features of program
to fulfill our statutory mission. design and operating environment (including
external factors) that influence the chances of
Types of Evaluations successful achievement of program
objectives. An essential feature of formative
There are three principal types of evaluations. evaluations is identification of potential
Each is directed towards gaining a specific barriers to program success.
type of information, for a specific purpose.
Outcome evaluations examine the
achievements of a program; they therefore
are focused on the “big picture”: What are
the outcomes of implementing a program?
What are the intended and unintended results
of implementing the program? What are the
factors (both within and outside the Agency)
that are responsible for achieving those
outcomes? For example, the GPRA team is
considering an evaluation of the outcomes of
the Pay-as-You-Throw outreach program.
Was the program successful in achieving
2
II THE ROLE OF PROGRAM account for program evaluations:
EVALUATION IN THE RCRA
PROGRAM Finally, the reports should also relate
performance measurement
The GPRA specified certain roles for program information to program evaluation
evaluation within the framework of the findings, in order to give a clear
performance measurement system the picture of the agency’s performance
legislation requires. This section explains and its efforts at improvement. 2
these roles, as well as other ways that
program evaluation can be of benefit to the Program Evaluation in
RCRA program. Implementation of GPRA
Legislative Specifications within The OSWER annual plan features a series of
GPRA goals, objectives, and subobjectives. The
goals are overarching missions for the
Although the GPRA statute does not detail a Agency; the objectives represent broad
required schedule of program evaluations, the targets at the AA level. The subobjectives are
law clearly presumes they will be conducted. generally specific to RCRA programs.
The committee report accompanying the
legislation and the legislation itself speak The Annual Performance Plan also specifies
specifically of the role of program evaluations the activities that will be conducted to
in the annual cycle of performance planning accomplish these goals. (Using the systems
and reporting. framework, GPRA also refers to these
activities as “outputs”). There is not
Strategic Plan: According to the GPRA, the 5- necessarily a one-to-one correspondence
year strategic plan should contain a between these activities and goals/
“description of the program evaluations used objectives/ subobjectives. In some cases,
in establishing or revising general goals and multiple activities are conducted to serve a
objectives.” The legislation also guides the particular subjobjective; in other cases, a
agency to include a schedule for future single activity contributes to several
progam evaluations to be conducted. subobjectives.
Annual Performance Report: The annual Evaluation of these activities: One use of
performance report itself renders a limited program evaluation in GPRA is to evaluate
evaluative function. Referring to the program the relationship between these activities and
objectives and measures in the annual subobjectives. Program evaluation may
performance plan, the end-of-year focus on the individual activity or an
performance report describes the agency’s individual subobjective. For example, an
activities during the year, and accounts for evaluation might look at a single
the degree to which objectives and measures subjobjective (for example, # 2.06 “Waste
have been accomplished. The legislation management facilities have approved
notes that the performance report “ could controls”) and examine how a variety of
include results of program evaluations programs (LDRs, new listings, permitting,
conducted during the course of a year.” The etc.) are contributing to accomplishing the
Government Operations Committee also
explains how the performance report is to 2
p. 16.
3
subobjective. Conversely, an evaluation performance results that appear in the annual
might look at a single program (for example, performance report. The evaluation can
WasteWise), to assess how it contributes to describe and assess the various internal and
subobjective 7.03 (MSW recycling rates) as external factors that affect program results,
well as subobjective 7.04 (waste generation thus providing a clearer idea of program
rates). results.
Clarification of Program Results: Each year, Program Evaluation as a
the Agency must account for progress on Management Tool
performance measures in a Performance
Report. This report focuses on the Program evaluation has been in use at all
relationship between EPA inputs (budget and levels of government, prior to and outside of
FTEs), outputs (program activities), and the enactment of GPRA. These agencies
outcomes (environmental results). have seen program evaluation is a useful tool
Unfortunately, a simple tallying of numeric for managing programs and communicating
program measures and progress toward results.
objectives may provide a misleading idea of
program activities and achievements. Program evaluation can facilitate program
management by helping managers
This opportunity for confusion largely exists understand how programs are working: by
because EPA may not have meaningful identifying particularly successful aspects of
control over the “outcomes” that GPRA projects and programs, managers may choose
prefers us to measure, since many of our to divert resources to these more productive
goals are dependent on actions by a number or fruitful activities. Evaluation may also
of public and private entities. For example, identify external or exogeneous factors,
EPA is not able to cause generators to outside of the Agency’s control, that are
produce less waste; similarly, the quality of hindering program success. By evaluating
groundwater may be affected by many factors not only the extent but the reason for
which are not accountable to activities success, RCRA managers get valuable
conducted under the RCRA program information to help adjust programs to
(including releases which are not regulated improve progress toward achieving desired
under Subtitle C). environmental outcomes.
Despite the minimal level of influence that Under the GPRA reporting framework,
RCRA may have over these measures of program evaluation can serve a
environmental quality, EPA nevertheless communicative function, too, by
needs to be able to explain the relationship demonstrating the value of programs (even
between the activities of the RCRA program those outside of formal GPRA reporting). A
and the environmental outcomes. Measuring program evaluation might focus on a
goals and progress alone won’t be sufficient particular program (rather than a particular
to explain the intricacies of individual performance measure), and show how it
programs, or the role of external factors such contributes to achievement of various
as state actions, industry activities, economic subobjectives. In other cases, RCRA may
factors, or legislative constraints. wish to highlight successes that are not
directly related to GPRA objectives. Outside
Program evaluation can have a strategic value of program evaluation, there is little
by providing clarifying explanation of the
4
opportunity to recognize the value of these
programs.
5
III MANAGING PROGRAM possible merit or significance of conducting
EVALUATIONS an evaluation with reference to the value and
feasibility of an evaluation. (see Figure 1).
The GPRA team is proposing that OSW For each candidate program, the team will
create a self-directed RCRA Program
Evaluation Team to manage and conduct
Figure 1
evaluations of RCRA programs. Since
evaluations could cover all phases of Some Determinants of the Merit or
planning and implementation of RCRA Significance of a Program Evaluation
programs, the team should include members
from both OSW and Regional RCRA staffs. ! Could this program use assistance
One OSW special assistant should always be in communicating its strengths and
a member of the team, as well. The team worth?
will answer to the Deputy Office Director. ! How important is this program in
The Program Evaluation Team will be reaching RCRA objectives?
responsible for ! How much potential is there for
improving outcomes and/or
# identifying candidates for program process?
evaluation, ! Have there been changes (or are
# managing and oversee evaluations, changes anticipated) in the
and regulated industry, facilities,
# assisting program managers in legislative decisions, budget, or
understanding and acting on other aspects of the operating
evaluation results and environment that would make this a
recommendations good time to examine the program?
! Does the program use a significant
Identifying Candidates for Program percentage of RCRA resources?
! How will the results of the
Evaluation evaluation help senior managers
make better decisions?
To identify information needs which could be ! Do we expect that an evaluation of
answered by program evaluations, the this program will identify potential
program evaluation team will review the program improvements?
GPRA strategic plan, annual performance ! Has any source outside of RCRA
plans, annual performance reports, Division requested an evaluation of this
operating plans, and the Beginning-of-Year program?
Plan. They may also review reports from the ! Does this program have areas that
GAO, OIG, or other EPA offices. Other are suspected to be weak?
information needs may be discovered by ! Has this program received criticism
polling management and staff on a formal or that the program managers and
informal basis. others involved with RCRA would
like to address?
From all of these sources, the program ! Will an evaluation of this program
evaluation team will develop a “long list” of support reporting on progress
candidate programs for evaluation. The toward GPRA goals?
team will meet periodically to review the list,
and make a preliminary assessment of the
6
identify the type of evaluation suggested, on programs to evaluate and will allocate
and the value that an evaluation could bring appropriate funds to the team for these
to the program or division. evaluations.
The team will submit this “long list” of Conducting Evaluations
candidates for evaluation to OSW Senior
Staff. When division directors submit their Staffing can be a difficult issue for internal
annual workplans to CIRMD each fall, they program evaluations. The nature of the task
will also submit their preferred candidate requires independence and expertise.
programs to be evaluated in the upcoming
year, with a rationale for each suggested Among the options available for staffing
program evaluation. internal OSW program evaluations:
Selecting Candidates for Evaluation EPA staff : OSW personnel have the most
expertise in the program, and would be able
The submissions from the division directors to identify issues and answers more quickly
will constitute the “short list” of candidates than anyone else. Objectivity and
for evaluation. The program evaluation team availability of time could be problematic
will meet (shortly after the submission of issues in producing thorough evaluations.
division workplans) to review and rank the
candidates, and choose among them. Existing OSW mission contractors These
contractors have significant expertise and
The team will use a formal or informal understanding of OSW programs; they also
weighing and scoring process to recommend have the easiest access to resources to
1-3 evaluations each year. The scoring conduct the evaluations. However, they
process will involve an assessment of the likely have little expertise at conducting
value and feasibility of conducting the evaluations, and may not feel the
evaluation. The feasibility will depend on independence necessary for an objective
question such as timing, the availability of evaluation.
relevant data, and the cost in time and funds
to collect the data and complete a report. Independent Contracts (small purchases) :
This mechanism could provide the most
Based on the criteria of value and feasibility, independence from management, and the
the program evaluation team will recommend widest latitude for evaluation and
candidates for evaluation. Their report to recommendations. It is likely that a
OSW senior staff will include contractor would be able to provide either
recommendations for program evaluations, expertise at evaluation, or expertise in RCRA
staff and resource needs, and projected time issues, but probably not both. This
frames. The team will develop its report with mechanism could also be the most time-
assistance from program staff in the divisions. consuming to arrange.
OSW senior staff will make the final
decision The GPRA team recommends that program
evaluations be conducted by OSW staff with
contract support, where possible. OSW
might also wish to investigate establishing
program evaluation capacity jointly with
other OSWER offices. This would have the
7
advantage of concentrating expertise in considered:
evaluation.
< The level of resources expended
In all cases, the program evaluation team will should be commensurate with the
have responsibility for managing, conducting benefit to be derived from the study
guiding, and/or overseeing program results. The more important the
evaluations. Evaluations will need a answers, the more resources it is
combination of knowledge of evaluation justifiable to devote to the study.
methodology and knowledge of the program < What level of detail is necessary? If
itself. In addition to members of the the evaluation is raising questions
program evaluation team, therefore, each about causes and effects, how
evaluation will require at least 15-20 hours conclusive do results need to be?
from OSW program staff, and 4-6 hours from < Travel could be a component of many
division management to help guide the evaluations. Since GPRA measures
evaluation. are focused on environmental
outcomes, a major focus of any
Since implementation of RCRA programs evaluation is program implementation
takes place at the Regional and state level, to meet those outcomes. The
many evaluations are going to focus more on majority of that implementation is
field activities than headquarters. We can taking place in the Regions and
expect that OSW participants will be more States. It will likely be necessary for
versed in the techniques and conduct of investigation and review to take place
evaluations than Regional personnel, and will in the Regions and States, as well.
need to assist Regional personnel in
managing those evaluations. It would be If we use an outside consultant to conduct an
helpful to develop an understanding of evaluation, we anticipate that a program
program evaluation within RCRA’s regional evaluation could cost between $25,000-
branches and divisions. $125,000. The great variation depends on
the scope of the evaluation, the evaluation
In most cases, OSW staff should be supported skill and RCRA background of the consultant,
by contractor assistance. Contract resources and the scope and travel issues noted above.
could be critical in conducting evaluations.
OSW will secure evaluation expertise in one The number of evaluations to be conducted
of two ways: each year should depend on the importance
of the information to be gained, as tempered
1) Existing contract vehicles within OSW, or by the availability of resources.
with other OSWER or EPA program offices.
2) The program evaluation team will secure
expert services on an individual basis, as
small purchase procurements.
Since evaluations will be selected and
conducted on an ad hoc basis, resource
needs will vary from year to year. When
deciding how much to allocate for
evaluations, the following issues need to be
8
Related docs
