Project Assesssment Reports by vom48746


More Info
									Program Design & Evaluation   Ethics and Values
Every profession has distinct ethical obligations to the public. These
  obligations include professional competency, integrity, honesty,
       confidentiality, objectivity, public safety, and fairness.
All of these obligations are intended to preserve
         and safeguard public confidence
          Stem cell field rocked by scam of star scientist
          The star scientist at the heart of that deception –
          Hwang Woo Suk -- resigned Friday from his
          university post, his meteoric rise to fame coming
          to a wrenching end.

Unfortunately, all too often we hear reports in the
media of moral dilemmas and unethical behavior
by professionals. These reports naturally receive
considerable attention by the public, whose
confidence in the profession is undermined with
each report.
Educators are not immune to unethical practices

       Houston, Texas

                                Testing Scandal
                                in Texas Schools

                                 NPR News
                                March 21, 2005

                               by Claudia Sanchez
    Existing Ethical Standards
• Concerns about ethical practices in assessment
  are not new.
• As early as 1972, the National Council on
  Measurement in Education (NCME), the
  Association for Measurement and Evaluation in
  Guidance (AMEG), and the American
  Counseling Association (ACA) developed a
  position paper on the responsible use of tests
  that was intended to ensure that tests are given,
  and examinees are treated, fairly and wisely
  (AMEG, 1972).
 Testing & Evaluation Standards

• Later in the 1970s, ACA developed a
  statement on the responsibilities of the
  users of standardized tests and evaluation
• That document was revised as recently as
  1989 (ACA, 1989).
   Assessment Code of Ethics
• Both of these early documents recognized
  the need to positively influence the
  practices of those who use tests in ways
  that promote responsible use.
• These statements have been followed by
  the development of ethical standards by a
  number of other organizations having an
  interest, or directly involved, in testing
  assessment, and evaluation.
What About Evaluation Ethics?

     Principles & Standards
      Program Evaluators
American National Standards Institute
 The American National Standards Institute (ANSI) is
 a private, non-profit organization (501(c)3) that
 administers and coordinates the U.S. voluntary
 standardization and conformity assessment system.
 The Institute's mission is to enhance both the global
 competitiveness of U.S. business and the U.S. quality
 of life by promoting and facilitating voluntary
 consensus standards and conformity assessment
 systems, and safeguarding their integrity.
                   Founded in 1918
           Evaluation Standards

Created in 1975, the Joint Committee is a coalition of major professional
associations concerned with the quality of evaluation. The Joint Committee
is housed at The Evaluation Center, Western Michigan University.
The Joint Committee has published three sets of standards for evaluations:

1. The Personnel Evaluation Standards published in 1988
2. The Program Evaluation Standards published in 1994
3. The Student Evaluations Standards was published in 2003

The Joint Committee is accredited by the American National Standards
Institute (ANSI). Standards approved by ANSI become American National
Four Program Evaluation Standards
“…as it is written, so it shall be done!

   Utility Standards
   Feasibility Standards
   Propriety Standards
   Accuracy Standards
             Utility Standards (7)

    To ensure that the evaluation will serve the information
    needs of intended users
   U1 Stakeholder Identification Persons involved in or
    affected by the evaluation should be identified, so that
    their needs can be addressed.
   U2 Evaluator Credibility The persons conducting the
    evaluation should be both trustworthy and competent to
    perform the evaluation, so that the evaluation findings
    achieve maximum credibility and acceptance.
              Utility Standards

   U3 Information Scope and Selection
    Information collected should be broadly
    selected to address pertinent questions about
    the program and be responsive to the needs
    and interests of clients and other specified
   U4 Values Identification The perspectives,
    procedures, and rationale used to interpret
    the findings should be carefully described, so
    that the bases for value judgments are clear.
              Utility Standards

   U5 Report Clarity Evaluation reports should
    clearly describe the program being evaluated,
    including its context, and the purposes,
    procedures, and findings of the evaluation, so
    that essential information is provided and
    easily understood
   U6 Report Timeliness and Dissemination
    Significant interim findings and evaluation
    reports should be disseminated in a timely
               Utility Standards

   U7 Evaluation Impact Evaluations should be
    planned, conducted, and reported in ways that
    encourage follow-through by stakeholders, so
    that the likelihood that the evaluation will be
    used is increased, and to ensure that the
    evaluation will serve the information needs of
    intended users
          Feasibility Standards (3)

To ensure that an evaluation will be realistic, prudent,
diplomatic, and frugal

F1 Practical Procedures The evaluation procedures should
be practical, to keep disruption to a minimum while needed
information is obtained.
              Feasibility Standards
   F2 Political Viability The evaluation should be
    planned and conducted with anticipation of the
    different positions of various interest groups, so that
    their cooperation may be obtained, and so that
    possible attempts by any of these groups to curtail
    evaluation operations or to bias or misapply the
    results can be averted or counteracted.

   F3 Cost Effectiveness The evaluation should be
    efficient and produce information of sufficient value,
    so that the resources expended can be justified
          Propriety Standards (8)
  Intended to ensure that an evaluation will be conducted
  legally, ethically, and with due regard for the welfare of
  those involved in the evaluation, as well as those affected
  by its results.

P1 Service Orientation Evaluations should be designed to assist
  organizations to address and effectively serve the needs of the
  full range of targeted participants.
P2 Formal Agreements Obligations of the formal parties to an
  evaluation (what is to be done, how, by whom, when) should
  be agreed to in writing, so that these parties are obligated to
  adhere to all conditions of the agreement or formally to
  renegotiate it.
               Propriety Standards

   P3 Rights of Human Subjects Evaluations should be
    designed and conducted to respect and protect the rights and
    welfare of human subjects.
   P4 Human Interactions Evaluators should respect human
    dignity and worth in their interactions with other persons
    associated with an evaluation, so that participants are not
    threatened or harmed.
   P5 Complete and Fair Assessment The evaluation should be
    complete and fair in its examination and recording of strengths
    and weaknesses of the program being evaluated, so that
    strengths can be built upon and problem areas addressed.
               Propriety Standards
   P6 Disclosure of Findings The formal parties to an evaluation
    should ensure that the full set of evaluation findings along with
    pertinent limitations are made accessible to the persons
    affected by the evaluation and any others with expressed legal
    rights to receive the results.
   P7 Conflict of Interest Conflict of interest should be dealt
    with openly and honestly, so that it does not compromise the
    evaluation processes and results.
   P8 Fiscal Responsibility The evaluator's allocation and
    expenditure of resources should reflect sound accountability
    procedures and otherwise be prudent and ethically responsible,
    so that expenditures are accounted for and appropriate
             Accuracy Standards
  Intended to ensure that an evaluation will reveal and convey
  technically adequate information about the features that
  determine worth or merit of the program being evaluated.
A1 Program Documentation The program being evaluated
  should be described and documented clearly and accurately, so
  that the program is clearly identified.
A2 Context Analysis The context in which the program exists
  should be examined in enough detail, so that its likely
  influences on the program can be identified.
A3 Described Purposes and Procedures The purposes and
  procedures of the evaluation should be monitored and
  described in enough detail, so that they can be identified and
              Accuracy Standards
A4 Defensible Information Sources The sources of information
  used in a program evaluation should be described in enough
  detail, so that the adequacy of the information can be assessed.
A5 Valid Information The information-gathering procedures
  should be chosen or developed and then implemented so that
  they will assure that the interpretation arrived at is valid for the
  intended use.
A6 Reliable Information The information-gathering procedures
  should be chosen or developed and then implemented so that
  they will assure that the information obtained is sufficiently
  reliable for the intended use.
             Accuracy Standards

A7 Systematic Information The information collected,
  processed, and reported in an evaluation should be
  systematically reviewed, and any errors found should be
A8 Analysis of Quantitative Information Quantitative
  information in an evaluation should be appropriately and
  systematically analyzed so that evaluation questions are
  effectively answered.
A9 Analysis of Qualitative Information Qualitative information
  in an evaluation should be appropriately and systematically
  analyzed so that evaluation questions are effectively answered.
              Accuracy Standards

A10 Justified Conclusions The conclusions reached in an
  evaluation should be explicitly justified, so that stakeholders
  can assess them.
A11 Impartial Reporting procedures should guard against
  distortion caused by personal feelings and biases of any party
  to the evaluation, so that evaluation reports fairly reflect the
  evaluation findings.
A12 Meta-evaluation The evaluation itself should be
  formatively and summatively evaluated against these and other
  pertinent standards, so that its conduct is appropriately guided
  and, on completion, stakeholders can closely examine its
  strengths and weaknesses.
    What is Meta-Evaluation?
Simply stated, meta-evaluation is the evaluation of an
evaluation, evaluation system or evaluation device to
enhance the validity of the evaluation results.

Operationally, metaevaluation is also defined as the
process of delineating, obtaining, and applying descriptive
information and judgmental information - about the utility,
feasibility, propriety, and accuracy of an evaluation and its
systematic nature, competent conduct, integrity/honesty,
respectfulness, and social responsibility - to guide the
evaluation and/or report its strengths and weaknesses.
      Four Program Evaluation Standards

   Utility Standards – serve the information needs
    of the intended users
   Feasibility Standards – realistic, prudent,
    diplomatic, and frugal
   Propriety Standards – legal, ethical, and due
    regard for welfare of participants
   Accuracy Standards –reveal and convey
    information to determine the worth, value, and
    merit of the program evaluated
           American Evaluation Association
        Five Guiding Principles for Evaluators

   Ratified by the AEA membership, July 2004

   Supports Four Program Evaluation Standards
Principle 1 Systematic Inquiry
Evaluators conduct systematic, data-based inquiries
1. To ensure the accuracy and credibility of the evaluative
   information they produce, evaluators should adhere to the
   highest technical standards appropriate to the methods they
2. Evaluators should explore with the client the shortcomings
   and strengths both of the various evaluation questions and the
   various approaches that might be used for answering those
3. Evaluators should communicate their methods and
   approaches accurately and in sufficient detail to allow others
   to understand, interpret and critique their work..
Principle 2 Competence
Evaluators provide competent performance to stakeholders

   1. Evaluators should possess the education, abilities, skills and
    experience appropriate to undertake the tasks proposed in the
   2. To ensure recognition, accurate interpretation and respect
    for diversity, evaluators should ensure that the members of the
    evaluation team collectively demonstrate cultural competence.
    3. Evaluators should practice within the limits of their
    professional training and competence, and should decline to
    conduct evaluations that fall substantially outside those limits.
   4. Evaluators should continually seek to maintain and improve
    their competencies, in order to provide the highest level of
    performance in their evaluations.
Principle 3 Integrity/Honesty: Evaluators must display
honesty and integrity in their own behavior and in the
evaluation process.
   1. Evaluators should negotiate honestly with clients and relevant
    stakeholders concerning the costs, tasks to be undertaken,
    limitations of methodology, scope of results likely to be obtained,
    and uses of data resulting from a specific evaluation.
   2. Before accepting an evaluation assignment, evaluators should
    disclose any roles or relationships they have that might pose a
    conflict of interest with their role as an evaluator.
   3. Evaluators should record all changes made in the originally
    negotiated project plans, and the reasons why the changes were
   4. Evaluators should be explicit about their own, their clients',
    and other stakeholders' interests and values concerning the
    conduct and outcomes of an evaluation.
Principle 3 Integrity/Honesty (continued)
 5. Evaluators should not misrepresent their
  procedures, data or findings.
 6. If evaluators determine that certain
  procedures or activities are likely to produce
  misleading evaluative information or
  conclusions, they have the responsibility to
  communicate their concerns and the reasons for
  them. If discussions with the client do not
  resolve these concerns, the evaluator should
  decline to conduct the evaluation.
 7. Evaluators should disclose all sources of
  financial support for an evaluation, and the
  source of the request for the evaluation.
     Principle 4 Respect for People: Evaluators respect the
     security, dignity and self-worth of respondents, program
     participants, clients, and other evaluation stakeholders.

   1. Evaluators should seek a comprehensive understanding of the important
    contextual elements of the evaluation.

    2. Evaluators should abide by current professional ethics, standards, and
    regulations regarding risks, harms, and burdens that might befall those
    participating in the evaluation; regarding informed consent for participation in
    evaluation; and regarding informing participants and clients about the scope
    and limits of confidentiality.

   3. Because justified negative or critical conclusions from an evaluation must
    be explicitly stated, evaluations sometimes produce results that harm client
    or stakeholder interests. Under this circumstance, evaluators should seek to
    maximize the benefits and reduce any unnecessary harms that might occur,
    provided this will not compromise the integrity of the evaluation findings.
Principle 4 Respect for People (continued)
   4. Knowing that evaluations may negatively affect the interests
    of some stakeholders, evaluators should conduct the evaluation
    and communicate its results in a way that clearly respects the
    stakeholders' dignity and self-worth.
   5. Where feasible, evaluators should attempt to foster social
    equity in evaluation, so that those who give to the evaluation may
    benefit in return. For example, evaluators should seek to ensure
    that those who bear the burdens of contributing data and
    incurring any risks do so willingly. Program participants should be
    informed that their eligibility to receive services does not hinge on
    their participation in the evaluation.
   6. Evaluators have the responsibility to understand and respect
    differences among participants, such as differences in their
    culture, religion, gender, disability, age, sexual orientation and
    ethnicity, and to account for potential implications of these
    differences when planning, conducting, analyzing, and reporting
Principle 5
   Responsibilities for General and Public Welfare

Evaluators articulate and take into account the diversity of
general and public interests and values that may be related to the
 1. When planning and reporting evaluations, evaluators
  should include relevant perspectives and interests of the full
  range of stakeholders.
 2. Evaluators should consider not only the immediate
  operations and outcomes of whatever is being evaluated, but
  also its broad assumptions, implications and potential side
Principle 5
           Responsibilities for General and Public Welfare
   3. Freedom of information is essential in a democracy. Evaluators
    should allow all relevant stakeholders access to evaluative information
    in forms that respect people and honor promises of confidentiality. In
    all cases, evaluators should strive to present results clearly and simply
    so that clients and other stakeholders can easily understand the
    evaluation process and results.
   4. Evaluators should maintain a balance between client needs and other
    needs. Evaluators necessarily have a special relationship with the client
    who funds or requests the evaluation. By virtue of that relationship,
    evaluators must strive to meet legitimate client needs whenever it is
    feasible and appropriate to do so. However, that relationship can also
    place evaluators in difficult dilemmas when client interests conflict
    with other interests, or when client interests conflict with the obligation
    of evaluators for systematic inquiry, competence, integrity, and respect
    for people.
Principle 5
           Responsibilities for General and Public Welfare

   5. Evaluators have obligations that encompass the public
    interest and good. These obligations are especially important
    when evaluators are supported by publicly-generated funds;
    but clear threats to the public good should never be ignored in
    any evaluation. Because the public interest and good are rarely
    the same as the interests of any particular group (including
    those of the client or funder), evaluators will usually have to
    go beyond analysis of particular stakeholder interests and
    consider the welfare of society as a whole.
Summary of 5 Guiding Evaluation Principles

   1. Systematic Inquiry: Evaluators conduct systematic, data-
    based inquiries about whatever is being evaluated.
   2. Competence: Evaluators provide competent performance to
   3. Integrity/Honesty: Evaluators ensure the honesty and
    integrity of the entire evaluation process.
   4. Respect for People: Evaluators respect the security, dignity
    and self-worth of the respondents, program participants, clients,
    and other stakeholders with whom they interact.
   5. Responsibilities for General and Public Welfare:
    Evaluators articulate and take into account the diversity of
    interests and values that may be related to the general and
    public welfare.
         Program Design & Evaluation
           Standards and Principles
   Four Program Evaluation Standards
    Utility, Feasibility, Propriety, & Accuracy

   Five Guiding Principles for Evaluators
    Systematic Inquiry, Competence, Integrity/Honesty,
    Respect for People, Responsibilities for General &
    Public Welfare
            Program Design & Evaluation

    Standards                      Principles
 Four Program                  Five Guiding Principles
Evaluation Standards                for Evaluators

   UFPA                           SCIRR

 Utility,                       Systematic Inquiry,
 Feasibility,                   Competence,
 Propriety,                     Integrity/Honesty,
 Accuracy                       Respect for People,
                                Responsibilities for
                                General & Public
     &                             Define Purpose
EVALUATION                             Engage
  PROCESS                           Stakeholders

                               5 Guiding Principles     Describe
                  Report                                Program
                Evaluation                             Evaluation
4 Standards    Ensure Use &
               Share Lessons
5 Principles
                                                      Focus Design,
6 Steps        Analyze Data                              Develop
                 Justify                              Methodology,
               Conclusions                            Data Collection
                               5 Guiding Principles

                                  Collect the
                                 Data & Gather
When conducting program evaluations, we must always
remember the words of wisdom coined by a famous scientific
researcher and evaluator:

           “Not everything that can be counted counts…and
           not everything that counts can be counted.”


To top