PPD Evaluation Tool

					Public-Private Dialogue
 DFID | WB | IFC | OECD


                   Part 1 : Evaluation of Organizational P

2 or 3 indicators are derived for each of the 12 elements of the Charter; with one or more indices defin
in the calculation of each indicator.



                     SUMMARY TABLE

     Mandate and institutional alignment
     Structure and participation
     Champion(s) and leadership
     Facilitation and management
     Outputs
     Outreach and communication
     Monitoring and evaluation
     Sub-national
     Sector specific
     Relevance to FDI
     Post-conflict/disaster/crisis
     Development Partners

     PPD SCORE




    Number         Operational Process Indicators
Mandate and Institutional alignment:
                   Existence of mission statement and capacity of
           1       participants to explain this mission statement.
                   Desk study Interviews
                   (minimum of 5 interviews with stakeholders)




                   Degree of anchorage of the partnership into existing
           2       public institutions as per its mandate.
                   Desk study Interviews




                   Institutional readiness to implement PPD
           3
                   recommendations

Structure and Participation


                   Existence of rules and regulations in the partnership,
           4       including formal mechanisms in place to balance power.
                   Desk study Interviews




                   Degree of participatory decision making.
           5
                   Desk study Interviews




Champions and Leadership


                   The presence and clear involvement of champions who
           6       are recognized as such by stakeholders
                   Interviews




                   Continuity of involvement of champions in dialogue or in
           7       partnership
                   Interviews

Facilitation and Management




                   Quality of facilitation of the PPD
           8
                   Desk Study Interviews
                Quality of facilitation of the PPD
          8
                Desk Study Interviews




                Quality of ppd logistics and management arrangements
          9     (responsibilities, tasks, structure, arrangements etc.)
                Desk Study Interviews




Outputs




          10    Hard output




          11    Soft output




                Impact Output :Degree to which dialogue or partnership
          12    has innovated or changed existing institutional structures.
                Interviews with external stakeholders




Communication and Outreach



                Quality and frequency of communication between
          13    different stakeholder groups
                Observation of meetings ; Interviews
                  Amount and kind of outreach and communication
          14      activities to civil society and media
                  Desk-study Interviews (internal and external stakeholders)




Monitoring and Evaluation

          15      Quality of internal monitoring tools used




                  Quality of reporting and documentation on activities of the
          16
                                          partnership




                     Degree to which monitoring results have resulted in
          17
                               changes in planning and targets




          18       Use of ex-post Assessment


Sub-national



                  Existence of local and regional structures or consultation
          19      mechanisms for the dialogue or partnership
                  Desk study Interviews; Interviews with beneficiaries and target groups
                  at the local level




                  Existence of activities of the PPD at other levels (local,
                  regional or national) through ad hoc activities or
          20
                  dedicated programs or working groups
                  Interviews (internal and external stakeholders)

Sector Specific
                   Degree to which the dialogue or partnership addresses
          21       specific problems of participants
                   Desk study Interviews


                   Capacity of the dialogue or partnership to generate
          22       concrete solutions to specific problems of participants
                   Desk study Interviews


Relevance to FDI

                   Usage of the PPD to promote cross border investment
                   policy reforms
          23




          24
                   Appropriate inclusion of foreign investors voice in the
                   PPD
Post-conflict/disaster and crisis mitigation/management



                   Capacity to put conflicts on the agenda of the dialogue or
          25       partnership and resolve them
                   Desk study ; Interviews (participants in the PPD)




                   Contribution made by the dialogue or partnership to
                   conflict resolution and peace building in its external
          26
                   environment.
                   Desk study ; Interviews (external stakeholders)




Development Partners



          27       Degree of dependence of the PPD on financial support of
                   development partners (DPs)
                   Desk study




          28       Degree of autonomy of the agenda of the PPD from
                   agendas of development partners
                   Desk study ; Interviews
29
     Degree of which the DPs give the needed assistance to
     the PPD facilitator
        ialogue
luation of Organizational Process and Effectiveness

12 elements of the Charter; with one or more indices defined -each scores from 0 to 10 - to be combined




                             SCORE                  Weight

                                8.83                        1                                      Development Partners

                                8.00                        1
                                8.50                        1                    Post-conflict/disaster/crisis
                                6.50                        1
                                6.78                        1
                                6.83                        1
                                                                                       Relevance to FDI 7.50
                                4.88                        1
                                5.50                        1
                                3.50                        1
                                7.50                        1                                 Sector specific

                                7.25                        1
                                6.67                        1

                                          6.73


                                                                 Average     Average
                                                      Indices
                                                                Scoring for Scoring for
                Indice measurement                 Scoring on 0                each
                                                                   each
                                                    to 10 scale              Charter
                                                                 indicator
                                                                             Element
                                                                                    8.8
Non-existence=0;existence ( in coherent
                                                 9
written document)=10.
 Percentage respondents who are able to              9
recite the substance of the mission              8
statement (none=0, all=10).
Percent of participants with decision making
power in their home institutions (none=0;        9
all=10).
                                                     9
 Mandate formally accepted and signed by
                                                 9
relevant public institutions (none=0, all=10).
Existence of a dedicated public institution in
charge of following up on the
                                                 9   9
implementation of the PPD’s
recommendations (none=0, yes =10)
                                                         8.0
Non-existence of documents with rules and
regulations=0, Complete set of rules and         8
regulations=10.
Equal participation of each stakeholder              8
group(in number and level representatives):
                                                 8
unequal or stakeholder groups missing=0;
exactly equal=10).
Percent of decisions reached by consensus
or vote during partnership meetings(none=0,      8
all=10).
                                                     8
Active contribution of all different
stakeholder groups in developing proposals       8
(none=0, all=10).
                                                         8.5
Existence of at least one champion in each
of the participating stakeholder groups          9
(none=0, all=10).
Percent of respondents that identify the             9
same champion(s) (all mention different
                                                 9
champions=0,all mention the same
one(s)=10).
Turnover rate of champions. (high, staying
on only a few months=0 to low , continuous       8   8
presence=10)
                                                         6.5
Existence of Terms of Reference
for facilitators and other
members of the Secretariat?                      8
(Nonexistent=0; coherent written
document=10)
% of reform proposals receiving significant
contribution from Secretariat staff (zero=0,     7
                                                     8
100%=10)
Existence of rigorous recruiting and training
programme for Secretariat members                8
(none=0, yes=10)
                                                     8




Percent of respondents who
indicate that facilitators perform               9
well (none=0, all=10)
Existence of standardized documents
addressing all PPD logistical aspects (0=
none, 5= some documents but some                 7
aspects are missing, 10= all aspects
covered)
Existence of task descriptions for
                                                     5
manager(s), and – if there is more than one
                                                 4
manager – clear division of tasks (non-
existence=0, clear description/division=10).
 Timely availability of project plans and
timelines for all stakeholders (no and not for   4
all=0 and yes for all=10)
                                                         6.8
Existence of evidence-based analytical
output: %of PPD recommendations that
include policy papers, position papers,          7
reviews or
assessments. (none =0, all = 10)
Volume of recommendation: Number and
kind of economic and/or reform proposals             5
                                                 5
(none=0, as planned =6, exceeding
planned=10)

PPD process outputs: number of working
group and forum meetings (none = 0, as           4
planned = 10)


% of respondents of stakeholders indicating
increase in trust, understanding and
                                                 8   8
cooperation between stakeholders; building
of social capital.(none=0, all=10)

% of respondents of external organizations
who indicate the PPD has had influence on
                                                 7
activities of their organizations. (none=0,
all=10)
Appreciation expressed by external                   7
stakeholders on the performance of the
PPD (no knowledge at all/low                     7
appreciation=0; detailed knowledge and
high appreciation=10)
                                                         6.8
Distribution of time between listening and
speaking of participants of different
                                                 7
stakeholder groups in meetings of the PPD
(extremely unequal=0 and very equal=10).             7
Number of misunderstandings or
disagreements in communication that are          7
clarified (none=0, all=10)
Frequency of updates provided about the
PPD (no updates provided= 0, updates
                                                  7
provided but in inconsistent format = 5,
regular and consistent updating = 10)
Diversity of tools used to reach key
audiences (none=0, one or two=5, at least         7   7
three=10)
Amount of (written, verbal, television)
external communication messages (none=0,
                                                  6
total amount (needs to be customized to
situation)=10
                                                          4.9
Application of tracking tools to keep abreast
of all stages of each reform proposal (none       5   5
= 0, on some issues = 5, regular and

Number and frequency of monitoring reports
                                                  6
(on a scale from 0 to 10)
                                                      6
% compliance of reporting with qualitative
targets set for monitoring (not at all=0,         5
entirely=10)
Existence of specific criteria (including cost-
benefit analysis) as basis for issue selection    7
(none=0, existence of specific
Existence of follow up process with
decision makers (none = 0, yes but
                                                  8
ineffective or irregular = 5, yes and                 6
effective = 10)
Percent of follow-up actions on
recommendations in monitoring reports (no
                                                  3
recommendations followed up=0, all
recommendations followed up=10)

%PPD recommendations that have ex-post
assessments carried out on reforms passed         3   3
thanks to the PPD (none=0, all=10)

                                                          5.5
Consultation of PPD (through formal
structures/channels) at further decentralized
                                                  4
geographical levels (no at all=0, many
channels and all relevant levels=10).
                                                      3
Percent of respondents at the level of local
target groups (indirect beneficiaries of the
                                                  2
PPD) who are satisfied with the
performance of the PPD (none=0, all=10)
Number of activities at other levels than the
dialogue and partnership itself (none=0,          8   8
many and at many different levels=10)

                                                          3.5
Number of (sub) sector -specific working
groups in the PPD (none=0, all relevant sub       2   2
sectors= 10)
Number of (sub) sector specific proposals
generated (none=10, at least one per year         4
for each (sub) sector or issue=10)                    5
Quality of these proposals rated by the
                                                  6
evaluator(s) (on a scale from 0-10)
                                                          7.5
Number of proposed reforms that tackle
issues directly related to FDI (none=0,           8
all=10)
                                                      8
Number of proposed reforms that include
FDI-related rationale in the reform proposal      8
text (none=0, all=10)
Given the reform agenda, the proportion of
foreign investors or associations
representing foreign investors is a direct
                                                  7   7
impediment to the progress of the reform
agenda (=0), is somewhat adequate (=5), is
perfectly balanced (=10)
                                                          7.3
Existence of an internal communication
strategy to mitigate conflicts (not existent=0,   7
existent (written and coherent) =10)
                                                      8
Percent of conflicts that have been
peacefully resolved within the PPD
                                                  8
according to respondents. Average % of all
respondents (none=0, all=10)
Existence of an external communication
strategy to mitigate conflicts in the direct
external environment of the PPD (not              7
existent=0, existent (written and coherent)
=10
                                                      7
Number of relevant conflicts in the direct
context of the PPD positively influenced by
the PPD, according to external                    7
stakeholders. (no influence at all=0, in all
conflicts positive contribution noticeable=10)

                                                          6.7
Amount of financial support from DPs as a
percentage of the total costs of the dialogue
or partnership (total budget provided by          4   4
DPs=0, more than 50% of budget provided
by own resources=10)

Number of points on the agenda that were
promoted by DPs as a percentage of total
                                                  9   9
issues on the agenda. (all points promoted
by DPs=0, no points promoted by DPs=10
Amount and quality of training received by
the PPD Secretariat members                  7   7
(insufficient=0, appropriate=10)
d Effectiveness

0 to 10 - to be combined


                                      PPD Evaluation Wheel
                                               Mandate and institutional
                                                     alignment
                                                        8.83

                Development Partners                                          Structure and participation
                                                                           8.00
                                            6.67


   conflict/disaster/crisis                                                               8.50
                                                                                            Champion(s) and leadership
                              7.25




    Relevance to FDI 7.50                                                             6.50      Facilitation and management


                                        3.50

                                                                                   6.78
           Sector specific                                                                 Outputs
                                             5.50
                                                         4.88
                                                                       6.83
                             Sub-national                                     Outreach and communication

                                               Monitoring and evaluation
www.publicprivatedialogue.org
Operational Guidelines for the
Charter of Good Practice in Using
Public-Private Dialogue
for Private Sector Development
 Public-Private Dialogue
    `


                          Part 2 : PPD Impact on the Reform Pro

NB: Following the different data collection methods, the evaluator needs to rate each step in terms of the level of influ
This level of influence can be indexed on a scale from 0 to 3. This would mean that for each given reform and given st

0                                        The PPD has no impact on this step.
1                                        This step benefited from input from the PPD
2                                        The role of the PPD was crucial in the accelerating this step.
3                                        The PPD was solely responsible for this step.



                                         PPD REFORM PROCESS TABLE - IMPACT ON STEPS


                                                       1. Diagnostic
REFORM 1:                                                    2.3
REFORM 2:                                                    0.0
REFORM 3:                                                    0.0
REFORM 4:                                                    0.0
REFORM 5:                                                    0.0
REFORM 6:                                                    0.0
REFORM 7:                                                    0.0
REFORM 8:                                                    0.0
REFORM 9:                                                    0.0
REFORM 10:                                                   0.0

                  1.5                                       2.33




                                                       1. Diagnostic


                                          * PS is able to identify the issue and its
                                                         root causes
PS - Capacity
                                             * PS is able to articulate issue and
                                                       arouse interest
PS - Confidence     * Confidence to share - trust in fellow PS




                        * Channel exists to raise issue among
                                          PS
PS - Access
                        * Mechanism exists for PS to achieve
                                     consensus

                        * Gov has access to relevant data and
Gov- Capacity                        knowledge
                              * Resources are available


                    * Issue strongly affects the Government
Gov- Willingenss
                         * PS lobbying has taken place




Gov - Opportunity         * Inputs from the PS are available




                    NB: Following the different data collection methods, the evaluator needs to ra
                    This level of influence can be indexed on a scale from 0 to 3. This would mean

                    0
                    1
                    2
                    3
REFORM 1:

                                   1. Diagnostic

PS Capacity                              0
PS Confidence                            3
PS Opportunity                           3
Gov- Capacity                            2
Gov- Willingenss                         3
Gov - Opportunity                        3
SUMMARY                             2.333333333

REFORM 2:
                                   1. Diagnostic

PS Capacity                               0
PS Confidence            0
PS Opportunity           0
Gov- Capacity            0
Gov- Willingenss         0
Gov - Opportunity        0
SUMMARY                  0

REFORM 3:
                    1. Diagnostic
PS Capacity              0
PS Confidence            0
PS Opportunity           0
Gov- Capacity            0
Gov- Willingenss         0
Gov - Opportunity        0
SUMMARY                  0

REFORM 4:
                    1. Diagnostic
PS Capacity              0
PS Confidence            0
PS Opportunity           0
Gov- Capacity            0
Gov- Willingenss         0
Gov - Opportunity        0
SUMMARY                  0

REFORM 5:
                    1. Diagnostic
PS Capacity              0
PS Confidence            0
PS Opportunity           0
Gov- Capacity            0
Gov- Willingenss         0
Gov - Opportunity        0
SUMMARY                  0

REFORM 6:
                    1. Diagnostic
PS Capacity              0
PS Confidence            0
PS Opportunity           0
Gov- Capacity            0
Gov- Willingenss         0
Gov - Opportunity        0
SUMMARY                  0

REFORM 7:
                    1. Diagnostic
PS Capacity              0
PS Confidence            0
PS Opportunity           0
Gov- Capacity            0
Gov- Willingenss         0
Gov - Opportunity        0
SUMMARY                  0

REFORM 8:
                    1. Diagnostic
PS Capacity              0
PS Confidence            0
PS Opportunity           0
Gov- Capacity            0
Gov- Willingenss         0
Gov - Opportunity        0
SUMMARY                  0

REFORM 9:
                    1. Diagnostic
PS Capacity              0
PS Confidence            0
PS Opportunity           0
Gov- Capacity            0
Gov- Willingenss         0
Gov - Opportunity        0
SUMMARY                  0

REFORM 10:
                    1. Diagnostic
PS Capacity              0
PS Confidence            0
PS Opportunity           0
Gov- Capacity            0
Gov- Willingenss         0
Gov - Opportunity        0
SUMMARY                  0
alogue
act on the Reform Process

needs to rate each step in terms of the level of influence, which the PPD had on the step being measured.
would mean that for each given reform and given step the following scores are possible.


 put from the PPD
ucial in the accelerating this step.
nsible for this step.



ABLE - IMPACT ON STEPS

                                                  3. Advocacy and Handover to Public
                     2. Solution Design                         Sector
                             1.2                                  1.7
                             0.0                                  0.0
                             0.0                                  0.0
                             0.0                                  0.0
                             0.0                                  0.0
                             0.0                                  0.0
                             0.0                                  0.0
                             0.0                                  0.0
                             0.0                                  0.0
                             0.0                                  0.0

                            1.17                                 1.67




                                                  3. Advocacy and Handover to Public
                     2. Solution Design
                                                                Sector

              * PS has capacity to research and
                            analyze               * PS is capable of preparing, present
               * PS is able to access necessary      and advocate to Public Sector
                           expertise
              * PS is confident to present and
                      support the issue             * PS is confident to engage in
              * PS does not feel threatened by discussions with government officials at
                 other PS and Government            ministerial and cabinet levels
            * Issue not to politically contentious


                                                    * PS has access to Governement to
               * Access to concerted solution
                                                   comment, amend and initial draft laws
                  design with Government


               * Gov has access to relevant
                                                   * Gov capacity to engage on substance
                        expertise
                                                    with the PS on the issues they forward
                 * Resources are available


             * Necessary internal Gov conferral    * Gov willingness to be accountable to
                     has taken place                   PS on the issues they forward



             * Government has the opportunity
                                                     * Gov has mobilized the appropriate
              to mobilize the administration to
                                                   institutional structure to respond to PS
             address the technical specificities
                                                         on the issues they forwarded
                 of issues forwarded by PS


 data collection methods, the evaluator needs to rate each step in terms of the level of influence, which the PPD had on the step b
be indexed on a scale from 0 to 3. This would mean that for each given reform and given step the following scores are possible.

            The PPD has no impact on this step and pre-condition
            The PPD contributed to this step and pre-condition
            The PPD had a significant impact on this step and pre-condition
            The PPD had a crticial impact on this step and pre-condition


                                                   3. Advocacy and Handover to Public
                    2. Solution Design
                                                                 Sector
                            2                                        0
                            3                                        3
                            0                                        2
                            2                                        1
                            0                                        2
                            0                                        2
                       1.166666667                              1.666666667



                                                   3. Advocacy and Handover to Public
                    2. Solution Design
                                                                 Sector
                             0                                     0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0



                     3. Advocacy and Handover to Public
2. Solution Design
                                   Sector
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0



                     3. Advocacy and Handover to Public
2. Solution Design
                                   Sector
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0



                     3. Advocacy and Handover to Public
2. Solution Design
                                   Sector
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0



                     3. Advocacy and Handover to Public
2. Solution Design
                                   Sector
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0



                     3. Advocacy and Handover to Public
2. Solution Design
                                   Sector
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0



                     3. Advocacy and Handover to Public
2. Solution Design
                                   Sector
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0



                     3. Advocacy and Handover to Public
2. Solution Design
                                   Sector
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0



                     3. Advocacy and Handover to Public
2. Solution Design
                                   Sector
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
        0                            0
had on the step being measured.
 s are possible.




               4. Legislative / Executive Process    5. Implementation, M&E, Follow-up
                               1.3                                    0.8                    1.5   1
                               0.0                                    0.0                    0.0   0
                               0.0                                    0.0                    0.0   0
                               0.0                                    0.0                    0.0   0
                               0.0                                    0.0                    0.0   0
                               0.0                                    0.0                    0.0   0
                               0.0                                    0.0                    0.0   0
                               0.0                                    0.0                    0.0   0
                               0.0                                    0.0                    0.0   0
                               0.0                                    0.0                    0.0   0

                              1.33                                   0.83




               4. Legislative / Executive Process    5. Implementation, M&E, Follow-up



                 * PS capacity to provide input to   * PS capacity to monitor, measure and
                   legislative/executive process              analyze the reform
             * PS is confident to engage in discussions
                                                              * PS can access the reform without
             with government officials at parliamentary
                                                                          prejudice
                                 level



                                                           * Government capacity and will to enforce
                * PS has opportunity and access to
                                                            the reform (e.g. no entrenched interests
                  Government to modify draft laws
                                                                            overturn)


              * Capacity and power to put the issue on
                                                       * Capacity to coordinate with other agencies
             the agenda and convince other agencies,
                                                             * Budgets are made available
                  parliament and political parties

                  * Gov willing to risk political capital
                   * Lack of internal vested interests    * Relevant Ministry (staff) has incentives to
             * Alignment between ministries, parliament                   implement
                               and parties

                                                           * Government opportunity to disseminate
                  * Sufficient coordination among
                                                             new implementation procedures to PS
                        Government entities
                                                           * Government opportunity to monitor and
                 * Proposal is consistent with legal
                                                                 evaluate the implementation
                      constitutional demands


of the level of influence, which the PPD had on the step being measured.
eform and given step the following scores are possible.



tion




                4. Legislative / Executive Process           5. Implementation, M&E, Follow-up

                                 0                                             0
                                 0                                             3
                                 2                                             2
                                 3                                             0
                                 2                                             0
                                 1                                             0
                            1.333333333                                   0.833333333



                4. Legislative / Executive Process           5. Implementation, M&E, Follow-up

                                  0                                             0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0



4. Legislative / Executive Process   5. Implementation, M&E, Follow-up
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0



4. Legislative / Executive Process   5. Implementation, M&E, Follow-up
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0



4. Legislative / Executive Process   5. Implementation, M&E, Follow-up
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0



4. Legislative / Executive Process   5. Implementation, M&E, Follow-up
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0



4. Legislative / Executive Process   5. Implementation, M&E, Follow-up
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0



4. Legislative / Executive Process   5. Implementation, M&E, Follow-up
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0



4. Legislative / Executive Process   5. Implementation, M&E, Follow-up
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0



4. Legislative / Executive Process   5. Implementation, M&E, Follow-up
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
                0                                   0
www.publicprivatedialogue.org
Operational Guidelines for the
Charter of Good Practice in Using
Public-Private Dialogue
for Private Sector Development

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:9
posted:10/26/2011
language:English
pages:49