The implication of Performance Monitoring and Evaluation for

Document Sample
The implication of Performance Monitoring and Evaluation for Powered By Docstoc
					The Presidency, Department
of Performance Monitoring
       and Evaluation
    Dr Solomon T. Bhunu
   Chief Director-Business
        Systems & IT
    SolomonB@po.gov.za
 Content of presentation

1. Background
2. Role of DPME
3.Outcomes Approach
4. Way-Forward




  2
Background

Services not integrated across agencies ( Direct Impact)
    There are no effective systems for coordination,
    Lack of incentives for coordination,
    Competition between agencies e.g Local & District municipalities,
    Lack of rigorous whole-of-government planning



Inappropriate delivery mechanisms in some cases ( Indirect Impact)
    Lack of awareness of different models,
    Weariness of trying different models,
    Inadequate use of new technology,




               3
……background

Inappropriate things are being done ( Direct impact)
    Lack rigorous planning e.g. poor theory of change,
    Lack of focus on key priorities/ drivers of change,
    Lack of feedback on what is working or not.
    Not using services from other stakeholders appropriately,




               4
…….background
Inadequate data to inform management ( Direct Impact)
    Lack of culture of using data to improve performance,
    Suitable data not available




Staff not performing effectively ( Indirect Impact-outcome 12)
    Lack of technical skills in public service
    Incentives for not taking risk does not improve implementation
    Bad implementation of management system



Lack of accountability ( Indirect Impact)




               5
……background
 Initial     Post 1994 massive policy overhaul
 M&E         Gradual awareness of M&E systems
 thoughts    M&E practices inconsistent, quality uneven


•Ensure transparency and accountability
•Promote service delivery improvement
•Ensure compliance with statutory and other
 requirements

•Promote the emergence of a learning
 culture in the public sector
        6
……background
                          Cabinet approved an
    Cabinet          Implementation plan to develop
     Memo
                     an M&E system across the whole
      2005
                              of government

Principles
                  Based on standard,
Monitoring
                  shared indicators - dashboard
Evaluation
Early warning     Implementation plan
Verification      Composite system
Data collection   Extractive
Analysis          Built over time
Reporting         Roll-out
       7
……background

  Period of      Assumption that Departmental M&E
  hiatus          systems will allow for extractive M&E
                 system

   Did not take complexity of inter governmental system
    (sub national) into account
   Ignored extensive monitoring work in Treasury on
    budget performance measures and accountability cycle


   Uninformed position on institutional alignment &
    coordination
   Need specific capacity in place (not recognised)
        8
..background                                          PRESIDENCY

                                                                     PoA                Dev Ind      PoA

                                NATIONAL                                                          OTHER NATIONAL
            DPSA                                                 NATIONAL SECTOR DEPT
                                TREASURY                                                          DEPT


During
                                                           PREMIERS
hiatus                                                     OFFICE
                                  PROVINCIAL
                                  TREASURY

          Human resource info
                                                        Output & outcome info
         ?planned other info    Financial info

                                           PPI info                        Policy outcome info




     PROVINCIAL DEPARTMENT


 •Systems initiated in response to specific needs
 •Hard working, but not smart working systems
 •Smart analysis to align information
          9
…..background

 2007               New direction
  Shift from Task Team to Coordinating Forum, chaired by Pres
     Close cooperation Treasury
     Members: Treasury, DPSA, StatsSA, DPLG(CoGTA),
      PALAMA, AG (Premier‟s Office, DoE)



  Simultaneous creation of two aspects
       Create policy platform that can provide direction to M&E
        environment
       Produce real M&E products for national oversight
           10
The Role of Department of Performance
Monitoring and Evaluation (DPME)
The mandate of the Department of Performance, Monitoring and Evaluation is (DPME)
is derived from Section 85 (2) (c) which states that the President exercises the executive
authority, together with the other members of the Cabinet, by coordinating the functions
of state departments and administrations.
                                   MAJOR MANDATES

       1. Outcomes approach
       To coordinate and facilitate the development and ongoing performance
       monitoring and evaluation (PM&E) of the prioritised whole-of-government
       outcomes
       To ensure that interventions take place where necessary to unblock
       bottlenecks in implementation



       2. Performance monitoring of individual departments
         Carry out PM&E of individual departments for the President; this includes
          assisting the President with the management of performance agreements of
          individual Ministers
                 11
………….the Role of DPME

3. General building of capacity for M&E across spheres, and use of M&E
   (M&E of both policy and implementation)
 Custodian of Government-Wide Monitoring Evaluation System (GWMES)
 Build capacity of M&E units across government
 Ensure that capacity of line functions to carry out M&E is developed across
   government
 Build management culture of using M&E information to effect change
 Use of all types of M&E information to identify and implement key service
   delivery interventions
 Formulate service delivery interventions based on results of all types of M&E
 Ensure that interventions are implemented




              12
……..the Role of DPME
                           Other   MANDATES

 A. Participating in the development of an African M&E network
    Share experiences with other African countries
    Build the network




 B. Advisory support to President, Deputy President and Minister (M&E
    related issues)

 C. Adhoc support to President, Deputy President and Minister (M&E related
 Issues)




             13
 The outcomes approach
     (Results Based Management)




14
Monitoring
     Traditional monitoring                        RBM monitoring adds a
                                                    new emphasis:
Focuses on implementation monitoring:               monitoring the
                                                    outcomes, i.e. the
    Tracking inputs (resources, strategies),       consequence of having
     activities (what actually took place),         delivered outputs
     and outputs (products or services
     produced)
    Monitoring how well a programme,
     project or policy is being implemented
    Assessing compliance with workplace
     and budget
Intent: to inform the efficiency of             RBM monitoring
implementation and to incorporate lessons
learned into decision-making

              15
Evaluation

Formative evaluations are
conducted during the
implementation of an           Are things being
                               done right?
initiative and focus on the
efficiency thereof

Summative evaluations
are conducted after the
implementation of a            Are the right things
                               being done?
developmental initiative and
focus on the effectiveness
thereof

           16
….the Outcomes Approach

     Inputs                Activities              Outputs              Outcomes

                       In put - ou tco me mod el
      Step 4                 Step3                   Step2                 Step1

  How much do              Where should           Which priority      What are the key
   we need to            the system focus        outputs should        outcomes that
 achieve the best           in order to          we measure to          Government
  mix of desired            achieve the           see if we are       wants to achieve
    outcomes                  outputs            achieving each
                                                     outcome
                        M o n i t o r i n g and E v a l u a t i o n
 •Determine              Measure Delivery        Measure outputs       •Derived from
 optimal allocation      of key activities       to test whether       Election
 of inputs                                       we are making         Manifesto
 •Reallocate                                     progress              •MTSF
 Inputs if
    Secret
 necessary
                  17
Outcomes-Logic Model




                                               Manage/
                       Ultimate            influence to
                       Outcomes                 achieve
                                                  these
                       Intermediate
                       Outcomes
                       Immediate
                       Outcomes
                         Outputs                Manage
Responsibility for                             & budget
execution                                      for these
                         Activities
                         Inputs
                                      18
             18
….the Outcomes Approach
Four guiding principles

1. Problem Analysis: a clear understanding of the problem
 assumptions about causes and their relationships to effects are explicit
 ensure that the problem is understood from the point of view of the needs and
concerns of the intended beneficiaries
 analytical tools such as problem trees can assist with problem analysis
2. Theory of Change: a clear understanding of key levers of change
 where do we place our focus, what will have the biggest impact?
 „theory of change‟ based on the best available knowledge about causes and
effects
 state assumptions clearly stated so that we can use evidence from M&E to test it
through experience and
 build reliable knowledge about what works in what circumstances




              19
….the Outcomes Approach
Four guiding principles (continued)
    4. Intervention Logic: chain of logic
    • assumptions about what results must be achieved to achieve the outcome,
    • how they will be achieved and
    • what resources will be necessary.


    5. Clear indicators, baselines & targets: clear basis for monitoring progress
    and evaluating results.
    • indicators for all levels check progress along the whole chain of delivery.
    • each indicator should have a clear baseline, and
    • targets and timelines should be clearly defined
    • indicators must be measurable




              20
….the Outcomes Approach
 The Government 12 Outcomes
 1.  Quality basic education
 2.  A long and healthy life for all South Africans
 3.  All people in South Africa are and feel safe
 4.  Decent employment through inclusive economic growth
 5.  Skilled and capable workforce to support an inclusive growth path
 6.  An efficient, competitive and responsive economic infrastructure network
 7.  Vibrant, equitable, sustainable rural communities contributing towards food
     security for all*
 8. Sustainable human settlements and improved quality of household life
 9. Responsive, accountable, effective and efficient Local Government system
 10. Protect and enhance our environmental assets and natural resources
 11. Create a better South Africa, a better Africa and a better world
 12. An efficient, effective and development oriented public service and an
     empowered, fair and inclusive citizenship**

              21
Development of high level outcomes,        Develop and implement detailed
outputs, activities and metrics            inputs, outputs, activities, metrics
                                           and roles and responsibilities

   Ruling Party election
   Manifesto: 5 priority areas                Establish Implementation Forum

  MTSF: 10 strategic priorities
                                            Negotiate detailed inputs, activities,
                                            metrics and roles and responsibilities
12 strategic outcomes             Step 1
(based on consultation process)   (Done)
                                                 Delivery Agreements                 Step 3
                                                 between                             (July /
    Performance                                  stakeholders                        August
    Agreements with                                                                  2010)
    Minister(s)
    • Based on outcomes                          Coordinate implementation
    • High level outputs,
                                                                                      Step 4
      indicators, targets and                       Monitor and evaluate             ongoing
      activities per outcome
    • Request to work
      together in                 Step 2      Feed back loop to annual
      Implementation Forum        (Done)      revisions of Delivery Agreements
      to produce a Delivery
      Agreement per
      outcome


                22
….the Outcomes Approach
                  GWM&E System: Data terrain
Conceptual
  clarity
                    Evaluations
                                     Social, Economic &
                                     Demographic Statistics



                     Sector
                              Registers and Admin data
         Dept



                                                         PERSAL
        Programme Performance
        Information
                                                         BAS

                                                         DPLG
             Derived information system                  DWAF

         23
…the Outcomes Approach
       GWM&E System: Policy platform

                                   Evaluation
 GWM&E              Evaluations    policy
 framework

                                  Social, Economic &
 Guidelines                       Demographic Statistics



                           Registers and Admin data
          Programme Performance
          Information                           SASQAF

                     PPI                        Common
                     framework                  definitions
              24
…the Outcomes Approach

 Status quo                Policy Platform formalised

Policy platform established 2007
 Pres: GWM&E framework – system goals, describes constituent components
   and institutional location, role of all civil servants, principles
     Guidelines for Premiers Offices
     Guideline on annual M&E plans “ policy vision to operational reality”
 Treasury: Programme Performance Information framework – role of M&E in
   plannning and budgeting, link to dept performance plan, AR
 StatsSA: SA Statistical Quality Assurance framework – evaluation of statistics
   used by dept
 Public Sector Training Agency curriculum - M&E training secure footing
 DPLG(CoGTA): Draft LG framework

 Common definitions document
 Data Forum project
               25
….the Outcomes Approach
        GWM&E : Products / methodologies



                Evaluations

                                 Social, Economic &
                                 Demographic Statistics
                      Development
                      Indicators
                  Sector
       Dept

                        Registers and Admin data
      Programme Performance
      Information
                              PoA

       26
…the Outcomes Approach

   Status quo
                         GWM&E : Products / methodologies
Cross cutting M&E products
 Development Indicators – annual 76 Indicators progress of society
 PoA –priority programmes progress reported to cabinet bi-monthly
 Public Mang Watch – 13x HR indicators, 2 Financial, 1 audit, weighted to provide
   composite index
 Annual performance plans and service delivery targets - 500x indicators -
   monitored quarterly
     IGR publication of non financial information
 OPSC reports – Annual State of the Public Service Report
 AG office – audits of performance information
 „Functionality report‟

 GIS platform (uncoordinated)




                27
…the Outcomes Approach


    PoA
  (Program of Action)


      Annual State of the Nation – priorities
       are announced
      Web based system 300 users
      Bi-monthly report to Cabinet
      Established a culture of reporting



           28
….the Outcomes Approach




29
      29
….the Outcomes Approach




30
      30
….the Outcomes Approach




31
      31
….the Outcomes Approach

                          Conceptualised POA Workflow

               PROVINCIAL                 NATIONAL       PME

                                 P      Depts, Forums,
 Health                                                  POA
                      Concurs    R        Clusters
                                                         trigger
                         &
 Education            updates    E
                       POA       M
                                 I
 Human
 Settlements
                                 E
                                           updates
                                 R          POA
                                                         validation




                      POA
                    updates
                     trigger
                                            Concurs      Publish

               32
….the Outcomes Approach

                       International Comparison

   Political leadership and championing of M&E
   Incentives for M&E
   Top down and bottom up convergence
   Monitoring before evaluation
   Information and data constraints
   Capacity building
   Ownership of M&E system by line ministries




               33
….the Outcomes Approach

                                    Fault lines

 Coordination on M&E design between core national departments
    Different conceptualisation of role of state
    Different paradigms of public sector reform
 Decentralised approach vs national direction on policy monitoring
    Sector experts vs overarching direction
    Naïve view that single decision point in government that will decide on all
     indicators
    Deep dialogue approach




              34
….the Outcomes Approach

                                     Fault Lines
 Frameworks set out requirements and expectations
    … but workplace culture remains stagnant
    Instances of „malicious compliance‟
 Information sharing
    Negative reaction to incorrect data
    Govt buying data that should be available from sister departments
 Evaluation policy lag
    Focus on M&E – emphasis on management information
    Some evaluation, not institutionalised




                35
….the Outcomes Approach
                                  Fault Lines
 Complexity of sub national government
    Policy making separate from budgeting and implementation
    Guidance to Provincial Premier‟s Offices




             36
…the Outcomes Approach

                              Lessons Leant
   Appropriate balance between design and evolution of system
     Original design not adequate




   Coordination essential
     Not all users have same need
     Necessary tension between line ministries specialist view and overarching
       strategic discussion
     Create dialogue, discussion, coordination, good practice




              37
….the Outcomes Approach

                                 Lessons Leant
   Dual implementation strategy
     Good policy platform
     But… also some real products
   M&E is a management system not an IT system
   Change management
     Resistance inevitable
     Takes about 3 budget cycles to bed reform down
     Target appropriate people
   Art & Science
     Pragmatic response that speaks to political, developmental and
        government context



              38
Way-Forward
 PME will work with Various stakeholders in the Establishment of an M&E
  IT Framework

 POA will further be developed into a robust M&E system
  that will be a “system ” of “systems”.

 DPME through various structures e.g. Data forums, M&E networks etc,
  etc, will work closely with all partners in achieving success to M&E
  culture and improved Service delivery




          39
Acknowledgements

1.   SITA...........PoA technical Support

 2. TAU (Technical Assistance Unit).........Results Based Management support

3. Dr Sean Philips......... Director General DPME


 4. Ms Ronette Engela......Deputy Director General DPME




                40
     Ke ya leboga
                Ke a leboha
                           Ke a leboga
                Ngiyabonga
     Ndiyabulela
                Ngiyathokoza
                      Ngiyabonga
                           Inkomu
41        Ndi khou livhuha
                Dankie
                      Thank you

          41

				
DOCUMENT INFO