2009 570 Lee DR eformation by cx89mEO

VIEWS: 0 PAGES: 17

									Reformation of National R&D Program
         Evaluation System



             Woo Chul Chai
    Contents



       I       Overview of S&T in Korea



       II      National R&D Program Evaluation
               System


      III         Concluding Remarks



2
    Overview of S&T in KOREA




3
Changes of S&T Environments in Korea


                1960s           1970s           1980s              1990s            2000s

     Policy      Industry-                                        Technology-
     trend       oriented policy                                  oriented policy

                    Building R&D                                     Enhancing technology
                                           Promoting R&D
                    infrastructure                                   innovation

                GRIs established
       S&T                             Enforcing the national         Increasing efficiency of
                (e.g. KIST)            R&D programs                   R&D investment
     policy     MOST established in                                   (coordination of S&T-
    direction   1967                   Promoting the university       related policies)
                Daedeok Science Town   -based researches              Planning Total Roadmap
                built in 1974          (e.g. SRC, ERC)


                                                Light &           Heavy
    Change in   Primary         Light                             industry         Electronic &
                                                heavy             goods &
    focusing    goods           industry                                           transport
                                                industry          electronic
     industry                   goods                                              products
                                                goods             products

4
Brief History of National R&D Programs


    Year   82    83   84   85   86   87   88   89   90   91    92    93   94   95   96   97   98   99   00   01   02   08

                                                                                    New Growth Engine Program
MOST       National R&D Program

                           MOCI   Industrial Technology Program
                            E MKE
                                MIC      Information & Communication Technology Program
                ’08
MEST                                                                MOH        Health & Medical Technology Program
                                                                     W
                                                    ME         Environmental Technology Program

                                                              MAF         Agricultural Technology Program

                                                              MOCT        Construction & Transportation Technology Program


MOE             Academic Research Promotion Program


5
S&T Development in Korea                                 (Quantitative Growth)



                  0                                      2th
                                                                                    5th
                                                                   6th       6th
                  5                              8th
    World ranks




                  10           12th                                12th      7th   14th
                       14th             16th               15th
                  15          17th             19th
                       21th
                  20

                  25                  27th
                  30
                       2001 2002        2003      2004     2005    2006 2007 2008

                               Science Competitiveness      Technology competitiveness

                                                                  Source : IMD reports
6
    National R&D Program Evaluation System




7
History of National R&D Program Evaluation


     Increasing Needs of Effectiveness of R&D Investment

                                                                                      Enhancement of
                                                          Performance                  Effectiveness,
                                                          Management                    Efficiency &
                                                                                       Accountability
                                                          & Evaluation
                               Performance
                                Evaluation              Transformational Stage
      Survey·
     Analysis·                                        ○ ’08 ~
                              Growing Stage
     Evaluation                                       ○ Law on Performance
                          ○ ’06~’07                   Evaluation and Performance    Effectiveness
    Entering Stage                                    Management of National R&D
                                                                                       of R&D
                          ○ Law on Performance        Program Revised (‘08.2)
○ ’97~’05                 Evaluation and                                             Investment
                          Performance Management      ○ Focus : Performance
○ S&T Framework Law       of National R&D Program     Evaluation & Management for
  Enacted (’01.1)         Enacted (‘05.12)            Enhancing Effectiveness of
                                                      R&D Investment
○ Focus : Appropriateness ○ Focus : Performance
  of Input and Output     Achievement based on the
8                         Results of R&D Activities
Architecture of Program Evaluation in
NES
                                                        NSTC
                                       Review of Plan           Reports of Results

              Specific                                                                                Meta
             Evaluation
                                                        MOSF                                        Evaluation
                                      Basic Plan for Performance Evaluation (5 yr)
                                      Action Plan for R&D Evaluation (annually)



                                                                                         To evaluate the appropriateness
       In-depth evaluation on major                                                      of Self-Evaluation
        national R&D programs                                                            To review evaluation procedure
                                                                                          and methods of Self-Evaluation
       Long-term/large-scale             Evaluation            Evaluation
        programs                          Guideline             Results
       Joint program among
        ministries
       Programs which need to                                                                         Self
        remove redundancy and                        Ministries                                     Evaluation
        require connectivity                 Planning Self-Evaluation
       Programs at national issue            according to MOSF’s guideline

       Mainly examined by the               Implementing Self-Evaluation
                                                                                             Annually practiced by
        Evaluation Committee                                                                  Ministries
                                                                                             Implementation based on
                                                                                              self-made performance
9                                                                                             indicators and methods
Lifecycle of National R&D Program Evaluation



Evaluation
  Range




                                              Specific (In-Depth) Eval
                                   Self-                                  Self-    Self-




                                                                                                    Follow-up Eval
                                   Eval                                   Eval     Eval
             Feasibility
              Analysis




                                   Meta-                                 Meta-    Meta-
                                   Eval                                  Eval     Eval


                                                                                           Closin
                           Start    + 1year                              +3year   +3year                             Time
                                                                                             g
                           year
             Ex-                                                                            year
                                                                                                    Ex-
             ante                                    Implementation                                 Post




10
Self/Meta-Evaluation of National R&D Programs (1)

     Designed after PART (Program Assessment Rating Tool) of the
      U.S. Federal Government

     Line Ministries assess their own programs every 3 years

     The assessment is based on 25 checklist (questions) for all types
      of programs

     - Types of programs: R&D (Basic, Applied Research,
       Development), Infrastructure Investment, Procurement of Large-
       scale Facilities and Equipment, Human Resource Development

     MOSF reviews the assessment results and reflects them in annual
      draft budgets and the National Fiscal Management Plan



11
Self/Meta-Evaluation of National R&D Programs
  (2)
     Answers to the questions take the form of “Yes (5)” or “No (0)

     - In case of the questions regarding the achievement of

       program goals, 2-scale answers (5, 0) is given.



     A different score is assigned to each question and the result of

     assessment is given by the sum of score

     - Classified as “Effective (95-100),” “Moderately Effective (90-
      94),”Adequate (75-89),” and “Not Effective (0-74).”




12
Evaluation Items and Score (3)

      Section               Evaluation Items                             Evaluation Point               Score

                                                         - Clarity of Goals
                   1. Objectiveness and Feasibility                                                      5
                                                         - Feasibility of Program Contents


     Planning                                            - Appropriateness of Budget Supply
                                                         - Appropriateness of Program Delivery System
                   2. Rationality of Program System                                                      5
                                                         - Role between Program Players and
                                                          Appropriateness of Cooperation System


                   3. Appropriateness of Program         - Appropriateness of Budget Execution
                                                                                                         5
                     Management and Execution            - Appropriateness of Program Schedule

     Execution
                                                         - Appropriateness of Strategy and Plan for
                   4. Appropriateness of Performance
                                                           Achievement of Performance                    5
                     Management
                                                         - Level of Performance Management

       Result      5. Achievement of Performance         - Key Indicator (3)
                                                                                                         65
(Performance)         Standard and Indicator (5)         - General Indicator (2)

     Utilization   6. Utilization of Evaluation Result   -Implementation of Feedback                     15

13
     Concluding Remarks




14
Limitations

     Little enthusiasm from line ministries

     Line ministries did not set up a clear framework of mission and

     strategy

     Performance indicators were not derived from ministerial missions

     in a systematic way

     Assessment of performance relies on subjective assessment by

     outside experts and in-house staff, not systematically utilizing

     indicators

     Cooperation and coordination among players are not sufficiently

15   made
Future Works for Evaluation


       Developing Strategic Performance Management Frame
      - Reviewing Performance Indicators, Monitoring R&D Activities,
     etc.


       Enhancing Education & Consulting for R&D Program Officers
      - Organizing Education Program of Performance Management
      - Developing Logic Model Manual for Various R&D Program Types


       Linking Evaluation Result with Budget Allocation


16
Thank You!!

    Chai, Woo Chul
  wcchai@kistep.re.kr

								
To top