UNDESA Senior Fellowship Officers DL Presentation v3 by ExQYZ9

VIEWS: 4 PAGES: 15

									17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies


                   Examples of Training Activity Evaluations
                                   including
                     Different Methodological Approaches


                                  London, 17 – 19 November 2008




                                             Presented by:
                                  Dorian LaGuardia, Third Reef Solutions
Overview
How do we measure the real value of fellowships?
How do we adequately capture and measure the impact, outcomes and
results (the value) of fellowship programmes?
       •           Focus needs to be on results; rather than solely on process or “participant
                   focused outcomes,” e.g. satisfaction
       •           Professional development is inherently different than academic learning as it
                   has both individual and systemic implications (and fellowships are relatively
                   unique unto themselves)
       •           Kirkpatrick (a confirmative model—reaction, learning, behavior, results & ROI)
                   and other standard evaluation levels provide a good starting point but we need
                   to develop an evaluation framework from inception that enables more complex
                   and nuanced analysis
       •           A summative model is better aligned with individual and systemic implications
                   (outcome, impact, cost effectiveness, secondary analysis, meta-analysis)
       •           A meta analysis, that combines different data sets and analyses the implications
                   of how these data sets describe different facets of objectives, processes, and
                   outcomes moves toward more valuable methodology

17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Objectives
SMART Objectives are Key

 Crafting specific measurable, achievable, realistic, and time
 based objectives at the outset facilitates matters greatly . . .

            •         Avoid impulse to design “grandiose” objectives that may resonate
                      with stakeholders but that may be difficult to demonstrate, e.g. the
                      fellowship programme will improve development

            •         Good objectives demonstrate valuable outcomes to diverse
                      audiences—making them concrete help

            •         Objectives then form the link to all aspects of monitoring and
                      evaluation—they provide focus, enable prioritization, and enable
                      effective management

                                Analytical                                        Standard           Focus on     Traingulate                Evaluate
       Objectives                                      Diverse Data                                                             Reporting
                               Framework                                          Protocols         Qualitative      Data                   Evaluation



17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Analytical Framework
Develop Analytical Framework that Focuses on Results

 Develop a standard analytical framework that focuses on real
 value. . .
            •         Should be developed at inception
            •         Should be the basis for all analysis, protocols, reporting, etc.
            •         Results Based Management provides a satisfactory model and is
                      being widely adopted in the UN System*
                                                     RBM is a management approach aimed at ensuring that activities
                                                      achieve desired outcomes
                                                     Some Agencies (WHO, UNESCO, WFP) are using frameworks
                                                      broadly aligned with Kirkpatrick, Hamblin, Guskey, et. al. and that
                                                      naturally link with RBM**

                        By combining RBM with standard evaluation levels, one has the
                        basis for effective evaluation


                                 Analytical                                       Standard               Focus on                             Traingulate                                                                   Evaluate
       Objectives                                       Diverse Data                                                                                                                 Reporting
                                Framework                                         Protocols             Qualitative                              Data                                                                      Evaluation

                                                                                                    * A 2006 UNEG survey indicated that 15 of 18 survey UN agencies have introduced RBM while 4 are using RBM as a primary basis for evaluation.

                                                                                                    ** The UN Secretary General’s Bulletin on “Regulations & Rules Governing Programme Planning, the Programme Aspects of the Budget, the
17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies   Monitoring of Implementation, and the Methods of Evaluation,” (ST/SGB/2000/8) states that the objective of evaluation is to measure activities’
                                                                                                    relevance, effectiveness, impact & efficiency.
Example
Sample Analytical Framework



                                                                                                     Relevance




                                                                                                    Effectiveness




                                                                                                       Impact




                                                                                                    Sustainability




                                                                                                     Efficiency &
                                                                                                         Cost
                                                                                                    Effectiveness




17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Methodology
Diverse Data Collection Methods
While surveys will tend to serve as a primary data collection
tool, these should be aggregated as much as possible . . .
                        •         Survey designed for participants
                        •         Survey designed for administrators/officers
                        •         Survey designed for other stakeholders/third parties
                        •         Interviews with participants, relevant
                                  administrators/officers, and other stakeholders
                        •         Focus groups amongst and between cohorts
                        •         Full documentation review—any and all related
                                  documentation can provide a basis for analysis
                       . . .and use analytical framework as a basis for all protocols.
                               Analytical                 Diverse                Standard            Focus on     Traingulate                Evaluate
      Objectives                                                                                                                Reporting
                              Framework                    Data                  Protocols          Qualitative      Data                   Evaluation



17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Standard Protocols
To validate evaluation, questions/protocols should be standardized
Survey protocols should use a mix of question type and a
balance between “ranking” questions and open-ended
qualitative questions . . .
      Need to translate as much good QUALITATIVE data
         into good analytical models as possible
     • Ranking/quantitative questions are important but
         they tend to have inherent and broad variances
     • Ranking Dilemmas: 1-3; 1-4; 1-5; 1-10
     • All should be aligned with analytical framework--
         standardize
     • Interviews can be used to further validate and to
         investigate trends that are appearing in the data
                               Analytical                 Diverse                Standard            Focus on     Traingulate                Evaluate
      Objectives                                                                                                                Reporting
                              Framework                    Data                  Protocols          Qualitative      Data                   Evaluation



17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Focus on Qualitative
Qualitative Data Demonstrates more than Quantitative Data

While ranking/quantitative data sets are typically easy to
organize, some struggle with qualitative data . . .
 •         In fact, qualitative data—what people say and the precise
           words they use to describe various evaluation levels—is far
           clearer than quantitative data:
                      I was lucky enough to work with both a supervisor and a Director I instantly got along
     Positive
                      with and I was able to conduct my work with a great deal of support.
                      It has been difficult to define my position within the [organisation]. The Director was
     Neutral          not keen to give me access to key information. We struggled for awhile before I could
                      get to work effectively.
                      My colleagues did not understand my role because the Fellowship Programme was not
     Negative         clear to them, they did not understand if I was a volunteer, an intern. They treated me
                      quite badly. I wasn’t able to do anything of value.

                                 Analytical                                       Standard           Focus on     Traingulate                Evaluate
       Objectives                                       Diverse Data                                                            Reporting
                                Framework                                         Protocols         Qualitative      Data                   Evaluation




17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Example
Organizing Qualitative Data
By organizing the qualitative data accordingly, clear trends are established . . .




                                                              The number of responses on any topic
                                                              (prevalence)also represent a form of
                                                              analysis.
17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Triangulate Data
Once you have various data sets, you can triangulate data to establish trends,
identify inconsistencies, review the impact/effectiveness of evaluation. . .




                                                                                  Qualitative Data




                                                     Quantitative/Ranking                            Document Due
                                                             Data                                      Diligence




                            . . .and use interviews as a means of exploring inconsistencies
                            and the evaluation’s impact/effectiveness
17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Example
Triangulation: Grandiose Expectations for Generation Y




              While a majority of fellows stated that the posts often did not
              draw on their existing experiences and skills . . .                                    . . .Midterm and Final
                                                                                                    reports demonstrate
                                                                                                    good evidence that
                                                                                                    their posts did draw
                                                                                                    on relevant
                                                                                                    experience and skills.


  ** This may be due to a combination of the fellows’ professional maturity and the traits
  associated with “Generation Y,” a sense of entitlement and self esteem that exceeds actual
  experience and capacity—they think they should be doing more than is reasonable. This is
  supported by the abundance of evidence that demonstrates that relevant skills were
  developed that supported short and long term career prospects. **

17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Reporting
Use Analytical Framework
Once the data has been properly analyzed, analytical
 framework can be used to organize report. . .




May also include a
separate “Data & Analysis”
report that contains
all collected data.




                                Analytical                                        Standard           Focus on     Traingulate                Evaluate
       Objectives                                      Diverse Data                                                             Reporting
                               Framework                                          Protocols         Qualitative      Data                   Evaluation



17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Evaluate Evaluation
What impact did the evaluation have?

Once formal evaluation has bee completed, take some time to
analyze evaluation’s impact . . .
         •         Who learned what when and how did they learn it, e.g.
                   in direct contact with evaluators, from the report, etc.?
         •         What impact did the evaluation have on stakeholders?
         •         What other impact did the evaluation process have?
         •         How can the evaluation/report be used for other
                   activities, e.g. advocacy?



17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
Conclusion

   SMART Objectives provide focus, enable prioritization,
    and enable effective management
   RBM/standard evaluation levels are key for analytical
    framework to be deployed from inception
   Need to translate as much good QUALITATIVE data
    into solid analytical models as possible
   Need to start early and continue well after
    programme’s practical “end”



17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies
About Consultant
Dorian LaGuardia, Director, Third Reef Solutions

With over fifteen years experience in senior management and consulting roles in the financial, electronics, automotive, technology and public service
sectors, Dorian works with organisations in delivering strategic management and organisational development programmes. He brings a unique perspective
as both a “front-line” manager and a theoretician who has developed several leading approaches recognised in the field.

Dorian works with leading clients including Dell Computers, General Motors, UK Inland Revenue, Cabinet Office, Royal Bank of Scotland, Ford Motor
Company, GoldmanSachs, 3M, Granada Learning, Coutts & Co., CitiGroup, Microsoft, Scottish Executive, and the British Council, among others.

Recently, he has built a practice with the United Nations, focused on programme evaluation, management development, and organisational performance.
This includes a range of activities, from results-based workshops to independent evaluations. He has worked with the United Nations Secretariat (UN),
United Nations Food & Agriculture Organisation (FAO), International Labour Organisation (ILO), United Nations Educational, Scientific & Cultural
Organisation (UNESCO), United Nations Children’s Fund (UNICEF), World Food Programme (WFP), United Nations High Commission on Refugees (UNHCR),
Office for the Coordination of Humanitarian Affairs (OCHA), and International Fund for Agricultural Development (IFAD).

Dorian was Chief Development Officer of the software company ITM Solutions Ltd., a “green field” initiative with Dell Computers and Walsh Western
International. Dorian managed operations from inception and built a team of 35 that delivered each objective to time and budget. He oversaw the trade
sale to Dell upon completion of unique outbound logistics software that managed simultaneous delivery networks.

He was Vice President, Organisational Development, at MG Capital LLC, a global private equity firm and subsidiary of GoldmanSachs & Co. that conducts
“roll-ups” of 1st tier suppliers in critical manufacturing supply chains. Dorian was responsible for post-acquisition reorganizations and human resource
strategies in the automotive and electronics industries. He led over half a dozen reorganizations in partnership with Ford, Chrysler, GM, Motorola, and 3M
among others.

Dorian has also founded and run a number of smaller organisations, including a company that pioneered on-line consumer behaviour modelling, a Fair
Trade global trading platform for indigenous craftwork, and a development programme for Afridi & Angell, one of the largest law firms in the Middle East.

He is an active member of the Academy of Management and the European Group for Organisational Studies. He has written broadly on the subject of
organisational development and change management including a recent article in T+D Magazine on organisational culture.

He has a BA from the University of California, Berkeley, and a MA from Harvard University. A native English speaker, he has a working knowledge of
Arabic, French, and Italian.




17th Meeting of Senior Fellowship Officers of the United Nations System and Host Country Agencies

								
To top