Reflective Evaluation About existing evaluation and self-evaluation

Document Sample
Reflective Evaluation About existing evaluation and self-evaluation Powered By Docstoc
					Reflective Evaluation
About existing evaluation and self-
evaluation instruments and reasons
for inventing something new for CVT




   Irina Michel, Gerald Heidegger,
    Wiebke Petersen
Reflective Evaluation
Transcultural aspects
 Copenhagen Process:
 No „unification“ of systems in VET
  and CVT but subsidiarity principle
  (contrast to Bologna process for
  higher education)
 Reason: VET and CVT systems too
  strongly embedded in cultural and
  economic conditions
 Therefore reduction of the aims to the
  following four fields:
Reflective Evaluation

 With respect to our project:

 European Dimension
 ---- transcultural mutual learning
 Accreditation of non-formal and informal
  learning
 ---- accreditation of CVT which includes non-formal
  learning
 Quality (of processes and outcomes)
 ---- our new tool aims at self reliant quality
  assurance
 Transparency
 ---- the criteria for the self-evaluation should be
  transculturally transparent
Reflective Evaluation
 Transcultural comparison:
Four main types of
- Welfare system
- VET and CVT systems
- Re-Integration schemes

See overhead transparency
Aspects of the history of quality
management and its tranfer to the
educational sector
-   The idea of quality management stems from
    production industry (ISO 9000)

-   During the last decade the idea of quality
    management has been broadly transfered to non-
    profit organisations (EFQM – European Foundation for
    Quality Management)

-   The first non-profit sector that has taken over quality
    management was the care sector – soon stressing
    self-evaluation (because a lack of „hard“ outcomes)

-   Today a lot of public schools go through evaluation
    processes.

-   For disadvantaged young people: Our method QSED
    (Quality through Self-Evaluation and Development)
                                   Output
                                indicators or
                                  personal
                                development

                                common
                            understanding of
                            the phenomenon
                             that should be
                                evaluated


                            Contexts of
                            evaluation
     historical and                                        national annd
        cultural                                           international
   embeddedness of                                          evaluation
   the phenomenon                                           discussions
     of evaluation


From ISO                                                              different
9000 via                                                             evaluation
EFQM to                                                            interests, fears
 QSED                                                                and, targets
                      Figure 4.1: Contexts of evaluation
 Carried out by                                 Carried out by
OECD, CEDEFOP                                   IDEA, IOCE




  Indicator centered                           innovative
                       Evaluation trends       evaluation
      evaluation
                                               approaches



  strict, highly
   structured
                                                   targeting
   procedures                 Measuring soft   empowerment or
                               outcomes
                                                 participation
             Theoretical
             interests in
             evaluation
              and meta-
             evaluation

 Context                     Expertise
   of                           of
evaluatio                    evaluatio
    n        relevant            n
            aspects of
            evaluation

                            Purpose
                               and
Evaluatio                   meaning of
   n                        evaluation
practices
  and
methods        Target/
              Object of
             evaluation
External evaluation
 Advantages                  Disadvantages
 Seemingly „Objective „ ,    To measure „soft“
  Reliable Outcomes            outcomes is very
 Easily transferable          difficult/impossible
  results                     Processes are aiming at
 No self-deception            the targets of the
 No cheating(???)             external evaluation
                              In this way processes are
                               severely narrowed down
                              Practitioners are objects
                               of external powers
                              They become
 Should be combined           disempowered
  with (internal) self-       They become passive
  evaluation
 In order to….               avoid these
           Existing European evaluation approaches:
           example EFQM
                                     company results
                                          (9)
               leadership                                 societal
                   (1)                                 responsibility
                                                       and image (8)

politic and
 strategy
                                                                 staff
    (2)                         criteria of EFQM             contentedness
                                                                  (7)
staff orientation
       (3)                                                 customer
                                                        contentedness
                                                       /satisfaction (6)
                    resources
                       (4)                 processes
                                              (5)
                                                                Figure 6.1: Criteria of EFQM
European standard: the EFQM
Example for the criteria
 Processes
   This criterion is defined as follows: How does the
    institution recognise, identify, carry through tests
    and improve its processes?

   It is divided into five subsections which deal with the
    following questions and for which several starting points are
    given:
   5a ”How are processes identified which are significant for
    the company results?”
   Starting points could be, how the institution ...
   …defines core and support processes (1)
   …identifies core and support processes(2)
   …values repercussion on the business results(3)
European standard: the EFQM
 Using the EFQM as an evaluation tool
   In the first step it takes place through self-evaluation
   The second step includes the coming in of external
    evaluators
 Underlying assumptions and targets of
  evaluation according to EFQM
   “Is the client content with the product which is
    given?”.- Is it the “Customer”, that is the Labour
    office???
 Advantages of the EFQM in comparison
  with other evaluation tools:
 it tries to take care also of soft outcomes
Adaptation of the EFQM model to a re-integration institution for
disadvantaged youngsters



                                           client
                                        orientation
                  customer
                                                            Self
                 orientation
                                                         directing
                                        JAW-stamp
                                         of quality



                       continuous
                                                      Benchmark
                       improveme
                                                         ing
                           nt
     Figure 6.2: JAW-stamp of quality
A Self-Evaluation instrument for
schools: A different idea of quality
 Basic ideas of Q2E : Quality though Evaluation and
  Development („Entwicklung“) with a focus on self-
  evaluation


                       awareness of the                           optimizing
                         own quality                             through error
                          promise                                 recognition
                                           aspects of the
                                            new quality
                                           understanding



                                            client orientation
          Figure 6.3: Aspects of the new
          quality understanding
Example of a criterion in Q2E: “Shaping of teaching and
learning processes (methodical-didactical
arrangements)”
-   to be ticked from 1(weak) to 4 (strong) -


1)The teacher attaches importance to the goal that
   aims and intentions of the lessons are understood
   by the pupils. The learners see the importance of
   learning aims and contents.

2)The teacher arrives at explaining complex learning
   processes and difficult facts in relation to
   experiences and knowledge of learners.

3)The teacher arrives at initiating the pupils´ interest
   for the contents and engaging them for
   participation.
Example of a criterion in Q2E: “Shaping of teaching and
learning processes(methodical-didactical
arrangements)”
-   to be ticked from 1(weak) to 4 (strong) -)


4)Space of active participation and self-directed learning
   of the pupils in the lessons is offered

5)The teacher arrives at making the pupils aware of their
   responsibility for their learning and he supports this
   through adequate measures.

6)The teaching is shaped in a way that pupils could
   create a strong relation between theory and their own
   experiences.
                                          …    basic instrument for
                                             the development of a
                                                 quality model

             …  as basis for a quality                                  ...basic instrument for
               comparison between                                        the development of a
                      schools                                               school specific
                                                                        evaluation instrument

                                         Q2E can be
                                          used as ...
                                                                        …   basic instrument for
                 …as basis for the                                           focus evaluation
                 development of a
                 quality handbook
                                         …help for the development
                                         of a school specific profile
                                               of strengths and
                                                 weaknesses
Figure 6.4: Options for using Q2E
Our new IT-Instrument:
QSED
 Centred on self-evaluation of
  practitioners
 To be adapted to the specific
  circumstances
 Culturally adaptable
 QSED („Quality through Self-Evaluation
  and Development“) for support schemes
  for disadvantaged young people
  should now be adapted to
  CVT
Reflective Evaluation
Target group of the instrument:
Trainers and practitioners for
 People at risk in the labour market:
 - Unemployed, often long term
 - People under threat of being fired
 - Women returners
 - But also: People in conventional
  CVT courses
Some features of the QSED
(„old tool“)
 Six dimensions
 Three levels
 Highly(!!!) interactive IT-Tool
 Adaptation through the users
  themselves
 The users can completely change the
  instrument apart from the basic
  structure (dimensions and levels)
 Next: the dimensions
           Recognition of            Self-Evaluation
      skills/Assessment              and Reflection




                                               Collaborative
Situated
Learning                     QSED                   networks
                                                    of actors




              Funding/
            Administrative          Inclusiveness
              structures                               to the end
Transformation of QSED for the
new target group
 Some dimensions: the same,
  although with new aspects and new
  criteria
 Some dimensions: new, for the
  specific needs of the target group
  (both participants and practitioners)
 Next page: tentative outline of the
  dimensions for the new tool:
„Developing quality in
 CVT“
          Validation and
           accreditation               Self-Evaluation
          of competencies              and Reflection
               New!




 Didactical/                                       Collaborative
methodological              “DQCVT”                     networks
  Concepts                   New!                       of actors
    New!




             Funding/                 Life situation/
           Administrative              Experiences
             structures                   New!             to the end
Transformation of QSED for the
new target group
 Starting from QSED:
 Three levels („Multi-level approach“):
   Explanation: next page
 Macro: „Structural“
 Meso: „Institutional“
 Micro: „Individual“


 Also some other explanations
              Structural-level:
              on this level you find statements which point out power relations in
              society.
              Institutional level:
              On this level you find questions which deal with the activities and the
              processes within Reintegration programmes.
              Individual level:
              On this level you find questions which deal with the actions and
              experiences of practitioners and learners.
              Explanation:
              Here you find further information regarding the statements and
              questions.
              Suggestions:
              Here you can write in suggestions for influencing the societal,
              economic and political conditions.
              Answer and Change:
              Here you can answer or change questions in order to design your
              own tool.
 Back to Index Using this button you get      Back to    Using this button you get
              back to the cluster which       content    back to the content of the
              shows the dimensions.                      dimension.
Back
Developing Quality in CVT
 Now let´s try out and look at the
  „old“ QSED in order to get some ideas
  about the

 New

DQCVT