Get documents about "Evaluation of EU and international programmes and initiatives
Shared by: uoy21072
Tags
member states, the european commission, the european union, european commission, framework programme, civil society, higher education, monitoring and evaluation, international education, international initiatives, education and training, science and technology, university of malta, sound management, evaluation society
-
Stats
- views:
- 20
- posted:
- 5/26/2010
- language:
- English
- pages:
- 69
Document Sample
Evaluation of EU and international programmes and
initiatives promoting mobility: selected case studies
Wolfgang Hellwig, Uwe Lauterbach, Hermann-Günter Hesse, Sabine Fabriz
In:
Descy, P.; Tessaring, M. (eds)
Evaluation of systems and programmes
Third report on vocational training research in Europe: background report.
Luxembourg: Office for Official Publications of the European Communities, 2004
(Cedefop Reference series, 57)
Reproduction is authorised provided the source is acknowledged
Additional information on Cedefop’s research reports can be found on:
http://www.trainingvillage.gr/etv/Projects_Networks/ResearchLab/
For your information:
• the background report to the third report on vocational training research in Europe contains original
contributions from researchers. They are regrouped in three volumes published separately in English only.
A list of contents is on the next page.
• A synthesis report based on these contributions and with additional research findings is being published in
English, French and German.
Bibliographical reference of the English version:
Descy, P.; Tessaring, M. Evaluation and impact of education and training: the value of learning. Third
report on vocational training research in Europe: synthesis report. Luxembourg: Office for Official
Publications of the European Communities (Cedefop Reference series)
• In addition, an executive summary in all EU languages will be available.
The background and synthesis reports will be available from national EU sales offices or from Cedefop.
For further information contact:
Cedefop, PO Box 22427, GR-55102 Thessaloniki
Tel.: (30)2310 490 111
Fax: (30)2310 490 102
E-mail: info@cedefop.eu.int
Homepage: www.cedefop.eu.int
Interactive website: www.trainingvillage.gr
Contributions to the background report of the third research report
Impact of education and training From project to policy evaluation in vocational
education and training – possible concepts and tools.
Preface Evidence from countries in transition.
Evelyn Viertel, Søren P. Nielsen, David L. Parkes,
The impact of human capital on economic growth: a Søren Poulsen
review
Rob A. Wilson, Geoff Briscoe Look, listen and learn: an international evaluation of
adult learning
Empirical analysis of human capital development and Beatriz Pont and Patrick Werquin
economic growth in European regions
Hiro Izushi, Robert Huggins Measurement and evaluation of competence
Gerald A. Straka
Non-material benefits of education, training and skills
at a macro level An overarching conceptual framework for assessing
Andy Green, John Preston, Lars-Erik Malmberg key competences. Lessons from an interdisciplinary
and policy-oriented approach
Macroeconometric evaluation of active labour-market Dominique Simone Rychen
policy – a case study for Germany
Reinhard Hujer, Marco Caliendo, Christopher Zeiss
Evaluation of systems and
Active policies and measures: impact on integration
and reintegration in the labour market and social life
programmes
Kenneth Walsh and David J. Parsons
Preface
The impact of human capital and human capital
investments on company performance Evidence from Evaluating the impact of reforms of vocational
literature and European survey results education and training: examples of practice
Bo Hansson, Ulf Johanson, Karl-Heinz Leitner Mike Coles
The benefits of education, training and skills from an Evaluating systems’ reform in vocational education
individual life-course perspective with a particular and training. Learning from Danish and Dutch cases
focus on life-course and biographical research Loek Nieuwenhuis, Hanne Shapiro
Maren Heise, Wolfgang Meyer
Evaluation of EU and international programmes and
initiatives promoting mobility – selected case studies
Wolfgang Hellwig, Uwe Lauterbach,
The foundations of evaluation and Hermann-Günter Hesse, Sabine Fabriz
impact research
Consultancy for free? Evaluation practice in the
Preface European Union and central and eastern Europe
Findings from selected EU programmes
Philosophies and types of evaluation research Bernd Baumgartl, Olga Strietska-Ilina,
Elliot Stern Gerhard Schaumberger
Developing standards to evaluate vocational education Quasi-market reforms in employment and training
and training programmes services: first experiences and evaluation results
Wolfgang Beywl; Sandra Speer Ludo Struyven, Geert Steurs
Methods and limitations of evaluation and impact Evaluation activities in the European Commission
research Josep Molsosa
Reinhard Hujer, Marco Caliendo, Dubravko Radic
Evaluation of EU and international
programmes and initiatives
promoting mobility: selected case studies
Wolfgang Hellwig, Uwe Lauterbach,
Hermann-Günter Hesse, Sabine Fabriz
Abstract
International exchange and promotion programmes began to become relevant to vocational education
and training after World War II, initially operating at transnational level between nation states. In Europe
the binational or multinational programmes of nation states or specific providers have been gradually
complemented or replaced by those of supranational institutions, such as EU programmes. Programmes
have also been developed by national and international organisations (such as the World Bank, the OECD
and the ILO) and by supranational institutions (e.g. the EU) for vocational training cooperation with
threshold and developing countries.
Those arranging and funding these programmes, particularly at a political level, want programme evalu-
ation to provide feedback and explain and justify public expenditure. They also expect continuous
improvement of programmes through more precisely defined goals.
Formative evaluation requires an approach capable of explaining the origin of desired or undesired
results more so than summative evaluation. At a teaching level, the explanation is usually complex. Only
a sufficiently complex explanatory model can provide valid suggestions for improvement. Consequently,
evaluation must build on a design that takes potential success factors into account. However, an evalua-
tion design must not only fulfil the criteria of complexity and precision, but it must also be acceptable to
trainees and project organisers and economically viable. Because of the high costs involved, it is rare for
an evaluation to be carried out to a high scientific standard. It is important, however, to reach an accept-
able compromise. To help practitioners decide on the appropriate approach and on what are relevant
results, a description of evaluation tools and their relative advantages and disadvantages are included
here.
After an introduction focused on evaluation standards we selected four examples: Leonardo da Vinci
Programme of the EC, the Community Study Visits Programme of the European Commission, the
German-French Youth Foundation and the International Exchange Programme for VET Specialists of the
German Federal Government. A summary and recommendations for future practice conclude this study.
Table of contents
1. Premises and objectives of the study 97
1.1. Scope of study and evaluation 97
1.2. Reasons for the selection of case studies and their subdivision 97
1.2.1. Promotion of mobility 97
1.2.2. Programme goals and providers 98
1.2.3. Programme participants 98
1.2.4. Programme structures and programme organisation 98
1.2.5. Programme providers 98
1.2.6. Selection of case studies and evaluation research 99
1.2.7. Cases chosen 99
1.2.8. Structure of case studies 100
1.3. Evaluation standards and appraisal of EU, transnational and international programmes 100
1.3.1. Significance of evaluation standards for the appraisal of EU programmes 100
1.3.2. Criteria for programme evaluation 101
2. Case study 1: the LdV programme (mobility) 106
2.1. General framework 106
2.2. Programme objectives 107
2.3. Evaluation of the overall programme, individual programmes, particular aspects, etc. 108
2.3.1. Overall programme 108
2.3.1.1. Implementation of the programme in the EU 108
2.3.1.2. Implementation of the programme in Germany 110
2.3.2. Evaluation of individual measures and particular aspects, in Germany and elsewhere 113
2.3.2.1. LdV evaluation practice at the ZAV 113
2.3.2.2. LdV evaluation practice at InWent 115
2.4. Summary 115
3. Case study 2: European Community Study Visits programme for those responsible
for Vocational Training (CSV) 118
3.1. General framework 118
3.2. Programme objectives 118
3.2.1. Programme participants 119
3.2.2. Programme procedure and content 119
3.2.3. Peculiarities, reporting 120
3.3. Evaluation of the programme as a whole, individual programmes, particular aspects, etc. 120
3.3.1. Programme dimension, evaluation stakes and documentation of findings 120
3.3.2. Scope of the evaluation and focal points 121
3.3.3. Evaluation material, data, procedures and costs 122
3.3.4. Examples of evaluation 122
3.3.4.1. National evaluation (Germany 1995-98) 122
3.3.4.2. Overall evaluation of CSV 127
3.3.4.3. Measures to support (continuing) evaluation 128
3.4. Summary 128
4. Case study 3: the German-French Youth Foundation (DFJW) 130
4.1. General framework 130
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 95
4.2. Programme objectives 130
4.3. Evaluation of the overall programme, individual measures, particular aspects, etc. 131
4.3.1. Internal evaluation analysis 132
4.3.1.1. Intercultural relations and intercultural learning study conference 132
4.3.1.2. Acquiring skills for Europe 133
4.3.2. Future planning 134
4.4. Summary 134
5. Case study 4: International Exchange Programme for VET Specialists (IFKA) 136
5.1. General framework 136
5.2. Programme goals 137
5.2.1. Objectives 137
5.2.2. Participants 137
5.2.3. Use of electronic media for preparation, implementation, dissemination and lasting effect 137
5.2.4. Partner countries 137
5.3. Evaluation of the programme as a whole, individual programmes, particular aspects, etc. 138
5.3.1. Principles and focuses 138
5.3.2. Chronology of the German Institute for International Educational Research (DIPF)
evaluation of IFKA 138
5.3.3. Interdisciplinary team of evaluators 146
5.3.4. Dissemination of findings 146
5.4. Summary 146
6. Summary and recommendations for future practice 148
6.1. Four practical models 148
6.2. Implementation of evaluation and practical examples 148
6.2.1. Programme organisation, evaluation planning and preparation 149
6.2.2. Programme goals, development of suitable methods and tools for evaluation 150
6.2.3. Quality of evaluation, optimisation potential 150
6.2.4. Transformation of evaluation results and financial framework 152
6.3. Prospects for a new evaluation culture and 12 proposals for the implementation of evaluation 153
List of abbreviations 155
References and bibliography 157
List of tables and figures
Tables
Table 1: Applications for LdV mobility projects: target groups and implementation authority 113
Table 2: Raster for assessment of the most important project phases of LdV mobility projects
from the point of the participants 114
Table 3: The CSV: figures for Germany 2000-02 (incoming, outgoing) 119
Table 4: Evaluation of the CSV referred to Germany 1995-98: research tools and data sets 123
Table 5: Checklist for assessment of participants’ reports 124
Table 6: 1979-83: documentation of findings and formative evaluation 138
Table 7: 1983-94: formative evaluation and documentation of findings 139
Table 8: 1983-96: summative evaluation, added value of IFKA, formative proposals for
programme redesign 140
Table 9: 1995-2001: annual formative evaluation 145
Table 10: Mobility in VET – one of many ways of moving towards the principal objective 148
Table 11: Mobility in VET – two concrete measures 148
Figures
Figure Evaluation structures of the LdV mobility programme
1: 117
Figure 2001 CSV evaluation structure before introduction of new evaluation practices in 2002-03
2: 128
Figure DFJW evaluation structures
3: 134
Figure Structural analysis of evaluation (survey of participants) 1984-96
4: 142
Figure Schematic model of ‘programme success’ factors, taken from surveys of participants
5:
and assessment of participants’ reports 144
Figure 6: IFKA evaluation structures 147
1. Premises and objectives of the study
1.1. Scope of study and rare for an evaluation to be carried out to a high
scientific standard (control group design, reliable
evaluation and construct-valid data capture, consideration
of many factors), because of the high costs
The tradition of international exchange and
involved. What is more important is reaching an
promotion programmes goes back more than a
acceptable compromise. To make practitioners
century. Their particular relevance to vocational
aware of this and enable them to make an
education and training (VET) began after World
War II. Initially they functioned at transnational informed decision on the approach to take and
level between nation-states. what results are relevant to their individual
The binational or multinational programmes of context, we consider it worthwhile to offer a
nation-states or specific providers (such as foun- description of evaluation tools, including their
dations, national and multinational societal insti- relative advantages and disadvantages.
tutions, etc.) have gradually been complemented
or replaced by those of supranational institutions,
such as European Union (EU) programmes within 1.2. Reasons for the selection of
Europe. case studies and their
The programmes mentioned above relate to
industrialised countries (including those outside
subdivision
Europe) and the geopolitical area of Europe in its
broadest sense. Programmes have also been
1.2.1. Promotion of mobility
developed by national and international organisa-
Examples of evaluation practices for mobility
tions (such as the World Bank, the OECD and the
programmes are examined at the following levels:
ILO) and by supranational institutions (e.g. the
(a) at an EU level, which programme organisers
EU) for vocational training cooperation with
expect to be directly beneficial to fostering
threshold and developing countries.
the European dimension of VET. Not only
Those arranging and funding the programmes,
supranational programmes promote interna-
particularly at a political level, want programme
evaluation to provide feedback and explain and tional skills, such as mobility and intercultural
justify public expenditure. They also expect competence or the international political
continuous improvement of programmes through dimension.
more precisely defined goals. (b) At a national level, mobility programmes with
Formative evaluation requires an approach a binational or international dimension have
capable of explaining the origin of desired or also been developed. Their programme struc-
undesired results even more so than summative tures, which are less complicated than those
evaluation. At a teaching level, the explanation is of the EU, render them entirely appropriate as
usually complex, that is, we may presume that case studies. Furthermore, because of conti-
groups of factors are responsible for a given nuity and long experience, the considerable
result and may replace or compensate for one expertise of those involved in the evaluation
another. Only a sufficiently complex explanatory of programmes and projects (measures) can
model can provide valid suggestions for improve- be exploited.
ment. For this reason, evaluation must build on a The four case studies from the above geopolit-
design that takes potential success factors into ical areas, contrasted with an ‘ideal evaluation’,
account. However, an evaluation design must not will constitute the basis for a scientific conclusion
only fulfil the criteria of complexity and precision, to this study. We shall focus on applicability, in
but must also be acceptable to trainees and order to offer EU-programme organisers pointers
project organisers and economically viable. It is for their own programmes and projects.
98 Evaluation of systems and programmes
1.2.2. Programme goals and providers are any number of special cases within EU and
In order to identify potential case studies more international programmes.
accurately in the complex field of the EU and
international, binational and national VET 1.2.3. Programme participants
providers, we need to distinguish between Core VET participants comprise:
measures which: (a) partners and social partners in enterprises,
(a) aim directly at promoting development or business and educational establishments;
(re)structuring of VET in a State or region, etc. (b) apprentices/trainees/students/interns and
Typical programmes are those of the Euro- teaching personnel (teachers, trainers, etc.);
pean Social Fund (ESF), development coop- (c) the framework designers and administrators
eration and World Bank projects in transi- such as policy-makers, educational adminis-
tional countries; trators, providers bodies and other institu-
(b) intend to generate an indirect impact on the tions;
development of VET by creating ‘special (d) researchers and evaluators from academia,
scenarios’ to be used by programme partici- research and consulting.
pants, as in the case of the
Leonardo da Vinci (LdV) ( 1) programmes to 1.2.4. Programme structures and programme
promote vocational mobility in the EU, such organisation
as the Community study visits programme for Programme organisation creates a range of
those responsible for vocational training programme structures. These are characterised
(CSV) and national programmes to promote by such features as:
international training of VET staff, such as the (a) programme duration (short, medium or long
international exchange programme for VET term);
specialists (IFKA) in Germany; (b) individual and group programmes;
(c) set out to generate an indirect impact on (c) homogeneity, or the lack of it, in certain char-
binational relations and promote ‘eurocapa- acteristics;
bility’ and international skills, such as the (d) reciprocity, or the lack of it, in programmes;
German-French Youth Foundation, Deutsch- (e) direct incorporation into host country institu-
Französisches-Jugendwerk, DFJW) and tions (e.g. company or school placements,
support these programme goals with a projects, higher education) or ‘external’
number of measures from all spheres of life. observation, analysis, visits and counselling.
These give particular importance to voca-
tional training for the target group. 1.2.5. Programme providers
In addition to the character-forming of Case studies included here are not only deter-
programme participants from VET, the transfer of mined by the above criteria mentioned in
‘best practice’ or ‘good practice’ and the lending, Sections 1.2.1 to 1.2.4 but also by access to the
borrowing and adaptation of these to particular programme providers, the programme organisers.
scenarios play a significant role in the indirect (b The remarkable significance of the LdV
and c) and direct (a) promotion of VET (Lauter- programme requires the identification of the
bach, 2003, p. 108 et seq.). organisers of its projects. Chief among these in
This brief excursion into possible programme Germany is InWent (2) which, for more than
aims and structures, dissemination, sustainability 20 years, has been the principal provider of initial
and evaluation of results already shows that there training schemes abroad. The LdV experience of
(1) Since CSV falls within LdV, reference should be made to the EU Community action programme on vocational training, LdV
phase II, 2000-06. In this the European Commission lays down such matters as the content of mobility schemes: ‘Promotion
of cross-border mobility for people in vocational training, particularly young people and VET decision-makers’ (Mobility);
L146/40, 11 June 1999 (More information on LdV in Chapter 2).
(2) InWEnt – Internationale Weiterbildung und Entwicklung gemeinnützige GmbH (Capacity Building International, Germany) is an
organisation for international human resources development, advanced training and dialogue. It was established in November
2002 through a merger of Carl Duisberg Gesellschaft e.V. and the Deutsche Stiftung für Internationale Entwicklung (German
Foundation for International Development).
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 99
the German academic exchange service promoting international understanding. Ulti-
(Deutscher Akademischer Austauschdienst, mately, these underlie all the case studies.
DAAD) and the Central placement office (Zentral- Since LdV provides the framework for many
stelle für Arbeitsvermittlung, ZAV which is respon- measures (3), almost the entire range of criteria
sible for placement of young employees should such as ‘programme participants’, ‘programme
also be incorporated in the study. CSV, organised structures and organisations’ and ‘programme
centrally by the European Centre for the Develop- providers’ are covered. With regard to the crite-
ment of Vocational Training (Cedefop), is also rion ‘case studies and evaluation research’, we
administered in Germany by InWent. A wide shall need to verify whether the ‘mobility’ factor
range of programme evaluation literature on can be independently evaluated within LdV, or
DFJW schemes is available. whether some other path might be followed to
carry out evaluations within this complex
1.2.6. Selection of case studies and programme structure.
evaluation research Case studies 2, 3 and 4 are characterised by
Selection of case studies, based on modern eval- clearly defined goals. Programme goals corre-
uation research, should pay particular attention to spond to criterion ‘case studies and evaluation
certain key topics. Goal definition and coherence research’. Like case study 1, they relate to indi-
of project design must be verified. We should rect effects in the promotion of VET mobility
particularly guard against aiming to meet too (Section 1.2.2, b and c). They differ in programme
providers and the composition of the individual
many goals at too high a level in too short a time.
measures (mono-national, binational or multina-
Secondly, the goals chosen should be subdivided
tional) in programme structures and programme
into interim objectives to be attained in ascending
organisation.
order of difficulty, reflecting participants’ initial
Access for our research team programme
circumstances.
providers and the various levels of evaluation
hitherto completed, were also key factors in the
1.2.7. Cases chosen
choice of case studies, which differ in their target
The four cases selected were:
groups but all concern VET and mobility.
Case study 1: LdV action programme: mobility;
A complete evaluation of case study 1 is virtu-
Case study 2: CSV – Community study visits
ally impossible. (In this case evaluation is limited
programme (henceforth part LdV);
by the budget available, notwithstanding expedi-
Case study 3: the German-French Youth Foun-
ency concerns over its usefulness.)
dation (Deutsch-Französisches- Case study 2 offers a plethora of evaluation
Jugendwerk, DFJW); results but these differ greatly in quality. The
Case study 4: the international exchange development of a culture of continuing evaluation
programme for VET specialists has received particular attention since 2000 as
(IFKA). the framework has expanded.
They represent different types of programme, Case study 3 demonstrates broad experience
each corresponding to a greater or lesser extent of evaluation, particularly in the general context
to the criteria given in Sections 1.2.1 to 1.2.6. All of binational exchanges. This basis retains its
fulfil criterion ‘mobility’ (Section 1.2.1). The validity in the particular case of VET.
promotion of vocational training mobility is the Case study 3 is of particular interest to our
key programme goal. All possible aspects of indi- project for another reason. The entire DFJW does
vidual promotion and ‘system mobility’ in VET are not attain the dimensions of LdV (case 1) but, as
incorporated in case study 1. In case studies 2 a meta-programme, its structure is similar.
and 4 individual involvement is foremost, but the Case study 3 shows how evaluations of frame-
promotion of system and project mobility is also a work programmes containing many subpro-
necessary consequence. Case study 3 relates the grammes, individual measures, particular
issue of individual mobility to the political goals of aspects, etc., may be conducted to good effect.
(3) See Chapter 2 for programme description.
100 Evaluation of systems and programmes
Just as cases 1 and 3 resemble one another, greater the need for it to be guided by and linked
so do 2 and 4 (4). The aims and structures of the to scientific research methods. In contrast to
individual programmes, containing many annual surveys, if change is to be scientifically based, it
activities, are similar. The above-mentioned differ- is not sufficient for it to deal with obvious find-
ence between mono and multinational levels ings. The knowledge required to make changes
does not in fact influence evaluation design. Case requires research on implementation, intervention
study 4 was included because it represents more and dissemination. This must systematically
than two decades of evaluation culture, which investigate how programmes fare in practice and
enables it to apply complex social science the extent to which theoretically expected effects
processes, such as path analysis, as tools of may be empirically confirmed under quasi-exper-
evaluation. This continuity has also led to the imental controlled conditions.
incorporation of programme evaluation in In the light of recent trends in evaluation
programme planning and operations. The research and with particular regard to the debate
dissemination of evaluation results (publications, on the international educational achievement
events) related to programme findings was assessments TIMSS, PISA, IALS and the new
another factor in the choice of case study 4. DESI (5), we take evaluation to be a scientific
endeavour in which premises, processes and
1.2.8. Structure of case studies effects of programmes are considered and
The same basic form of reporting was employed compared with stated goals and evaluation and
for each case study: research are explicitly linked. Evaluation is as
(a) programme objectives; much subject to the methodological standards of
(b) evaluation of the programme as a whole, of scientific activity as is research.
individual measures, particular aspects, etc.; Evaluation models should take the theoretical
(c) summary; and methodological state of knowledge of rele-
(d) literature, sources, etc. vant disciplines as a starting point and make a
Subsections to clarify certain aspects are general contribution to research beyond the
included where necessary as are descriptions of immediate scope of their own concerns, so as to
the evaluation measures in Chapters 2 to 4. participate in the general future development of
evaluation standards. Unlike research, evaluation
projects serve as bases for institutional decisions.
They are oriented to the questions, goals and
1.3. Evaluation standards and
appraisal criteria of stakeholders and should be
appraisal of EU, of use to them. Therefore, the stipulations we
transnational and adopted were:
international programmes (a) evaluations should be scientifically based
empirical undertakings that set out
hypotheses and appraise the design, imple-
1.3.1.Significance of evaluation standards for mentation, organisation and effects of VET
the appraisal of EU programmes programmes;
The more an evaluation is intended to generate (b) evaluations should be planned to provide
knowledge that can be used for intervention to information and make recommendations on
improve a programme and take stock of it, the interventions to improve programmes.
(4) There are, however, some differences between these cases, in particular as regards the first and the last point of the IFKA
programme objectives. Furthermore, the duration is 8-14 days, as opposed to 3-5 days. The duration may also have an impact
on the objectives, which are: acquisition of additional specialist or vocational skills; improvement of intercultural skills; promo-
tion of international mobility and innovative capacities; creation of national and international networks; transfer of modern
system components; international cooperation in education and trade; international marketing of German continuing training
products.
(5) TIMSS: third international mathematics and science study; PISA: programme for international student assessment; IALS: inter-
national adult literacy survey; DESI: Deutsch-Englisch-Schülerleistungen-International (German-English international student
performance).
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 101
Evaluation projects are linked to quality. The between evaluation and research have shifted
question of what precisely constitutes quality is, dramatically. A positivist and ideals-based
however, often hard to answer. Admittedly, an conception of research used to dominate basic
evaluation project takes its lead from aims and social science research. This advocated the merit
appraisal criteria put forward by those involved criteria of reliability, validity, objectivity and
but these must first be ascertained, exposed and, internal validity and aimed to draw conclusions
if need be, argued over at an early stage. Quality on causal relationships between variables. These
is based on normative criteria: the prescribed merit criteria, although not explicitly canonised,
functions of training processes, learning goals or still play a de facto role today. They have,
guiding principles of ‘good practice’. Such criteria however, only rarely been attained in evaluation
may undergo various forms of legitimisation: they projects resulting in a split between research and
may be determined by social norms, implicit or evaluation (e.g. Wottawa and Thierau, 1998). In
explicit political or administrative requirements, the context of evaluation, the matter of standards
pedagogical considerations, the motives of the now receives more thought, indicating the
programme organisers or users’ expectations. specific problems associated with the attempt to
Quality criteria should be made accessible to comply with traditional research standards. The
rational discussion through operational practices, rise of constructivist approaches, combined with
rooted in theory and empirical testing of viability action research considerations, has led to an
and implementation requirements. Training emphasis on the merit criteria of communication,
processes have multiple aims distributed simulta- intervention, transparency and relevance in evalu-
neously across various levels (individual, institu- ation practice. The distance between the evalu-
tional and system goals) and may be variously ator and the matter evaluated, formerly consid-
conceptualised and weighted according to the ered essential to objectivity, was abandoned and
perspectives of the different groups involved. evaluators themselves have become tools in the
Possibly, a given aim may be realised by various evaluation process. Construction, optimisation
means and very different goals with varying and legitimisation of programmes and measures
results may be attained by the same method. have become key evaluation aims and the
This generates conflicts of interest when an presentation of alternatives has been stressed
option has a negative effect on one aim and a (Stockmann, 2000; Widmer, 2000).
positive on another. Multiple criteria, multiple In the current debate a rapprochement
levels and multiple perspectives call for complex between the different trends now seems
theoretical approaches, empirical designs and apparent. The division has given way to a combi-
research methods. It is, therefore, worthwhile nation of evaluation and research which stresses
examining not only attainment of goals, but also their respective characteristics. This synthesis is
the aims that are explicitly and implicitly set. apparent in the standards of the Joint committee
on standards for educational evaluation
1.3.2. Criteria for programme evaluation (JCSEE, 1994; 2000) (6). Although the JCSEE
To judge the efficacy of international and EU evaluation proposals were developed for broader,
programme implementation and impact, we need national teaching programmes, most are equally
to examine current evaluation criteria. This is not applicable to the evaluation of local programmes
easily achieved as different attitudes towards and to fields outside education. In their analysis
evaluation research and practice have developed. (in this volume) of ‘ethical and normative stan-
To ascertain which criteria are pertinent to evalu- dards for evaluation practices’, Beywl and Speer
ation projects, discussion needs to be focused on conclude that the standards are interculturally
the relationship between research and evaluation. transferable and applicable to VET.
Evaluation has greatly increased in standing in The standards for evaluation of the JCSEE
the last decade. Attitudes to the relationship postulate that evaluations should be:
(6) The German-speaking world has witnessed publication of a series of works that make reference to the JCSEE standards:
Hager et al. (2000); Posch and Altrichter (1997); Stockmann (2000a); Thonhauser and Patry (1999) and DeGEval (2002).
102 Evaluation of systems and programmes
(a) useful: does a given evaluation deliver prac- different fundamental principles of empirical
tical information on content, timing and research, merely that the methodological details
preparation? differ. Existing sociological methods must simply
(b) feasible and realistically, diplomatically and be applied consistently to evaluation projects.
economically capable of being implemented: Consequently we largely agree with Rost (2000)
is the evaluation planned and conducted in and Rossi et al. (1988).
accordance with real conditions and What are the peculiarities of programme evalu-
economic factors? ation to which sociological methods must be
(c) handled legally and ethically: does the evalu- adapted?
ation guarantee the protection of individual (a) The point of departure for any research is a
rights? question which should ideally be formulated
(d) accurate: are the results ‘technically appro- as a hypothesis. Evaluation, however, does
priate’ and formulated using ‘clean’ methods not examine a theoretical question, but inves-
according to recognised merit criteria? tigates a programme as a product. The
(Stockmann, 2000a). product is the result of a development
These four desirable evaluation characteristics process that is initially conducted without
define groups of more specific standards. regard to its capacity for being evaluated. The
Another project of the third report on VET in product is, therefore, already available and is
Europe will focus on these JCSEE standards in examined for effectiveness, generally in theo-
more detail. For our purposes it should be noted retical terms. This is even true of formative
that the standards serve: evaluations, in which the product is changed
(a) as a basis for critical reflection on one’s own and improved but where the changes serve
activities; optimisation rather than adaptation to the
(b) as an aid in the various phases of evaluation; methodological demands of the rest of the
(c) as a basis with which to justify one’s own research process. In evaluation it is therefore
methods in public; appropriate to work ‘backwards’ and to incor-
(d) as information to stakeholders on what may porate theory into the evaluation process
be expected from an evaluation and its structure.
report. (b) A programme is usually characterised by a
Equally, it is apparent that not all JCSEE stan- high degree of complexity. The analytical
dards may be entirely fulfilled. Often it has even process of isolation or systematic manipula-
been the case that in practice only accuracy tion of variables is thus questionable. The
standards have received appropriate attention isolated contemplation of individual compo-
(Schiffler and Hübner, 2000). This does not mean nents and subsequent investigation of their
that norms in other categories are ignored, but interaction does not do justice to the
their inclusion is often only implicit and they are complexity of the matter to be evaluated. In
seldom reflected in the discussion of the evalua- such cases, methods such as path analysis,
tion. Implicit agreement on certain essential prin- which can track complexity, are preferable (7).
ciples is reflected in the similar formulations of (c) Maintaining disruptive factors at a constant
standards by different institutions. level is rarely practicable, since local condi-
With regard to evaluation quality, central to this tions are so variable. Particular attention must
project, accuracy standards will take priority, as therefore be given to such factors which
the essential criteria for evaluation appraisal may might influence data.
be deduced from them. In deriving merit criteria, (d) Constructs and variables are already in oper-
this group of JCSEE standards is most likely to ation. Theoretical aspects of measurement
profit from research methods on quality assur- must therefore also be constructed ‘back-
ance. wards’.
The thesis can, therefore, be put forward that (e) The principle of randomisation does not
research and evaluation are not based on usually apply. The representativeness of the
(7) See remarks on IFKA in Section 6.2.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 103
sample population and the implications of the (a) Conceptualisation
results are therefore limited. The range of The theoretical bases of measures or pro-
applicability of the results of an evaluation grammes to be evaluated must be recon-
must therefore be borne in mind. structed in line with these considerations:
Despite these differences, structural similarities (i) What is the assumed impact?
exist between basic research and evaluation. (ii) What criteria are to be met?
Although the methodological formation may differ (iii) What is to be improved?
in some respects, the basic intentions remain (iv) Do the goals target specific or general
comparable. effects?
With regard to the three different phases put (v) Are goals precisely defined?
forward by Rossi et al. we can formulate the (vi) Do the goals fulfil European or national
structural similarities between research and eval- needs?
uation as: (vii) Does the choice of target groups comply
(a) conceptualisation phase: corresponding to with programme goals?
generating and incorporating the hypothesis (viii) What does the postulated process that
in basic research; triggers the impact consist of?
(b) implementation phase: corresponding to (ix) In what areas do effects come about?
making variables operational, commissioning (x) Is the concept internally consistent?
a design, ascertaining independent variables, (xi) Can means of optimising the programme
and the ‘manipulation check’, i.e. the test of
be derived from the structure of evalua-
the extent to which a desired treatment is
tion?
realised and can affect participants;
(b) Questions and hypotheses
(c) impact research: corresponding to the various
Conceptual bases must be linked to the
measures to ascertain impact in basic
measures or programmes to be evaluated in
research, e.g. capture of dependent variables,
the form of research questions and
controlling for third variables and statistical
hypotheses which may be empirically verified
checks and calculation of effect size.
with the help of a realisable evaluation design.
In the light of these structural similarities
between research and evaluation, compliance
Implementation phase
with standards does not mean using particular
Implementation is more important in evaluation
methods but showing that suitable methods have
than treatment control in basic research.
been applied in each phase, as follows.
In this phase the stress is not yet on the
Conceptualisation phase impact but on operationalising the active vari-
Every activity to be evaluated must have an ables.
underlying concept. This may have to be recon- The task of an implementation study is to
structed. Empirical practice without theory is ascertain the intensity of an activity and to iden-
inadmissible. Evaluation must also be charac- tify possible disruptive variables which are to be
terised by a ‘theory-driven approach’ (Chen and monitored in an impact study.
Rossi, 1983). Checks on the implementation of an activity
A scientific evaluation should include a theoret- should investigate the following aspects:
ical explanation of relevant processes which are (a) to what extent the realisation of an activity
to be given due consideration during planning of corresponds to its concept,
the study and interpretation of the results. (b) how great is the impact of the influences
A report on a programme’s impact is of less under investigation,
value if the results are not based on a theoretical (c) whether other influences favour or hinder
construct. Because of the frequent lack of potential impact.
systematic variability and the want of representa-
tive population samples, generalisation is only Impact research phase
possible via theoretical models. This phase, often considered the true task of
The criteria for the conceptualisation phase are evaluation, is the one most closely associated
therefore: with basic research.
104 Evaluation of systems and programmes
To consolidate impact, methodological princi- atic comparison of members of different
ples should be adapted to the peculiarities and configurations to arrive at explanatory
limitations of evaluation studies. hypotheses. By the further inclusion of
Treatment of the transferability of results and project participants, the plausibility of the
the causal interpretation of effects are of primary interim hypotheses may be further tested
concern. and modified as need be until a satisfac-
(a) Transferability of results tory ‘saturation’ has been reached. This
If no random sample is feasible, the sample is a financially viable alternative to more
investigated should be comprehensively complicated large-scale projects that are
described in order to clarify general validity. If essentially quasi-experimental from the
necessary, selected groups should be investi- outset.
gated in detail, to support generalisations and In the context of evaluation of microeconomics,
specific findings. Hujer, Caliendo and Radi refer in this report to the
(b) Causal interpretations debate on the varying levels of acceptance
Since randomisation enables some impact accorded to non-experimental methods. They
control of ‘moderator’ variables, any evalua- advocate methods capable of coping with the
tion which does not permit random popula- problems generated by non-experimentally-won
tion sampling should incorporate as many data, such as the before-after estimator, the cross
potential moderator variables as possible to section estimator, the matching estimator, the
ensure causal interpretations. difference-in-difference estimator and the duration
(c) Control groups model approach. All these processes attempt to
Control group design is an indispensable estimate an appropriate comparative standard
procedure in basic research. In evaluation within, or on the basis of the relevant data set.
studies, however, control groups are often Evaluations should, therefore, draw on varia-
hard to create for programme reasons. They tions in impact factors and programme variants
are unrealistic, unavailable or difficult to to strengthen the plausibility and general validity
define. The actual purpose of control group of findings.
design is the creation of a comparative group (a) Statistical checks
to highlight one variation of the independent If potential disruptive and moderator variables
variable. This is a prerequisite for contextual have been included, these can be incorpo-
validity. This aim of creating a comparative rated in the statistical analysis. With the help
standard by which impact can be ascertained of partial correlation, covariant analysis, step-
may also be achieved by: wise regression and linear structure compar-
(i) dosage variation, e.g. duration or inten- ison models, effects can be statistically
sity of the measure to which participants controlled or estimated.
are exposed; (b) Effect size
(ii) facet design: comparison of populations Particularly in evaluation, although not exclu-
exposed to different component combi- sively, mention of statistical significance is
nations of a given measure. This permits either superfluous because of sample size or
statements to be made on the size of self-evident. Nonetheless, the statistical
impact, depending on the effects of significance of ascertained differences and
various components and their interaction; relationships should be cited as evidence of
(iii) comparison of populations which differ in the impact of an activity. Effect size is partic-
theoretically defined characteristics from ularly important. Effect size describes the
the evaluation population. In general, size of the observed effect in proportional
individual programmes and their projects variance, correlation, standardised mean
create different patterns of initial require- deviation or in survey tool measurement
ments and implementation procedures units. These are instruments capable of
due to the given variables. Quasi-experi- quantifying the practical usefulness of
mental designs may be derived by induc- measures (Hager, 2000; Rost, 2000;
tion. The contrast method allows system- Sedlmeier, 1996).
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 105
The above criteria make high demands on programmes setting their own priorities. The
programme evaluation. We cannot expect indi- following case studies should be examined to
vidual programmes to meet all the above see what aims they strive toward and what
criteria. It is more a matter of individual criteria they fulfil.
2. Case study 1: the LdV programme (mobility)
2.1. General framework going training or endeavouring to enter the labour
market, young employed workers or job seekers,
The LdV action programme was launched in 1994. recent graduates, students registered in higher
education establishments. They may also be
As a central vocational training support
aimed at trainers or human resource managers
programme, it was partially or fully responsible for
and training scheme managers, at language
triggering several initiatives within this sector
specialists and at social partners.
(Comett, Eurotecnet, FORCE, PETRA and Lingua).
The basic point here is to strengthen the Euro-
The programme is now in its second phase
pean dimension of initial and continuing voca-
(2000-06). Its priorities were established in a
tional training, to encourage people to gain expe-
Council Decision of 26 April 1999. They focus on
rience in activities involving theory and practice,
further developing the quality, innovativeness and
particularly work-linked training, to develop
European dimension of vocational training
language skills and transnational contacts and
systems by promoting transnational cooperation.
exchanges of good practices for trainers and
The countries eligible to participate in the
human resource managers.
programme are:
This type of measure is also useful in consoli-
(a) the associated countries of Central and
dating transnational cooperation in a broad
Eastern Europe (CEECs), under the European
sense, bringing in all actors on the vocational
Treaty and additional protocols, already or yet
training scene. It is also a good way of strength-
to be concluded, on the participation of these ening the links between working life and training.’
countries in Community programmes; The actual implementation takes place within
(b) Cyprus (its participation is financed by addi- three main areas of action, depending on the
tional funds under provisions also applicable groups of beneficiaries involved.
to countries in the European Free Trade Area (a) Transnational placement projects:
[EFTA] which are members of the European (i) for people in initial vocational training: in a
Economic Area [EEA] ); vocational training institution or enterprise.
(c) Turkey and Malta, under procedures yet to be Objective: promoting the acquisition of
agreed with those countries. complementary vocational skills in another
The measures in the second phase of LdV can Member State to improve access to the
be divided into five separate categories: labour market;
(a) mobility, (ii) for students: in an enterprise. Objective:
(b) pilot projects, including theme-based initia- promoting entry to the labour market. This
tives, group of people can also help enterprises
(c) language skills, gain transnational experience and promote
(d) transnational networks, the application of technology transfer;
(e) comparative materials. (iii) for young employees and recent gradu-
The following comments refer only to the area ates: in a vocational training institution or
of mobility, which covers the type of case studies enterprise.
presented here. (b) Transnational exchange projects: these are
Extract from the general guidelines for appli- intended for trainers, careers advisors and
cants in the area of mobility (8): educational guidance counsellors, etc.
‘Proposals submitted under this heading must (c) Study visits: for vocational training decision-
relate to transnational mobility actions for people makers, including the social partners. Objec-
in training, more especially young people under- tive: to promote the transfer of technological
(8) Not to be confused with the CSV programme for vocational training decision-makers.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 107
innovations in SMEs and to provide contin- (g) encouraging vocational training measures for
uing training for company managers to keep disadvantaged young people lacking adequate
pace with technological and organisational training;
changes; to prepare transnational vocational (h) promoting equality of access to initial and
training campaigns. continuing training for disadvantaged persons;
(i) Placements usually last: (i) promoting equality of opportunity as regards
• from 3 weeks to 9 months for those access for women and men and their effec-
undergoing initial vocational training tive participation in vocational training;
• from 3 to 12 months for students (j) promoting equality of opportunity as regards
• from 2 to 12 months for young access to and effective participation in, voca-
employees and recent graduates tional training for migrant workers and their
(ii) Exchanges last between 1 and 6 weeks. children, and for disabled people;
The exchange and placement mobility (k) promoting cooperation in respect of skill
projects are subject to national procedure A (a requirements and training needs and encour-
single-step procedure). In the case of aging the acquisition and transparency of
Germany, applications are submitted either to qualifications and an understanding of the
the Education for Europe National Agency at key skills relevant to technological develop-
BIBB or to the relevant implementation centre ment;
(InWent, DAAD, Bundesanstalt für Arbeit [BA] (l) promoting vocational training in the light of
or ZAV). The implementation centre checks the results of technological research and
applications and prepares the choice of development programmes;
projects for the selection committee. It also (m) promoting the gradual development of an
advises on all questions relating to project open European vocational training and quali-
application, implementation and supervision. fications area;
(n) supporting activities aimed at developing
linguistic skills as part of vocational training
2.2. Programme objectives measures;
(o) promoting the development of vocational
The programme was primarily introduced to guidance facilities;
promote a ‘Europe of knowledge’ and to consoli- (p) fostering the development of methods of
date European cooperation in general and VET. self-training in the workplace and of open and
The Council Decision of 6 December 1994 on an distance learning and training;
action programme to implement a European (q) encouraging the development and integration
Community vocational training policy formed the of key skills in vocational training measures;
basis for the formulation of objectives. That (r) giving all young people in the Community the
document lists the common goals: opportunity to undergo one or, if possible,
(a) improving the quality and the innovative two or more years of initial vocational training
capacity of Member States’ vocational on top of their compulsory education.
training systems and measures; The second phase of the programme follows
(b) developing the European dimension in voca- three general objectives:
tional training and guidance; (a) improving the skills and competences of
(c) promoting lifelong vocational training and people, especially young people, in initial
supporting associated policies; vocational training at all levels. This may be
(d) encouraging vocational training measures for achieved inter alia through work-linked voca-
unskilled adults; tional training and apprenticeship, with a view
(e) enhancing the status and attractiveness of to facilitating vocational integration and rein-
VET, and fostering equivalence between tegration;
academic and vocational qualifications; (b) improving the quality of, and access to,
(f) promoting vocational training for young continuing vocational training and the lifelong
people and preparing them for adult and acquisition of skills and competences, with a
working life; view to increasing and developing adapt-
108 Evaluation of systems and programmes
ability, particularly in order to consolidate 2.3. Evaluation of the overall
technological and organisational change;
programme, individual
(c) promoting and reinforcing the contribution of
vocational training to the process of innova-
programmes, particular
tion, with a view to improving competitive-
aspects, etc.
ness and entrepreneurship, and also consid-
ering new employment possibilities. Special
2.3.1. Overall programme
attention shall be paid in this respect to
We divide the evaluation of LdV mobility
fostering cooperation between vocational measures into two separate areas: implementa-
training institutions, including universities, tion of the programme in the EU and implementa-
and companies, particularly SMEs. tion of the programme in individual countries (e.g.
The objectives of this programme shall be Germany).
achieved by means of the following measures:
(a) support for the transnational mobility of 2.3.1.1. Implementation of the programme in the
people undergoing vocational training in EU
Europe and of those responsible for training; Objectives of evaluation of the overall LdV
(b) support for pilot projects based on transna- programme, focusing on the effects for the EU:
tional partnerships designed to develop inno- (a) comparing the programme’s priorities and the
vation and improve quality in vocational quantitative data collected;
training; (b) identifying differences in the individual coun-
tries involved in the programme;
(c) promotion of language skills, including less
(c) determining trends in European vocational
widely used and taught languages and under-
training systems;
standing of different cultures in the context of
(d) assessing vocational mobility promotion tools
vocational training;
practised throughout Europe (e.g. Europass);
(d) support for the development of transnational (e) presenting transnational difficulties regarding
cooperation networks facilitating the the programme context (application proce-
exchange of experience and good practice; dure, financing, etc.).
(e) development and updating of Community Evaluation reports:
reference material and comparative data. (a) Commission interim report of 23 July 1997 on
In implementing these actions, the Decision the implementation of the LdV programme
provides for particular support for transnational [COM (97) 399 final, not published in the Offi-
actions to promote and use information and cial Journal]
communication technologies in vocational The report is based on both the results of an
training. objective external evaluation and the partici-
The programme objectives have the following pating countries’ reports on the implementa-
priorities: tion and impact of the LdV programme.
(a) enhancing employability by making VET Extracts:
systems more efficient in Europe; Some general trends emerging from the
national reports are as follows:
(b) promoting partnership between training insti-
(i) concern for employment and employability
tutions at all levels, the business sector and
is becoming increasingly pronounced in
the social partners;
vocational training policies;
(c) facilitating the integration of disadvantaged
(ii) all the reforms undertaken are aimed at
persons in the labour market and countering improving vocational training and guaran-
discrimination; teeing its quality;
(d) encouraging companies to invest in human (iii) vocational training is becoming decen-
resources as a business strategy; tralised and individualised with more
(e) exploiting the potential of information and responsibility on those involved at local
communication technologies; and regional levels and the social part-
(f) improving the transparency of qualifications. ners.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 109
All the Member States stress the importance are being made to respond to the continuing
of measures geared to improving the changes seem promising. By encouraging the
prospects for social and occupational inte- parties concerned to work together, LdV has
gration of vulnerable target groups such as made a significant contribution to harmon-
unskilled young people, young unemployed ising vocational training.
people, the long-term unemployed, people One of the shortcomings of the programme is
with disabilities and immigrants. the poor dissemination of project findings.
The Council Decision estimated that the finan- The European Commission is currently paying
cial resources required at Community level to particular attention to measures aimed at
implement the LdV programme would be improving and disseminating findings at Euro-
ECU 620 million for the period from 1995 to pean level. The complexity of the programme
1999. The budgetary authority allocated an itself and the cumbersome administrative
overall sum of ECU 139.5 million in 1995 and procedures constitute an additional draw-
ECU 154.4 million in 1996. In 1995 and 1996, back. The conclusions drawn from experi-
providing funding for 1 542 projects totalling ences during the first two calls for proposals
ECU 201.8 million, the LdV programme from the national reports and the external
enabled more than 50 000 persons, including evaluation report can only help to improve the
22 000 young people undergoing initial effectiveness of the programme.
training, 10 000 young workers and (b) Commission report of 22 December 2000, on
13 000 higher education students and gradu- the implementation of the first phase of the
ates, to benefit from a European mobility grant Community action programme LdV [COM
for periods of guidance and training in the (2000) 863 final].
Member States and participating countries. The report is based on:
The Commission portrays the impact of the (i) an external evaluation report by the
programme as apparent on different fronts: consultancy firm Deloitte & Touche;
(i) the pilot projects are part of the develop- (ii) national reports from the Member States;
ment of the Member States’ vocational (iii) final reports from the social partners.
training systems and arrangements; the LdV has been invaluable in the promotion of
key players involved in training see the transnational initiatives and the internationali-
programme as an investment priority; sation of best practices in vocational training
(ii) by proposing innovative solutions with in the quality and content of learning, innova-
regard to new qualifications, occupational tion and the addition of a European dimen-
profiles and means of access, rapproche- sion. Its particular strengths have been the
ment of schools and enterprises and new enhancement of mobility and employability of
pathways for occupational integration and the participants in the projects. It has thereby
stabilisation, the programme has a signifi- benefited not only the participants from
cant impact vis-à-vis innovation (European Member States and EFTA-EEA countries, but
laboratory for innovation); also those from the applicant countries where
(iii) the programme is often perceived as an LdV projects have assisted in the preparation
opportunity to amplify an existing project of national training systems for accession.
by fostering transnationalism and These successes are the foundation upon
multi-partner cooperation; which the second phase of the programme is
• the programme makes a major contri- being built.
bution to the development of transna- However, not only the strengths but also the
tional mobility; weaknesses of the first phase must be
• the impact is even greater when the addressed in order to create a solid founda-
project succeeds in developing genuine tion for the second phase of the programme.
training strategies. The problems experienced during implemen-
The impact of the programme will no doubt tation of the first phase relate not only to the
take time to materialise, but the cooperation unsatisfactory performance of the contracted
which has been initiated and the efforts which technical assistance office, but also to
110 Evaluation of systems and programmes
complexities within management at the Commission has paid more attention to the ques-
central level, and the lack of reciprocity with tion of evaluation in LdV II. Another consequence
other training-related programmes. The Euro- is that evaluations have been commissioned at
pean Commission has ensured that the an earlier stage, which has left more time to
lessons learned from these experiences are develop the methodology.
put into practice in the second phase, Methodological approach of the evaluation:
through the simplification of procedures and (a) LdV I, steps in the investigation (Friedrich and
further decentralisation of the programme’s Schumacher, 2000)
management. It also anticipates a clear Performing a brief overview analysis which
dissemination strategy for the products and examines the situation and the current trends
outcomes of the many transnational pilot and problems of the national education
projects in the first phase. system. This analysis provides a frame of
LdV stands to become a key instrument in the reference for assessing the appropriateness
implementation of lifelong learning strategies of the measures implemented and of their
which offer synergies between the European emphases.
policies for education, training and employ- Appraising implementation reports, compendia,
ment. Furthermore, the second phase will application documents, accountability reports
seek to involve some players more fully in from national coordination centres and
transnational vocational training, particularly projects, etc.
the social partners and SMEs. This strategy Drawing up tables and overviews summarising
reflects the European Commission’s policy of
the material and financial progress of each part
promoting reciprocity between related
of the LdV I programme.
programmes, which receives a strong
Systematically recording the projects
emphasis in the second phase.
supported. A survey of around 600 projects
was conducted. The sample reflected the
2.3.1.2. Implementation of the programme in
distribution of projects across the individual
Germany
project fields. The survey identified the central
Objectives of evaluation of the LdV programme,
goals, the dissemination measures under-
focusing on the effects for Germany:
taken and how the project findings could be
(a) identifying the effects and deficits of LdV in
utilised. Over 200 projects responded to the
Germany;
survey. The time restrictions mentioned above
(b) comparing programme objectives with actual
effects; made it impossible to increase the response
(c) summarising the findings from evaluations of rate by sending out reminders. The data of
individual measures or from evaluations of those projects which provided answers were
programme parts conducted by implementa- weighted to relate to the total number of
tion centres (DAAD, InWent, ZAV); projects supported. The findings thus provide
(d) assessing the work of implementation centres; a representative picture of all measures
(e) improving efficiency of LdV measures in supported.
Germany; Analysing the strengths and weaknesses of
(f) submitting proposals to the national agencies the programme by consulting the players
and the implementation centres. involved and independent education experts.
Evaluation practice: The latter can be divided into three cate-
The Institut für Wirtschaft- und Sozialforschung gories: members of the national LdV
(Economic and Social Research Institute, WSF) in committee and subcommittees, members of
Kerpen evaluated the first phase (1995-99) of the project advisory boards or similar project
LdV programme. It is also engaged in the evalua- support bodies and project partners. Around
tion of the second phase. The methodological 80 independent experts provided information
approach has now changed. No time was avail- on the strengths and weaknesses of LdV.
able to conduct thorough surveys for the evalua- In addition, programme coordinators shared
tion of LdV I because the programme was their experiences through structured inter-
approved at the last minute. The European views.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 111
(b) LdV II, method 1995-1999 (Evaluation of implementation of the
In contrast to the evaluation of LdV I, which LdV programme in Germany from 1995 to 1999).
had to be completed within just three months, An enquiry commissioned by the Federal Ministry
it was possible to issue a long-term contract of Education and Research. Kerpen: January
for the evaluation of LdV II. This made it 2000.
possible to adopt a multi-stage methodolog- (a) The key findings were:
ical approach, something that is particularly (i) the European Commission had a major
important for data on the sustainability of influence on the choice of projects;
mobility projects. (ii) in Germany coordination was divided
Examination of the LdV programme draws a among seven centres; BIBB had overall
fundamental distinction between mobility responsibility;
measures and ‘innovative projects’. The latter (iii) 56 % of participants were women;
have particularly required and still require, (iv) social partners, experts and authorities
close cooperation with the National Agency. were adequately represented on commit-
Since the evaluation project was awarded to tees;
WSF, several preparatory meetings have (v) application forms were too long and
taken place (as many as six per year) with the complicated;
National Agency and the implementation (vi) online project applications did not func-
centres. tion properly.
The evaluation of mobility measures is (b) Extracts from the report relating to mobility
primarily based on participant questionnaires. projects:
In the first stage at the end of 2001, WSF sent About 2 500 LdV projects have been imple-
questionnaires to around 1 000 interviewees mented in Germany to date. Pilot projects,
three to six months after they had partici- multiplier projects, surveys and analyses
pated in the LdV exchange programme. This account for approximately 13.7 % (340) of all
first group consisted of school pupils, young projects since LdV started. Placements and
people in initial vocational training, young exchange programmes account for 86.3 %
employees and multipliers. Because of the (around 2 100 projects).
specific length of the programmes students LdV exchange measures have involved
were not involved in the first year. approximately 30 000 people. The figures
The questionnaire was drafted in close coop- showing the percentages of different types of
eration with the implementation centres, participants in LdV exchange and placement
which also use the WSF’s questionnaire programmes are: trainees 42 %, young
model for their surveys. employees 17 %, students/recent graduates
In the second stage at the end of 2002, 30 %, trainers, etc. 10 %. The programmes
1 000 new participants (this time including the have only involved about one thousandth of
students) were surveyed, again three to six the total population of all these groups,
months after completing an exchange except for trainers. Consequently the
programme. The 600 members of the first programme has had no remarkable quantita-
group who had completed questionnaires the tive effects as might be expected given the
first time were also questioned again, so that low investment. However, we should mention
the respondents now totalled 1 600. In the that national bodies provide few opportunities
third year, 1 000 new participants and the for comparable support programmes, so that
people from the first and second groups will LdV has at least represented a useful
be surveyed. enhancement.
In this way the evaluators hope to gain a Project organisers did not conduct systematic
detailed picture of the effects of exchange impact analyses of these placement and
programmes by the end of the study. exchange projects. Furthermore, insufficient
Evaluation report: time and resources were available for the
Friedrich and Schumacher: Evaluierung der intended ex-post evaluation. This was due to
Umsetzung des Programmes LdV in Deutschland budget restrictions imposed by the Commis-
112 Evaluation of systems and programmes
sion. But analyses of this nature are crucial if increasing specialist competence, presenting
we are to identify the conditions under which new techniques and introducing different
these measures are particularly successful. cultures. Each of these goals received 75 points
Nevertheless, the findings of expert surveys on a scale of 0 to 100. Far less significant were
during ex-post evaluations and isolated boosting company competitiveness and, aston-
investigations undertaken independently by ishingly, preparing young people to embark on
the relevant specialist coordination centres careers. Each received 23 points. These assess-
suggest that these measures are the most ments clearly demonstrate the priorities of
effective of all support programmes. short-term exchanges for young people in initial
Target-group specific exchange programmes: vocational training, who overwhelmingly domi-
Most projects have been aimed at young nate this programme area.
people undergoing initial vocational training. In programme area II the main priority was to
Since 1995, 900 measures have been carried out respond to the demand for European mobility
under heading I.1.2.a. This represents 36.5 % of (100 points), followed by improving specialist
all measures implemented in Germany and competence (98 points) and acquiring language
42.3 % of all exchanges. The longer-term skills (93 points). Promoting the appeal of voca-
exchanges offered to the same target group have tional training and general exchange of informa-
played a relatively insignificant role, totalling 64, tion and experience had little or no relevance.
i.e. 2.6 % of all projects, 3 % of all exchange Impact of LdV further training measures:
projects. The measures gave young people a (a) Participants
chance to gain additional vocational skills. There has been no systematic investigation of
Exchanges for young employees have the impact of LdV placement and exchange
consisted of continuing training schemes and projects at participant level (e.g. via surveys).
work placements abroad. Continuing training The intended analysis of this kind could not
courses account for 6.2 % and placements be realised as part of the ex-post evaluation
abroad for 15.3 % of all projects supported since because of time and cost constraints.
1995. However we can affirm that, while the
Transnational placement programmes for exchange programmes do not fully fill a gap
students or recent graduates account for 3.5 % in the German education system, they have
of all projects. The low number of student place- made a significant contribution towards doing
ments should not detract from their significance. so. The findings from the expert survey hint at
This type of project is characterised by the large the same conclusion. The experts believed
number of partners involved and the high number that of all programme components, place-
of participants (almost 9 000 since 1995). The ments and exchanges had had the most
projects have contained bundles of measures impact. These were awarded the very high
involving students at German universities going score of 81 on a scale of 0 to 100.
abroad and foreign students assigned to German Since no participant impact indicators are
companies. available, the following programme impact
All programme areas have featured projects analyses are mainly based on providers’ and
encouraging exchange of trainers. However, only experts’ comments. It should be borne in mind
those incorporated in programme area I are that providers’ remarks are often biased since
significant in both number of projects and they tend to overestimate their own projects,
number of participants, representing 21.4 % of all at least as far as scale of impact is concerned.
projects. Additional exchange projects have (b) Projects
included improvement of language skills (a total The impact of placement and exchange
of six projects) and schemes for educational programmes on project providers is similar to
decision-makers (although no precise figures are that of the pilot schemes. The programmes
available). also achieved a distinct improvement in
The survey of project providers revealed the competence. Characteristic of these projects
objectives of exchanges in programme area I, is that they directly target participants,
which focused on initial vocational training, to be yielding the following effects:
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 113
(i) a more positive attitude towards European (g) no systematic assessment of data (much of
mobility among participants (78 points); this occurring directly).
(ii) improved key skills with particular rele-
vance for the European internal market 2.3.2.1. LdV evaluation practice at the ZAV
(73 points); In Germany applications for LdV mobility projects
(iii) increased transnational experience (72 are handled and approved by different offices,
points); depending on which target group applicants
(iv) improved language training (71 points). belong to. These are:
(c) Programme level The following statements refer to ZAV’s partic-
The impact of the exchanges seems to have ular area of activity, providing work placements
been extraordinarily high (81 index points) but abroad for young employees.
one must remember that the quantitative This is a heterogeneous target group, since
scope was limited from the outset due to the people of all ages can participate in the
low level of funding available for measures of measures. Because ZAV is an institution of the
this nature.
German Federal Labour Office, its implementa-
The programme also helped to reinforce the
tion of LdV projects usually focuses on the labour
important cooperation between schools,
market and worker employability. This distin-
training institutions and enterprises (68
guishes the organisation from the other two
points), a crucial area of practical initial and
implementation centres, InWent and DAAD, which
continuing vocational training.
concentrate more on education, training and
2.3.2. Evaluation of individual measures and pedagogical issues.
particular aspects, in Germany and Two different categories of evaluation exist:
elsewhere (a) Category 1: LdV programme evaluations in
Objectives of the evaluation of measures: Germany
(a) quality assurance, WSF in Kerpen, which carried out the evalu-
(b) provision of support and advice to projects, ation of the first phase, is also evaluating
(c) feedback for evaluators’ own work, the second phase of the LdV programme.
(d) comparison of project objectives with actual Evaluation has involved surveys of the
impact, implementation centres. A report on the first
(e) increased efficiency of individual projects, phase of the LdV programme is available. To
(f) often simply a source of information for the date, however, it has received limited atten-
implementation of the programme in Germany, tion.
Table 1: Applications for LdV mobility projects: target groups and implementation authority
Target group Duration Implementation authority
Young people undergoing initial short: 3-12 weeks InWent
vocational training long: 13 weeks – 9 months
Young employees: 8 weeks – 12 months • ZAV
• in work placements abroad • InWent
• in vocational (school-based) continuing training
Students and graduates on work 3-12 months (students) DAAD
placements (from university/college) 2-12 months (graduates)
Vocational training decision-makers: 1-6 weeks • InWent
• Trainers • DAAD
• Academic and business staff exchanges
114 Evaluation of systems and programmes
(b) Category 2: evaluation of the measures super- evaluations. The implementation centres
vised by the relevant implementation centres exploit the reporting obligation prescribed by
(ZAV; DAAD; InWent) the European Commission. Participants and
ZAV pays greatest attention to the evaluation providers have submitted to ZAV interim and
of measures. During the first phase of LdV it final reports on approximately 50 approved
conducted its own surveys of participants projects (out of 60-80 applications). The
who had spent between three and six months reports have a uniform structure.
working abroad. The response rate was Before the final project payment can be
between 60 and 70 %. The focus was on made, the success of a project is assessed
collecting data for labour-market statistics. by comparing the application with the final
European Commission regulations stipulate report. The results of this assessment are not
that implementation centres must evaluate and recorded.
monitor LdV II projects. ZAV is therefore currently Participants’ reports are analysed with the
conducting a participant survey based on experi- help of a table which records their assess-
ences gleaned during LdV I. The survey has been ment of the most important project phases.
adapted to the requirements of InWent and DAAD The implementation centre records any
so that it can be used for programme evaluation conspicuous entries and discusses them
(category 1). In contrast to earlier studies, which directly with the providers of the relevant
served purely statistical purposes (e.g. reintegra- programme. No further assessment of the
tion into the labour market, levels of satisfaction data from participant reports is planned.
with ZAV) and were therefore anonymous, the (c) Criticism and recommendations
latest participant surveys refer to specific indi- More thorough recording of the attitudes and
vidual measures. opinions of enterprises providing placements
The three implementation centres are planning would be desirable. Since ZAV focuses its
a joint strategy for conducting future project eval- activities on the labour market and on
uation. employability, the enterprises involved in LdV
(a) Monitoring clearly have a crucial role.
Continuing project evaluation or monitoring A sustainability study should certainly be
will increase in significance. InWent has conducted but should be assigned to an
developed guidelines for project visits in outside body because of implementing
Germany and abroad. They are intended to centres’ limited time and financial resources.
facilitate systematic assessment of these This would have the advantage of alleviating
visits. Projects received spontaneous visits providers’ fears of being disregarded when
during the first phase but no cross-project new project contracts are awarded.
assessment of findings took place. Another way to increase the range of quality
(b) Providers’ interim and final reports, partici- assurance measures would be to integrate
pants’ reports individual evaluation steps into the project,
Providers’ and participants’ reports are still i.e. project providers would conduct sustain-
the most important element of LdV project ability studies themselves.
Table 2: Raster for assessment of the most important project phases of LdV mobility projects
from the point of the participants
Good Average Unsatisfactory Reasons
Information/preparation/organisation
Language course
Accommodation/catering
Accompanying programme
Support offered in host country
Placement
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 115
2.3.2.2. LdV evaluation practice at InWent annual supervisory visits abroad are planned
(a) Conference, annual general meeting for the second phase.
The LdV annual general meeting is regarded A set of guidelines for supervisory visits has
as the most important link in the programme been developed as methodological support
and project chain. It is seen as a service for for monitoring. It categorises results as
providers, permitting dialogue between the follows: overview of the entire project; prod-
providers of individual measures and the ucts/interim results/final results; dissemina-
implementation organisations. It offers tion strategy; work plan; partnerships;
250-300 project providers a platform to financing/budgeting of grant and results of
discuss any problems that have emerged and financial audit.
any difficulties that may be foreseen. Since In Germany 10 projects will be subjected to
2002, the annual general meeting has been financial audit. The most important selection
jointly staged by the three implementation criterion for these 10 projects will be the value
centres exclusively for LdV projects. The of the contract, followed by the target group
and other pertinent data. Of these
advantage of this over earlier meetings, which
10 projects, 5 will be chosen to be the focus
also included Socrates providers, is that
of supervisory visits by InWent in the context
discussions are focused on the matter in
of an assessment seminar. Standardised
hand. The details given in project applications
reports of the outcomes of these reports will
on the intended evaluation process are one of
be drawn up.
the selection criteria for project sponsorship.
(d) Treatment of evaluation findings
(b) Providers’ interim and final reports, partici-
Once a year, about three months before the
pants’ reports
application deadline, there is an opportunity
The appraisal of providers’ and participants’ to make use of the findings and experiences
reports has changed from LdV I to LdV II. In of previous years in designing new projects.
the first phase, only around 10 % of For example, evaluation reports are used
providers’ reports (final reports and interim when updating Internet homepages in order
reports if projects lasted over 18 months) to draw attention to possible changes, e.g.
were closely examined. Even participants’ accountancy procedures, duration of place-
reports were only spot-checked. ments, etc. Various InWent task forces focus
InWent is aiming for a higher standard of on different issues during these three months.
report evaluation for the second phase of the This has led to publication of a booklet on
programme so that 20-30 % of providers’ long-term measures.
reports are investigated according to a stan-
dardised process. The German Craft Trades
Institute in Cologne has drafted an assess-
2.4. Summary
ment form expressly for this purpose. Partici-
pants’ reports remain relatively unstructured, The second phase of the LdV programme pays
but more are assessed. more attention to evaluation design for both indi-
In the EU’s view, the provider of an activity is vidual measures and the programme as a whole.
the key partner of the implementation centre. At the application stage itself and during subse-
Assessment results are submitted to the quent monitoring of activities, questions of evalu-
European Commission once a quarter. ation play a significant role in determining
(c) Monitoring whether the measure receives support.
In the future, monitoring will play a key role in Each year the measures which fall under LdV
quality assurance. Arranging more supervi- are evaluated. This is a major task but the coordi-
sory visits is one of the European Commis- nation of the individual measures could be
sion’s requirements for the implementation of improved by those involved in this area. Often
LdV II. Onsite visits were rare in the first data are collected which help implementation
phase and only occurred if problems arose in centres to improve the quality of the programme
the host country. At least four and usually six or its measures in the short term but which are no
116 Evaluation of systems and programmes
longer available for subsequent, more in-depth In general, the method adopted for the evalua-
investigations. tion of the LdV programme and its impact is
To date, evaluation methods have not usually praiseworthy, since the three-year study involving
appeared to take into account the stated several groups should permit conclusions to be
programme objectives. Instead, the priority has drawn about objectives (a) and (b).
seemed to be whether a project was successfully However, useful findings regarding objective (c)
organised and whether participants did as well as are unlikely. At the moment, evaluators do not
possible. systematically consider the comments and opin-
A look at the most important objectives of ions of the enterprises involved. The only contri-
LdV I (Section 2.2) reveals that the programme bution the evaluation might make towards fulfilling
designers wanted to avoid expressing measur- objective (c) would be in providing a best practice
able goals. In principle they were hoping to gain model (Lauterbach, 2003, p. 108 et seq.).
support for vocational training throughout Europe More systematic assessment of providers’ and
without spelling out the opportunities for inter- participants’ reports could provide additional
vention offered by transnational exchange information on the impact of mobility measures
programmes. on vocational training. However, appraisals of this
Therefore, the evaluation reports on LdV I nature are very demanding, and implementation
contain little information on the actual, verifiable centres are not normally able to carry them out.
impact of LdV mobility measures on vocational One prerequisite would be the introduction of a
training in Germany. The general assumption has reporting system with a standardised report
been that a suitable number of successful indi- model, increased assistance in writing reports
vidual measures with corresponding numbers of and investigation of appropriate electronic
participants would automatically have a positive assessment procedures. It would be sensible to
effect on the Europeanisation of vocational commission third parties to carry out these tasks.
training in Germany. However, little empirical If the standards of the JCSEE (1994, 2000) and
support for this hypothesis has been apparent to the German Evaluation Society (DeGEval) are
date. taken as yardsticks, the value and usefulness of
It is to be expected that the final report on the LdV evaluation could certainly be increased.
evaluation of LdV II will contain more references According to DeGEval the four basic characteris-
to goal attainment. tics of an evaluation are utility, feasibility, fairness
Core objectives of LdV II: and accuracy. These four basic characteristics
(a) improving skills and competences of people, can be divided further into sub-categories. Those
especially young people, in initial vocational of particular relevance to increasing the quality of
training, with a view to promoting their LdV evaluation are to be found under Point 1 as
employability at all levels; ‘clarification of the purposes of the evaluation’
(b) improving quality of continuing vocational (more precise definition of the purposes of the
training and access to lifelong acquisition of individual stages of programme and project eval-
skills and competences, with a view to uation) and ‘timing of evaluation’ (contracting
increasing and developing adaptability, evaluations early so that soundly based methods
particularly to technological and organisa- can be developed and sufficient time allowed for
tional change; coordination with the institutions involved, imple-
(c) promoting and reinforcing the contribution of mentation centres and national agencies; under
vocational training to the process of innova- Point 2 as ‘appropriate procedures’ (more effi-
tion with a view to improving competitiveness cient use of the reporting system, avoidance of
and entrepreneurship and creating new repetition of statements of participants in reports
employment opportunities. and questionnaires) and as ‘efficiency of evalua-
Improved coordination of evaluation of projects tion’ (sensible division of labour, e.g. drawing in
(ZAV and InWent) and programme (WSF) will third parties or providers of individual projects);
make it possible to comment about employability and under Point 4 as ‘context analysis’
(ZAV) and continuing vocational training (InWent (supporting research into the Europeanisation of
and WSF). vocational training in training institutions and
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 117
enterprises) and ‘meta-evaluation’ (processing of Key characteristics of cluster evaluations are
data for further investigations). as follows:
Evaluation of the practical and financial aspects (a) cluster evaluations investigate a group of
of the LdV mobility programme and its individual projects to identify shared themes through
projects should be guided by JCSEE or DeGEval comparative observation;
standards. Our recommendation would be to take (b) they do not just try to understand what
a bottom-up approach, since the findings of happens within projects, but also ask why;
studies into individual projects could be incorpo- (c) they are carried out in cooperation with all
rated into evaluation of the mobility programme those concerned. Projects managers
and subsequently of the entire LdV programme. It providers, those commissioning the evalua-
would seem sensible to divide up evaluation of tion and evaluators themselves are all
the programme as a whole according to project involved in the process and everyone can
type (mobility, innovative projects, etc.) because profit from the findings;
of the diversity of measures. (d) project managers and evaluators have a
Despite this prior division, cluster evaluation confidential relationship. Information is only
would be a suitable methodological approach to released if it is relevant to the programme, but
evaluating LdV mobility projects. The heterogeneous data on individual projects are never divulged.
financial and structural frameworks of the mobility This guarantees that project coordinators are
projects demand an evaluation method developed able to report frankly to evaluators on the
for programmes with a common purpose but whose realities of their work, on problems, frustra-
individual projects are relatively independent and tions and successes. This increases the worth
have varying strategies, contexts, etc. of the evaluation findings.
Figure 1: Evaluation structures of the LdV mobility programme
Planning,
national agency,
implementation centres (InWent, etc.)
Self evaluation
dialogue with programme planners and
Annual programme 1997, etc.
implementers not systematic
Individual formative
and summative
Summative evaluation:
Various projects and providers evaluation by
implementation
centre
1 2 3 4, etc. Questionnaire,
observations, etc.
Range of assessment, procedures for assessment of Report on
individual, programmes by implementation centres evaluation
of the entire
Evaluation of the entire programme, programme
not individual projects
3. Case study 2: European Community Study Visits
programme for those responsible for Vocational
Training (CSV)
3.1. General framework This partial allocation of costs to national
budgets explains why countries are interested in
CSV has operated since 1985 under a resolution evaluation of the segments of CSV organised by
of the Council of Ministers of 13 July 1983 as an their national support agencies.
action programme to implement a European
Community vocational training policy in the 10
Member States (Belgium, Denmark, Germany, 3.2. Programme objectives
Greece, France, Ireland, Italy, Luxembourg,
the Netherlands and the UK). ‘The prime aim of the programme is to encourage
The accession to the EU of Portugal (1986), European-level exchanges and discussion among
Spain (1986), Austria, Finland and Sweden (1995) those responsible for vocational training on subjects
has significantly expanded the territory of appli- of common interest.’ (Cedefop, 2002, p. 1) (11).
cation. The programme was integrated into LdV Besides personal familiarisation with a voca-
in 1995 as a supporting measure (Council Deci- tional training system in one participating country,
sion of 6 December 1994 on LdV I). other important benefits of the programme are
That year, Norway and Iceland received permis- information flow and exchange of experience and
sion to participate. LdV II (Council Decision of views (supranational exchange) between
26 April 1999) integrated CSV into the mobility members of the study visit group, who generally
policy (II 1.c). Apart from the above-mentioned represent various stakeholders such as a social
Member States, in 2003 the future Member States partner (e.g. a programme in France with 10 to
(Bulgaria, Cyprus, the Czech Republic, Estonia, 15 experts from various countries involved in
Hungary, Latvia, Lithuania, Malta, Poland, CSV, excluding the host, France).
Romania, Slovakia and Slovenia, together with The visits are opportunities for participants to
Liechtenstein and Turkey (both under negotiation) profit from the experiences of other vocational
will participate, totalling 31 nations. training systems, especially from VET innova-
The European Commission entrusted Cedefop tions. The aim should be to establish international
with programme implementation. Cedefop organ- networks and to encourage active involvement in
ises CSV in conjunction with national liaison offi- other pertinent activities, such as planning and
cers (NLOs), who may be supported by national implementing vocational training policy measures
agencies (9). Cedefop rules (e.g. participant cost in Member States and other countries taking part
sharing) govern programme budgeting. CSV is a in the programme.
separate Cedefop budget item. Each country The fundamental European policy objective of
usually finances the national support agencies encouraging reciprocal understanding of voca-
and the many programme organisation expenses tional training systems, stands over and above
(e.g. lecture fees) (10). these narrower, more focused objectives.
(9) Each country has its own structure. In many cases implementation is carried out by members of another organisation which is
involved in educational programmes and/or initiatives, e.g. LdV agencies, CDG (now InWent), the British Council, etc. Func-
tions and tasks are divided differently between the NLOs and programme organisers in each country in accordance with
varying national government structures and policies.
(10) PHARE funds cover grants for participants from future Member States who are members of PHARE programmes. Other coun-
tries such as Norway and Iceland must cover participants’ bursaries out of their own resources.
(11) The catalogue is trilingual (English, French, German). In German this passage is worded: ‘Mit diesem Programm sollen in erster
Linie Austausch und Diskussion zwischen Verantwortlichen im Berufsbildungsbereich zu Themen von gemeinsamen Interesse
auf europäischer Ebene gefördert werden.’ And in French: ‘L’objectif premier de ce programme est de stimuler l’échange et la
réflexion entre responsables de la formation professionnelle sur des sujets d’intérêt commun au niveau européen.’
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 119
CSV, a wide-ranging Cedefop initiative aimed Since many months may elapse between
directly at the general public, raises the profile of application and visit, high cancellation rates are
the organisation and indirectly promotes aware- common. Reducing the number of dropouts is a
ness of the EU programmes supporting voca- permanent challenge for the CSV central admin-
tional training in the countries involved. istration at Cedefop, the national support agen-
cies and NLOs.
3.2.1. Programme participants Cedefop awards participants a grant toward
The programme targets vocational training deci- their travel and accommodation expenses (12).
sion-makers and multipliers. Addressees include
government and business representatives, social 3.2.2. Programme procedure and content
partners, vocational training institutions and VET Individually themed study visits for groups of 10
researchers. There should be arrangements for to 15 members from the countries involved (see
the social partners to designate a certain number above) with a variety of professional ‘horizons’,
of participants. meet in a given country whose residents do not
This often rigid ‘quota system’ is not unani- ‘participate’ in ‘home’ sessions. Usually a guide
mously applauded. However, the study visit accompanies the group as a ‘cultural’ mediator
programme is the only programme that between the guests and the host country.
addresses the social partners explicitly. For this The themes are set annually and are valid for
reason Cedefop and the social partners on the two to three years. They are the product of nego-
Cedefop Management Board are also keen to
tiations between the theme selection committee,
ensure participation in this programme by both
the NLOs and the Cedefop Management Board in
labour and management, who contribute heavily
accordance with LdV specifications. The themes
to vocational training in many CSV countries. EU
are prescribed but NLOs have scope to set
educational policy requires increased social
national emphases/priorities within the annual
partner involvement and this is the rationale
theme definitions. Even the ‘general’ topics
behind participation ‘targets’ for these groups.
(vocational training for young people, adult VET)
A working knowledge of at least one other EU
language in addition to the mother tongue is desir- are usually given a certain focus.
able. The lingua franca will be predetermined sepa- Each CSV country is involved in two kinds of
rately for each measure but with rare exceptions it programmes, incoming measures at home and
will be English. There are 70 measures planned for outgoing measures abroad. For instance, in
2003, four will be in German, one bilingual (German Germany individuals are chosen to join Danes
and English), one in Spanish and seven in French. and Spaniards at a measure organised in France.
Cedefop offers interpreting support to facilitate The German programme coordinator organises
communication between visitors and the CSV incoming measures for persons selected in
subject-matter experts in the host country who are other participating countries. This arrangement
not sufficiently versed in the chosen language. ensures that groups contain members from a
Applications are submitted to the NLOs, who range of nations.
make a preselection and forward the shortlist to The national agency (13) organises the
Cedefop. The coordinators in Thessaloniki deter- measure, and the NLO is in charge. In 2000-02,
mine the constitution of the multinational group. the figures for Germany were:
Table 3: The CSV: figures for Germany 2000-02 (incoming, outgoing)
2000 2001 2002
Outgoing (Germans) 76 71 71
Incoming (other countries) 47 52 72
(12) Cf. Footnote 11 and Section 3.1 General framework.
(13) Cf. Footnote 9.
120 Evaluation of systems and programmes
In principle, populous countries organise sion of measures as the number of participating
several measures (e.g. six each in the case of countries increases (e.g. accession of newcomers
Germany and the UK), while ‘smaller’ countries to the EU (15) and associated countries), partici-
often only host one. Member States have organ- pation is growing steadily (from 714 in 2001 to
ised approximately the same number of 750 in 2002). This trend is expected to continue.
programme measures for many years, about All participating countries are advised to send
56-59 annually. However, the scale of the their prospective participants to orientation semi-
programme and consequently the burden on the nars. Their format depends on financial and
central organisation at Cedefop are constantly personnel resources and other circumstances in
increasing due to the participation of other Euro- individual countries. Germany is one country that
pean countries (Section 1). In 2003 a total of conducts such seminars. Comparability and
26 countries will be involved in the planned transferability of vocational training systems was
70 measures with 7 thematic focuses. If neces- the theme in March 2003, for example.
sary, e.g. for a new field such as ‘informal
training’, they are prepared by having one NLO 3.2.3. Peculiarities, reporting
offer a workshop for all of his/her counterparts. For several years participants have written short
Normally measures last five days and used to group reports. These are intended to concentrate
concentrate on the (initial or continuing) voca- on analysis and assessment of the theme of the
tional training system of the host country. The visit, with more general consideration of what
1997 meeting in Copenhagen led to the introduc- was presented and experienced and the organi-
tion of supplementary three-day (14) thematic sation of the measure in question. Portrayals of
measures. Thematic events have now become the system investigated are not the purpose,
standard. The special themes for 2001 and 2002 descriptive elements should always be analytical.
in Germany were: ICT in SMEs, the quality of
vocational training systems and the role of univer-
sities in initial/continuing training.
3.3. Evaluation of the programme
Currently the host country decides on the
duration of each measure. Themed events may
as a whole, individual
therefore last four or five days and general programmes, particular
sessions on the vocational training system only aspects, etc.
three days. Variation in the length of measures
provides flexibility to adapt to national variants of
the vocational training system and to implement 3.3.1. Programme dimension, evaluation
European and national priorities. stakes and documentation of findings
Specification of themes through discussion The CSV programme description, in which it was
and agreement takes place together with a possible only to outline the developments since
critique of the current programme at annual its inception in 1985, nevertheless gives an indi-
meetings (Copenhagen 1997, Vienna 1998, cation of the many parameters requiring evalua-
Helsinki 1999, Lisbon 2000, Thessaloniki 2001, tion. Programme figures have climbed steadily.
Berlin 2002) of the NLOs, staff from the coordi- In 2003 the established programme offers
nating body (Cedefop) and appropriate Commis- almost 800 persons from 31 countries 70 measures
sion officials from Brussels, social partners and in 26 countries encompassing 7 thematic areas.
Cedefop Management Board members. Since 1985 several thousand individuals have
In 2000 a total of 634 people participated in benefited. CSV findings are expected to make a
63 study trips. Seventeen dealt with vocational major contribution to mastering the challenges
training for young people. Thanks to the expan- currently addressed by the LdV II focus on mobility.
(14) Experts on a certain theme usually only have a limited amount of time and can rarely afford to take a whole week off, including
travel to and from the visit.
(15) Future Member States may already participate (linked to LdV participation).
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 121
‘Programme results’ consisting simply of (b) organisation of the overall programme
Cedefop, NLO and/or national support agency (Cedefop);
annual reports can hardly do justice to these (c) organisation of measures independently
figures and to ambitious programme goals such conducted by NLOs and/or agencies;
as sharing thoughts, disseminating vocational (d) national selection (free and quota system) and
innovations, international networking and mutual survey of participants (e.g. motivation, match
understanding of national VET systems. They between themes and occupational field, yield
merit independent evaluations and supplemen- and report);
tary publications documenting programme find- (e) implementation of general themes in national
ings, e.g. participants’ reports, workshop presen- measures;
tations, dissemination projects. It would benefit
(f) participants’ reporting (e.g. analysis of struc-
evaluation projects if this aspect, which reflects
tures observed in the host country, conclu-
the complexity of the situation, could be
sions for their own work) and circulation of
expanded and refined.
reports;
(g) group dynamics in the individual measures;
3.3.2. Scope of the evaluation and focal points
Evaluation could focus on the entire programme (h) dissemination in the national context and on a
in a given year; on individual types of measures, European scale,
e.g. a particular theme or the length of the (i) further development of a contextualised
measure, three vs. five days; on specific visitor understanding of confrontational analysis in
groups, e.g. social partners, divided into the host country;
employers and employees, educational adminis- (j) development of intercultural skills;
trators or researchers or on a particular partner (k) continuing training behaviour following the
country. The populous countries involved since measure as an outcome of deficit analysis;
the programme began in 1985 would be appro- (l) international networking;
priate because of the greater sample size in both (m) correlation between ‘prior educational level’
longitudinal and cross-sectional studies. The (intercultural understanding, system famil-
selection criteria should be based on social iarity, language skills) and success of the
science research criteria rather than educational programme/measure;
policy considerations such as a ‘particularly (n) NLO share in the success of outgoing and
advanced vocational training system’ or ‘greater incoming measures;
need for reform’, in order to evaluate the extent to (o) support given by electronic platforms as a
which the study visits contribute to reflection, function of programme success;
further development, etc. (p) role of annual meetings.
In order to study the process of change, there It is easy to identify and to appreciate the
should also be discussion of whether only one
correlations between three complexes to which
annual programme should be evaluated or
most of the above questions can apply:
whether the preceding periods should be included
(a) national programme implementation (host
in the investigation. A distinction should be made
country);
between case studies and overall accounts.
(b) national programme participation (partici-
Regardless of the scope of an evaluation in
pants);
terms of scale of programme, the focus of the
evaluation must also be defined in programme (c) overall organisation by Cedefop combined
goals, programme implementation and with the European dimension.
programme success (dissemination and durability). These could be given concrete form in terms of
For the 70 measures in 2003, each boasting the focal points/themes cited as illustrations.
multinational groups of participants, it would also Developing the list of themes for evaluation
be necessary to explore the following focal and the consequent design of the investigation
points/topics: requires thorough consultation between the eval-
(a) the match between the themes (framework uation team and the programme organisers as a
themes and individual thematic focuses) and basis for decisions on the depth and scope of the
(CSV, LdV) programme objectives; evaluation.
122 Evaluation of systems and programmes
3.3.3. Evaluation material, data, procedures are vital steps toward the Europeanisation of
and costs their field. The planners hoped that CSV evalu-
The undisputed merits of evaluation are kept ation could help to promote transparency and
within manageable bounds by the scope and acceptance. Consequently, BMBF commis-
emphases of the evaluation. These should dictate sioned a short-term evaluation of CSV for the
what materials and data form the basis of the 1995-98 programme years, confined to
evaluation. This selection cannot be made German participants and outsiders attending
reasonably without sifting evidence such as CSV measures in Germany.
assessments of organisers’ and participants’ Here is a brief outline of this 1995-98 study.
reports, questioning organisers, interviewing
Initial analysis of background material and
participants (by phone, questionnaires, face to
literature, EU policy papers on the LdV action
face, etc.) and complex use of qualitative and/or
programme and interviews with German
quantitative evaluation tools such as auditing,
programme managers allowed definition of
e.g. path analysis and document analysis.
the goals of the investigation, which were
The available financial backing (evaluation
subsequently modified in dialogue with those
costs) indirectly influences the scope of the eval-
responsible for the programme (InWent,
uation and, consequently, evaluation procedure.
BMBF). The questions the investigation was
3.3.4. Examples of evaluation to answer were:
Two examples serve to demonstrate evaluation (i) Who represents Germany in the CSV
options for a thoroughly comprehensive and programme and which participants
sophisticated programme. One is a national eval- responded to the survey? (What criteria
uation in a populous country, namely Germany. are applied in selecting participants?)
The second is an overall CSV evaluation. More (ii) How can we explain the high participant
insight is provided by citing the measures taken cancellation rate?
by programme organisers to support (continuing) (iii) What are the motives of German partici-
evaluation, i.e. direct integration of evaluation into pants in CSV?
the programme. Evaluation findings interest a (iv) How do participants rate CSV programme
wider circle than the European Commission, implementation?
Cedefop and the NLOs. Achieving the intended (v) Does the CSV programme contribute to
multiplier effect necessitates informing target establishing contacts, international
groups in areas designated for dissemination. networking and skilling?
This applies particularly to reporting by partici- (vi) Does the CSV programme perform its
pants, including results of dissemination projects, function of conveying knowledge about
etc., and to workshops scheduled to prepare
foreign VET systems?
programmes and to establish programme
(b) Evaluation design
outcomes. Publication procedure will therefore
Different methods of investigation may lead to
also be included in this case study.
either conflicting or similar results to the same
questions. Therefore, an overall approach,
3.3.4.1. National evaluation (Germany 1995-98)
(a) Evaluation goal and operationalised investiga- analogous to the multitrait-multimethod
tion objectives system, was designed. The model delineates
From 1985 to 1998 a total of 1 341 persons and portrays the ability of various research
participated in CSV measures conducted by tools to identify similarities and differences
German organisers on behalf of Cedefop and and the thinking behind them. This approach
the Federal Ministry of Education and permits internal validation of the interpretation
Research (BMBF). This figure shows the procedure. Statements are internally valid if
significance of CSV within LdV and led the use of different research tools produces
programme planners in Germany to reflect on comparable conclusions.
why VET policy-makers’ short stays abroad (c) Research tools and data sets
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 123
Table 4: Evaluation of the CSV referred to Germany 1995-98: research tools and data sets
Data sets Populations
1997 and 1998 questionnaire responses German participants
Observations (field notes) from 1997 and 1998
German participants
preparation and networking seminars
Observations (field notes) on a 1997
Multinational group of participants
Cedefop workshop in Munich
1995-96 participants’ reports on measures
Multinational group with German participants
in non-German EU countries with German participants
1995-96 reports by participants visiting Germany Multinational group excluding German participants
This gave the evaluation three focal points: (i) • reports by German participants and
evaluation of participants’ reports and other accounts of groups which included
files/documents; (ii) questioning of German Germans;
participants; (iii) participatory observation of • reports by participants in CSV measures
sample programme implementations. in Germany, excluding Germans.
(i) 1995-96 participants’ reports The 1995-96 mandatory reports were
At the end of their trip, participants were subjected to a qualitative content analysis
required to draft a short common report to check the statements drawn from the
to provide an indication of how the study questionnaires and to answer questions
visit had gone, and of its usefulness. on the implementation and benefits of the
Cedefop furnished an outline of desired CSV programme. The following checklist
contents, but this was not binding. for assessment of participants’ reports
Consequently the quality of the reports was derived from the questions posed by
varied significantly. Some were very short the evaluation study.
and too general and most were group (ii) Survey of German participants
accounts (16), the products of a multina- A questionnaire was used to measure the
tional group which offered no scope for expectations, attitudes, impressions,
an individual’s statement. It was not opinions and judgements of former and
possible to determine whether all partici- future German participants. The descrip-
pants had always contributed equally to tive and statistical data in the question-
the composition of the report or whether a naire responses were analysed, while the
few ‘spokespersons’ with linguistic profi- characteristics and preconceptions of the
ciency were the primary authors of the participants were treated separately.
opinions and assessments. Since more The 1997 questionnaires were distributed
than one report was submitted from most to former and future CSV participants and
countries however, we can presume that covered the following areas:
the accounts adequately reflected the • critical confrontations and their assumed
views of the entire sample. A question- causes;
naire was devised to obtain individual • positive interactions and their assumed
statements, primarily from German partic- causes;
ipants. The evaluation assessed partici- • perceptions about participants from
pants’ reports covering 1995 and 1996. A different cultures and contacts in the
distinction was made between: host country;
(16) Participants were asked (by Cedefop) to draw up a group report; this did not exclude individual reports (e.g. by people who
might find it easier to express themselves outside the group).
124 Evaluation of systems and programmes
• assessment of networking through the on funding of training activities served to
CSV programme; evaluate the quality and group dynamics
• positive and negative impressions of of these measures.
the implementation of the measure; Participatory observation is designed to
• participants’ objectives and major produce the greatest possible rapport
aspects of the measure. between the researcher and the subjects
Since, with rare exceptions, only future in order to obtain insider perspectives. A
participants attended the orientation semi-standardised procedure is the best
seminar in 1998, the questionnaire was that can be achieved. Before seminar and
then modified. The 1998 version was workshop participation, guidelines for
divided into five clusters: observation were drawn up on the basis
• general data on personal characteristics; of questions arising out of the evaluation
• goals related to the CSV programme; study. The notes made during participa-
• preparation for the CSV programme; tory observation were analysed qualita-
• networking; tively and presented in the evaluation.
• assessment of and suggestions for (d) Data analysis and findings were subdivided into:
improving the implementation of CSV (i) Evaluation of organisation and general
programmes. CSV objectives that covered:
(iii) Participatory observation • 25 mandatory reports by the 122 foreign
Observers attended a 1997 networking participants in 1995 and 1996 in
seminar in Köln-Hürth and a 1998 orienta- Germany (a survey of the entire sample),
tion seminar in the same district of • responses to the questionnaire from
Cologne. Participatory observation of the Germans who attended the two
two events was intended to estimate the Köln-Hürth seminars,
significance and quality of these prepara- • 99 mandatory reports by the German
tory measures. Participation in the participants taking part in measures in
three-day Cedefop workshop in Munich 16 countries, constituting a virtual blanket
Table 5: Checklist for assessment of participants’ reports
1 VET aspects
1.1 Can VET lessons be learned from the host country?
1.2 Can the host country learn VET lessons from the report writers?
2 Programme implementation aspects
2.1 Satisfaction with the conduct of the programme (content and organisation)
2.2 Time management
2.3 Opportunity for exchange
2.4 Opportunity to forge contacts
2.5 National coordination
2.6 Funding
2.7 Evaluation of materials furnished to prepare the trip
2.8 Establishment of contacts, networking
2.9 Multiplier effects
2.10 Suggested improvements
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 125
survey, with the exception of contact before the trips if a list of partici-
Germany (17), pants’ addresses had been circulated in
• findings of the participatory observa- advance. Devoting insufficient time to
tion of the two Köln-Hürth seminars, (e) introductions at the beginning of the tour
the Munich workshop. made it harder to make contact and iden-
The investigation goals, the guidelines for tify common interests. Respondents indi-
thematic analysis referred to above and cated that chances for interaction were on
the questionnaire clusters were translated the whole inadequate.
into the following focal points: Information to help people plan their trips
• content and composition of the was often insufficient and frequently
programme; arrived too late. Those polled welcomed
• relevance to the theme of the study the preparatory meetings in their home
visit; country. Many respondents stressed the
• accommodation of individual wishes; importance of preparing by reading and
• style of lecture presentation; attending seminars. The preparatory and
• allowances for participants’ linguistic networking seminars were highly rated
ability; because of the interaction between future
• written information material; and former participants. Follow-up semi-
• time management; nars offering opportunities for post-
• opportunity for exchange and reflec- mortems were rated as highly as prepara-
tion; tory measures.
• opportunity to forge contacts; Daily conferences were proposed to boost
• national coordination; networking. Since many participants had
• hospitality in the host country; not previously been on an exchange, they
• funding; lacked experience in making international
• materials furnished to prepare the trip; contacts and requested help in this area.
• networking and contacts made, initia- Group dynamics within tour groups were
tives for future cooperation; consistently considered positive. They had
• multiplier effects: further sharing of led to the planning of more joint activities
impressions and insights. in future.
The findings from this evaluation, The lack of cultural activities was frequently
covering 1995-98, are summarised below. criticised. If offered, they were valued
Learning about another VET system on highly, particularly because they provided
the ground was the main incentive. There- an opportunity for informal contact.
fore, in addition to the tours and presen- This general evaluation also investigated
tations, participants wanted more time for German participants. Their average age
mutual comparison of vocational training was 44. More than half were men. One
systems, for sharing experiences and for quarter had already been abroad on other
discussing new impressions. VET exchange programmes. The participants
systems were portrayed by lectures, tours were highly qualified. Most were univer-
and discussions with those involved in sity graduates and held high-ranking
vocational training. Constant alternation posts. Only one in four had been through
between these media was greatly appre- a selection process.
ciated. The predominance of lectures was (ii) VET evaluation of the CSV programme.
criticised. Qualitative hermeneutic text analysis was
Opportunities to forge contacts and to applied to all reports of 1995 and 1996
reflect on experiences alone or as a group EU study programmes in which German
determined whether a visit was a success participants had been involved or which
or a failure. Participants could have made had taken place in Germany. The vast
(17) Cf. first bullet
126 Evaluation of systems and programmes
majority of CSV measures were, therefore, According to CSV programme the first
evaluated. The summary refers to the two objectives are particularly relevant to
findings of the expert evaluation of evaluation. The third requires a more
programmes carried out at that time in profound knowledge of systems and is
17 countries: Austria, Belgium, Denmark, the subject of comparative educational
Finland, France, Germany, Greece, Iceland, research. The investigation also checked
Ireland, Italy, Luxembourg, the Nether- whether programmes addressed problem
lands, Norway, Portugal, Spain, Sweden areas and whether participants commented
and the UK. on development trends.
Evaluation of professional VET aspects The obligatory reports were condensed,
focused on whether the CSV programmes brief and rather general. This ruled out
fulfilled the aim of conveying knowledge content analyses of the overall VET
of other VET systems. system. However, it was possible to find
It would have been an unacceptable responses which:
simplification to base an analysis merely • commented on the new experiences as
on this aspect, because the complexity of a contribution to the debate on reform
the matter to be evaluated would not then in the respondent’s own country (idio-
have been taken into consideration. graphic function);
Although the participants’ statements • were critical of the host country or
initially appeared to cover only vocational adopted an attitude of knowing more
training in the host country, they were about vocational training than was
actually referring indirectly to vocational found there (ethnocentric stance);
training in their homeland. Ultimately there • described problem areas and drew
was always an implicit comparison of VET conclusions on trends (evolutionist
systems and their components, since function).
value judgements require a benchmark. These statements are indicators of knowl-
Elementary issues of comparative voca- edge of the host country VET system and
tional training research were thus raised. of reflection on the differences between
These focused on three major areas: the ‘home’ system and ‘their’ system, or
• becoming acquainted with VET struc- on sub-themes which were highlighted in
tures in other countries and examining the implementation of the programme.
their characteristics in order to under- Since supplementary interviews with
stand these and to explain differences previous participants and questionnaire
between individual systems (idiographic surveys on new professional insights, akin
function); to the multitrait-multimethod approach,
• overcoming the melioristic approach to were beyond the scope of this brief eval-
comparison as a quest for the best uation and would have exceeded the
system and abandoning ethnocentric budget, statements from participants’
stances on VET issues; reports were the chief criterion for evalua-
• recognising common problems and tion of ‘system understanding’. Document
trends (evolutionist function) in various analysis, therefore, focused on whether
countries, identifying common regula- the author of the report was able to learn
tory provisions (nomothetic approach) from the host country (idiographic func-
and searching for universals (quasi- tion); or ‘knew better’ than the host
experimental function) between sys- country (ethnocentric stance).
tems. There was also examination of whether
Differing national environments markedly the reports had a bearing on problem
shape the educational and vocational areas or identified trends (evolutionist
training systems of Member States. approach). This document analysis was
Despite this diversity, there are a number not very fruitful because most of the
of common problems and trends. report writers concentrated fully on the
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 127
programme as they had experienced it. They were also shaped into a model
The group reports were ‘balanced’ programme (preparation, implementation,
compositions. follow-up, outcomes and requirements), in the
The findings of the VET evaluation were full knowledge that national implementation is
ambivalent. Cedefop guidelines for the based on the subsidiarity principle.
structure and content of the reports eval- (f) Dissemination of evaluation findings
uated were not strict enough. The reports The recommendations and the model
ranged from comprehensive treatises with programme constituted the basis for a work-
very detailed contents and in-context crit- shop with the organising agency and the
ical remarks to short findings limited to a NLO. Another result was the issue of
few sentences that provided little scope brochures on Financing of vocational training
for assessment. These statements were in Europe and Vocational training in European
frequently too general, unfounded and, dialogue.
therefore, not conducive to coherent In its capacity as central coordinating agency,
overall analysis. Cedefop supplemented studies of national
Many of the presumptuous suggestions, organisers with its own evaluation of findings.
which often assumed that there was a Numerous measures were planned and some
‘better’ VET system than the structure were implemented. Several were included in
studied in the host country (usually the the following description of evaluation
author’s own establishment was implied), measures.
proved to be based on misunderstandings
3.3.4.2. Overall evaluation of CSV
or insufficient information. Unfortunately,
In 2000, CSV was appraised as part of a Cedefop
only slightly over 50 % of participants
evaluation. The main purpose of the evaluation
exhibited full, unbiased understanding
was to assess the VET aspect of CSV in relation
and approval of the ‘foreign’ VET system.
to Cedefop tasks, its role in LdV operations and
Better recognition in national programmes
in the promotion of EU vocational training policy.
of the different circumstances of partici-
Since this evaluation did not directly address CSV
pating countries through ‘enlightenment’
objectives, it required different data sampling.
about the ‘foreign’ systems discovered
Investigators collected data by phoning and inter-
and combating of prejudices may signifi-
viewing policy-makers and organisers (Cedefop’s
cantly help to improve transparency and
CSV department staff, representatives of the
programme outcomes. European Commission and of international
(e) Recommendations and model programme interest groups and associates of the Cedefop
Evaluation findings were translated into Management Board) and by sending question-
recommendations with the following naires to stakeholders. The programme goals and
focuses: the impact on participants were not the focus of
(i) networking, the Cedefop study. Instead it concentrated on
(ii) linguistic competence, programme organisation from the standpoint of
(iii) methodological and didactic planning Cedefop and NLO organisers, external impact
and implementation of study visits, and the dissemination of programme findings in
(iv) a complex of requirements for successful Cedefop activities.
exchanges, Although the study testified to the great
(v) professional assessment of the implemen- commitment and organisational talent of those
tation of programmes (was knowledge of involved in implementing the programme, this
VET systems transferred and did partici- evaluation, which can more accurately be
pants gain professional inspiration?), described as a consultation about organisation,
(vi) participants’ reports, neglected the inherent purpose of evaluation,
(vii) areas of conflict between recommenda- which should focus on programme goals, partici-
tions and responses to the often-heard pants and subsequent impact. Instrumentalisa-
charges that study visits were nothing tion in the interest of EU vocational training policy
but ‘tourism’. and Cedefop operations in fact negated the huge
128 Evaluation of systems and programmes
potential of CSV measures to encourage partici- of the Cedefop CSV department and CSV stake-
pants and persuade them to adopt programme holders such as social partner representatives on
goals. The bottom-up effect, which would have the Cedefop Management Board.
fostered the long-term impact, was sacrificed for This ‘self-evaluation’ is meant to be
a top-down effect. Such a summative evaluation augmented by an external evaluation. Apart from
addressed neither programme aims and partici- a meta-evaluation of the self-evaluations, a
pants nor the organisation of the programme in follow-up survey of all participants is intended to
achieving the programme’s purpose. In the final provide a gauge of the lasting impact of
analysis, it did not do justice to Cedefop organ- measures. A pilot study was conducted in 2002.
isers, NLOs or participants. This claim can easily
be proved because measures initiated by
Cedefop to improve the programme are particu-
larly designed to reinforce bottom-up effects. 3.4. Summary
3.3.4.3. Measures to support (continuing) The CSV example lends itself to depicting the
evaluation evolution of the culture of evaluation. Up to the
Cedefop CSV coordinators and numerous NLOs mid 1990s evaluations resembled annual reports
obviously consider formative evaluation to be with numerous tables of statistics which said
essential to further CSV development. Since 2002 little about programme quality. In Germany, the
an appropriate action has been taken to stress NLOs (representatives of the Ministry of Educa-
preparation for measures, reporting, networking tion and Research) view that the potential of CSV
and dissemination. Participants receive a ques- should be enhanced to meet programme goals,
tionnaire, which focuses not only on organisa- led to an evaluation of data, amounting virtually
tional issues but also on networking and dealing to a complete survey, provided by the NLO and
with innovations in vocational training. NLOs also the implementing organisation. The findings of
implement participant monitoring by means of this formative evaluation, combining qualitative
questionnaires, all evaluated centrally by and quantitative methods, were recorded in an
Cedefop. This toolkit, devised by an independent expert report. Since programme revisions
team of social scientists, forms the basis for without input from the scientific community
formative (self-) evaluation. Implementation has would obviously be seen as top-down decisions,
been supported by numerous task forces made a booklet containing evaluation findings was sent
up of NLOs, national agency delegates, members out to specialists.
Figure 2: 2001 CSV evaluation structure before introduction of new evaluation practices in 2002-03
CSV Planning
1997 ff Annual programme
between programme planners
Informal evaluation: dialogue
Individual ‘dissimilar’ measures
and implementers
(Theme, target, country) Professional
1 2 3 4, etc. reports, projects,
annual meeting
Assessment,
conclusions,
Discussions: annual meeting, selective
annual reports, projects, etc. evaluation
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 129
Since then, the organisers involved have been involved in this approach to integrate formative
engaged in lively debate with the European evaluation into CSV programme structures.
Commission, the NLOs, the social partners and CSV can thus demonstrate how programme
members of the expert community, to name just structures and evaluations can be developed and
some of the most important participants. The find- aligned through what is ultimately a political
ings manifest themselves in a new organisational, process. The JCSEE and DeGEval criteria formed
reporting and dissemination structure, which the terms of reference for all four cases
supports continuing formative evaluation but also addressed in this study (18). In this context CSV is
permits self-evaluation. Participants and stake- a relevant example of the programme develop-
holders, such as the social partners, are actively ment process.
(18) Cf. Section 1.3.2.
4. Case study 3: the German-French Youth Foundation
(DFJW)
4.1. General framework Increasing internationalisation is reflected in
labour-market demand for long-term ‘skills for
The German-French Youth Foundation (DFJW) Europe’. Young employees are expected to have
was founded on 5 July 1963 as an organ of greater occupational mobility, familiarity with
Franco-German cooperation, following an agree- other countries and cultures and fluency in
ment between the foreign ministers of the two foreign languages. Occupational mobility depends
countries. DFJW’s self-appointed task is to largely on key skills. These include linguistic,
promote youth exchange between youth organisa- intercultural and methodological proficiencies. It
tions, sports clubs, language centres, vocational is not only a matter of young people acquiring
training institutes, professional associations, trade these skills: exchange programme supervisors
unions, schools, universities, communities and must also improve their knowledge in these
twinning associations. DFJW’s main role is that of areas. Stays abroad are becoming increasingly
mediator. It offers support and advice on financial, important components of the vocational training
educational and linguistic aspects of exchanges. of young people and stepping stones for their
The foundation also informs and guides providers entry to the labour market (currently hindered by
on the content of meetings and follow-up analysis. high unemployment levels).
It is financed jointly by the German and French DFJW addresses these developments in its
governments and public and private organisations. work on Franco-German partnerships, cooper-
Government assistance for 2001 totalled ating with various training institutions to focus
approximately EUR 20 million. Each year about activities on promoting these crucial career skills.
200 000 young people participate in exchanges The programmes are directed towards both indi-
and around 7 000 programmes are organised. viduals and groups. Special projects have also
Since 1963 more than 250 000 visits have taken been set up for young unemployed people and
place, involving over 7 000 000 young people. the socially disadvantaged. Examples of initial
In addition to the binational programmes, vocational training projects are partnerships
DFJW supports increasing interaction with other between chambers of craft trades. These allow
countries (in Central and Eastern Europe, South young trainees to view their trade from the
Eastern Europe and the Balkans, the Mediter- perspective of another country with a different
ranean, etc.). Currently 350 trinational visits are culture and education system (19). The aim is to
arranged each year. give them additional perspectives on their profes-
sion and the opportunity to experience other
cultures at first hand.
DFJW also collaborates with the Danish PIU
4.2. Programme objectives (Praktik i Udlandet) Centre on the vocational
training initiative TransNet. TransNet is a Euro-
DFJW supports many intercultural exchanges in pean network of institutions which organise and
several different thematic areas. ‘Careers and support work placements abroad. The place-
training’ is a particularly important example. ments are temporary job-related activities which
(19) Three intercultural projects for young artisans:
– most chambers of craft trades in both countries are involved in Franco-German partnerships and organise exchanges
between young artisans with the assistance of DFJW. Visits normally last from 10 to 14 days. German trainees become
acquainted with working methods in French companies and with the people and the culture of their partner region;
– DFJW assists young artisans to participate in one-month ‘short placements’ in France during their training;
– trainees who wish to work in a French craft trade firm for at least six months after their training contract has finished receive
twofold assistance from DFJW. This pays both for a language course before the period abroad and for travel. Wages are set
according to the host firm’s policy.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 131
young people engage in during initial vocational p. 61 et seq.) states that, following a drastic
training, in the first few years in work, while slump in the 1970s, participant numbers had
studying or taking further vocational training been climbing continually since 1980 and had
courses or as training when unemployed. The again reached earlier levels. The 10 % increase in
objectives are to improve communication and the number of participants recorded in 2001 is
cooperation between local, regional, national and attributed to the programme’s extension to a
international institutions which organise or wider clientele. Statistics such as these are
support work placements abroad and to analyse updated and published annually.
and help to overcome mobility obstacles to The foundation enhances its reports by regu-
placements abroad for school pupils, young larly commissioning quantitative investiga-
employees, young unemployed people who have tions (20). These usually take the form of surveys.
completed initial training, students and recent Past studies have addressed topics such as the
unemployed higher education graduates. changing opinions of German and French young-
One of DFJW’s aims for the programmes sters and young people’s awareness and appreci-
involving third countries is to pass on to third ation of DFJW.
countries the experience of reconciliation DFJW’s guidelines stipulate that all its
acquired through Franco-German youth programmes must be evaluated (21). The require-
exchanges. ment is fulfilled by group reports on the quality of
activities, a report from the management team
and other assessment devices (surveys,
tests, etc.).
4.3. Evaluation of the overall These reports, which are written on nearly all
programme, individual international exchange programmes, are particu-
measures, particular larly useful for the evaluation of individual
aspects, etc. programmes. Instruments of this nature are
always influenced by the subjective interpreta-
Examination of DFJW activities is particularly tions and perceptions of the authors, who are
relevant to our present purpose since it offers a ultimately assessing something which they them-
profound insight into how international exchange selves are party to. However, this reporting
programmes are evaluated. The DFJW procedure can be improved and enhanced so
programme itself has generated a considerable that reports are standardised and more objective
body of literature, which has gone beyond its through the introduction of reporting criteria or
immediate aims and which significantly participatory observation by persons outside the
contributes not only to research into intercultural programme. Both approaches have been
mobility and exchanges but also to the evaluation adopted in the evaluation procedure.
debate. The DFJW’s long history of evaluating its Over the years, numerous assessment reports
programmes has produced many approaches have been written about supported providers’
which differ in both their procedures and their exchange programmes and experimental and
focus. Numerous statistics, annual activity research programmes. The research-oriented
reports and inventories of achievement have illus- further training programmes conducted in coop-
trated the foundation’s successes since its eration with associations and research institutes
creation 40 years ago. The activity reports record are usually designed as cycles of three or four
the number of participants sponsored annually visits over three years (with a 6 to 10-day visit
and their ages, nationalities, educational each year). This is applied research with and into
levels, etc. The report produced in the year of the binational and multinational groups of approxi-
programme’s 30th anniversary (DFJW, 1993, mately 30 people. Critics complain that the
(20) One example is a representative survey of 804 French youngsters and 866 young Germans carried out in 1985 by EMNID and
a French partner.
(21) ‘Each programme shall be jointly assessed. This can be achieved via a group report, a report from the management team or
by other forms of assessment (surveys, tests, etc.)’ (extract from the DFJW guidelines, 2002b, p. 3).
132 Evaluation of systems and programmes
findings of these experimental programmes can specialist vocabulary in order to facilitate
rarely be transferred to real-life situations. work-related communication and to make greater
However, they serve to facilitate more precise use of project work, interdisciplinary procedures
description of actual exchange programme and interoccupational initiatives based on experi-
processes and can demonstrate their specific ences generated by craft trade work. The
characteristics and potential. increased role that young people are playing in
Findings from specialist conferences and designing programmes and making decisions is a
congresses staged fully or partially by DFJW are further consequence of assessing previous
also available, in addition to the quantitative eval- programmes.
uations and reports on individual programmes, DFJW also has a stake in quality assurance
participatory observations and research-oriented through the TransNet project mentioned above.
programmes (22). A series of texts and books and This has set itself the goal of providing details
many articles which have been published on the about international placement exchanges and
practical experiences of exchange programmes sharing experiences with all interested specialists
also consider evaluation issues. In general, DFJW via the Internet. The foundation also plans to
follows a clear goal of formative evaluation. This define quality criteria for international placement
corresponds to one of the tenets in the preamble exchange schemes and thus contribute to
to its guidelines: enhancing the quality of participants’ stays
‘These guidelines shall be regarded as pointers abroad.
to practical action; they must accommodate the However, DFJW is not immune to the problems
interests of young people and be adapted to of putting theoretically determined principles into
reflect changes in these interests on the basis of practice. For example, it has not been possible to
continuous evaluation of programme results’ make full use of the knowledge gained by DFJW
(DFJW, 1994, p. 3). and other bodies on improving the quality of
This accords with the demand for continuous visits because of the additional burdens on the
dialogue between providers and the foundation responsible institutions. It would therefore seem
about the evaluation results for individual imperative that DFJW implement quality policies
projects, in the aim of continually developing and by working closely and sharing tasks with its
improving programmes. partners. This would involve the foundation taking
These efforts and the work in recent years of on a more active service role, providing advice for
DFJW and external bodies on quality standards its partners.
and evaluation in international youth services
have produced a relatively broad consensus on 4.3.1. Internal evaluation analysis
several basic issues. For example, it is apparent A striking feature of DFJW activities is the benefit
that there is no automatic mechanism for meeting derived from scientific analysis of the
people and getting to know them, indicating that programme, its content and methods. This was
the potential of intercultural interaction has not clearly shown at a study conference on the theme
been fully exploited. The increased number of of Intercultural relations and intercultural learning
trained mediators and moderators should help in 1990 and in the 2001 publication Europakom-
overcome this as they are familiar with the petenz lernen by Demorgon et al.
cultures involved, have adequate language skills,
know about bicultural and multicultural dynamics 4.3.1.1. Intercultural relations and intercultural
and have ample practical experience with these learning study conference
group processes. Employing these support staff The conference was jointly organised by the
incurs additional cost, but the benefits of an Social science study group on international prob-
improved exchange situation far outweigh this lems (SSIP), the Exchange research working
drawback. A further programme improvement group, the École normale d’instituteurs and the
would be to develop and publish glossaries of DFJW. It focused on the need for intercultural
(22) A good example is a publication on a study conference in 1990 with French and German researchers on the theme of ‘Inter-
cultural relations and intercultural learning’ (Thomas and Abdallah-Pretceille, 1995).
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 133
exchange and the form that this should take. cannot be based on other evaluation models but
From experience of the DFJW programme, the must derive from the matter to be evaluated. The
conference deduced that the mere availability of desire to evaluate intercultural work is, however,
opportunities for intercultural meetings was not paradoxical, in the sense that evaluation presumes
sufficient to ensure the intended positive and the existence of generally agreed standards. This
lasting outcomes. Intercultural encounters can problem is aggravated by the large number of
have various negative effects such as the denial evaluations. In fact, an evaluation of evaluations is
or trivialisation of cultural differences. This is necessary to establish which methods can actually
often because organised intercultural visits can provide useful and, more particularly, reliable infor-
easily acquire the character of a halfway house, mation. However, in order to ensure reliability and
starkly isolated from the real world of cultural a certain standard of quality, a definition of ‘objec-
clashes. If a visit is not professionally organised, tive’ criteria is essential. However, this is particu-
misunderstandings can lead to disapproval, larly difficult in international youth work, since
dislike or even premature departures. On the cultural and local differences manifest themselves.
other hand, modern, romantic ‘multi-culti’ aspira- It is precisely these differences that give activities
tions are also doomed to failure if they cannot be their intercultural character. This affects both the
properly channelled. Gaining a better picture of search for reliable standards and assessing educa-
the problem and finding the right approach to tional methods, including evaluation methods,
designing successful intercultural visits demands which are not culturally neutral but which reflect
more precise and thorough analytic approaches national traditions, value judgements, etc. The
which take into account all cognitive, affective,
foundation, however, has not reacted to this
behavioural and personal factors. Only
problem by abandoning its clearly defined demand
programmes of this complexity will make best
for quality and resigning itself to inconsistency.
use of intercultural exchanges.
Instead, it has made it a goal to establish a culture
The conference also revealed that the German
of communication on assessment criteria and
and French programme objectives differ funda-
debate on quality and approaches to quality
mentally. The Germans are interested in the value
criteria. The chosen means to achieve this is
of exchanges as an opportunity for Germans to
‘self-assessment’, which involves programme
gain skills in international interaction. The French
managers from the different cultures working
focus more on questions of their own cultural
together to develop categories and criteria for
identity. They prefer more hermeneutic,
interpretation which they then apply to the
symbolic-interactive and ethnomethodological
arrangements for visits. The advantage of
research approaches, whereas the Germans
follow positivist deductive traditions, oriented self-assessment is that it can be conducted while
towards action theory and strongly influenced by a programme is running, permitting findings to be
Anglo-American emphases. This has conse- directly incorporated and necessary changes to be
quences for the design and implementation of immediately introduced. In practice, international
evaluation, particularly since those involved are exchange visits at all levels represent a continuous
often not aware of these differences. Conflicts process of reciprocal testing, probing and
over objectives and lack of mutual understanding adapting, in a process of practical research and
are dangers likely to affect quality assessments of evaluation. External evaluations on the other hand,
international programmes, such as those of the do not so much serve the interests of the organi-
EU, particularly given the recent trend of greater sation itself as provide a basis for comparisons
interdisciplinarity. with other programmes.
The evaluation structure depicted below illus-
4.3.1.2. Acquiring skills for Europe trates how the different aspects of the evaluation
DFJW’s evaluation approach has much in process interact and how they are expressed in
common with the theory propounded by plans for future programmes. It is clear that
Demorgon et al. who have been involved with formative evaluation is the central idea and that
DFJW for decades. They advocate a comprehen- the whole structure is designed to facilitate
sive approach to evaluation that is closely linked continual refining of overall planning and the
to the topic. Evaluations of intercultural visits design of individual measures.
134 Evaluation of systems and programmes
Figure 3: DFJW evaluation structures
DFJW Planning
Integrated
Yearly programme 1997, etc.
Formative evaluation: dialogue
between programme planners
Various projects Individual evaluation
and implementers
Questionnaire,
1 2 3 4, etc. observations, etc.
Annual report
Various assessment procedures for
individual projects which are and individual
evaluated according to demand evaluations
4.3.2. Future planning Although it is not entirely possible to prove the
At its board of management meeting on effectiveness and success of the programme with
22 June 1999, DFJW presented a guideline report a list of annual figures, surveys and performance
for the following three years. Alongside an anal- statistics based on quantitative data, these do
ysis of the current situation, the report considered help to describe the foundation’s activities so that
future prospects. In relation to evaluation, the they can be discussed at a general level. At the
foundation proposed to build on self-assessment same time, while subjective individual reports
by drawing up notes for the general public on may not give a useful overview of programmes,
how to design and carry out this type of assess- or measure or prove anything, but they make an
ment. It also intended to use the Internet more important contribution to formative evaluation,
intensively as a service tool and to publish infor- since personal experiences can provide helpful
mation on quality issues, particularly examples of hints for possible improvements in the planning,
programmes as a stimulus for organisers. It also implementation and content design of visit
planned to do more to evaluate the increasingly schemes.
significant trinational programmes, particularly Certain aspects of DFJW’s evaluation proce-
with countries in Central and Eastern Europe and dure need to be discussed in the light of the eval-
to go on to prepare guidelines describing typical uation requirements set out at the beginning of
problems and ways to address them. this chapter. DFJW is taking the opportunity to
engage in scientific debate about issues arising
from international visits. This allows it to seek a
4.4. Summary theoretical basis for its activities and gain an
outsider’s perspective on its processes. Other
Because of its long history, DFJW has had much positive features include the research-oriented
experience with many kinds of evaluation projects, which directly target certain aspects
methods. However, the different types of evalua- under controlled conditions. In this way DFJW
tion are linked to varying information requirements. can continue to look for appropriate standards of
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 135
quality, which are crucial for impact research. and published research on the topic. It should
However, there is still scope and a need for also improve its organisation and cooperate more
further discussion. Standardising the design and closely with its partners. Using binding standards
practice of individual assessments is certainly a for individual reports, data collection, etc. would
step in the right direction towards increased make evaluation more efficient and useful. The
control over what features individual reports foundation should also satisfy the requirement for
should contain. meta-evaluation in order to compare various find-
As the foundation itself has stressed, it needs ings and draw useful conclusions for future
to make better use of existing evaluation models programmes.
5. Case study 4: International Exchange Programme for
VET Specialists (IFKA)
5.1. General framework exchange partner countries to Germany. The aims
of this process include:
In 1969 the Federal Republic of Germany and (a) giving a more international focus to voca-
Japan agreed on an exchange programme for tional training in Germany;
specialists. In 1978 this became IFKA at the insti- (b) ensuring that acceptance of formal and
gation of the German Education Minister, Helmut informal qualifications obtained abroad is not
Rohde. It is now the longest-running vocational hampered by bureaucracy;
training exchange programme of the Federal (c) promoting the comparability and trans-
Ministry of Education and Research (BMBF) and parency of qualifications by establishing
grew out of the political principle of promoting equivalences.
equality between academic and vocational To respond to these new extended require-
education. Implementation of this principle ments, the programme has, since 1999, adopted
concentrated on in-company training personnel, special focuses, tailored to each target country.
who were a primary concern of politicians and Some examples are: making vocational training
social partners, particularly after the VET Act with more flexible; IT skills systems; multimedia
its regulations on qualifications for trainers was learning; modularising vocational training; region-
passed in 1969. Most of these trainers were alising vocational training; training courses
skilled workers or master craftsmen and these providing double qualifications (equality of
are still, after 30 years, the key figures. Conse- academic education and vocational training);
quently, programme objectives are still specifi- training in the service sector; transition processes
cally tailored to the personal development of in Central and Eastern European States; and
in-company trainers. With these guidelines, the vocational training system innovations.
programme implementation explored the host When IFKA was founded in 1978, the former
country’s educational and vocational training Federal Ministry of Education and Science
system in relation to a specific project against the (BMBW) commissioned the Carl Duisberg
background of cultural, historical, societal, polit- Society (23) to conduct the exchange programme
ical and economic conditions. for VET specialists. Specialists are assigned to
BMBF’s proposed new exchange scheme, other countries and received as guests according
‘More international mobility in Vocational Training’, to guidelines laid down on 9 February 1984 by
which has been developing since the late 1990s, BMBW’s successor, the Federal Ministry of
maintains this programme objective but enhances Education and Research (BMBF). BMBF stipu-
it with new emphases. These reflect the increased lates which countries can be involved in the
internationalisation and globalisation which has programme and decides on the German appli-
occurred in the 20-plus years since IFKA was cants proposed by the CDG/InWent expert
founded. They build upon the expansion in training committee, who apply through a public recruit-
for vocational training personnel (outside voca- ment process. The foreign partners choose their
tional schools) which has taken place since 1969. participants according to their own guidelines.
Projects which had previously focused on the In general, IFKA is based on reciprocity. An
personal development of individuals have been exchange takes place every year with each
modified for practical purposes to facilitate the partner country. ‘Outgoing projects’ refer to visits
transfer of vocational training innovations from involving participants from Germany going
(23) Since November 2002 InWent – Internationale Weiterbildung und Entwicklung gemeinnützige GmbH is an organisation for
international human resources development, advanced training and dialogue. It was established in November 2002 through a
merger of Carl Duisberg Gesellschaft e.V. and the Deutsche Stiftung für Internationale Entwicklung (German Foundation for
International Development).
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 137
abroad, ‘incoming projects’ to visits by foreign (d) international mobility and cooperation in
groups to Germany. By the end of 2001 over education and trade is promoted, strength-
6 000 German and foreign specialists had taken ening Germany’s position as a provider of
part in more than 450 information and training training in the training market.
visits (15 outgoing projects per year) in 40 coun-
tries. BMBF and the foreign counterparts of 5.2.2. Participants
CDG/InWent directly responsible for the Participants come from backgrounds which fall in
exchanges had invested almost EUR 35 million in the broadest sense into the domain covered by
the scheme. This does not include the expenses the VET Act (BBiG). These include: vocational
of the companies, associations, etc., involved in trainers, training masters, managers and advi-
exchanges in connection, for example, with sors; apprentice counsellors; in-company contin-
hosting company visits, giving lectures and uing training personnel and trainers from
presentations. If these were included, the total approved providers; personnel managers and
amount would probably exceed EUR 50 million. staff development officers. Prerequisites for
participation are several years’ experience in
vocational training and involvement in inter-
company vocational training activities such as
5.2. Programme goals training or examination committees.
IFKA is a group scheme with approximately
15 participants per project. They are prepared for
5.2.1. Objectives their stay abroad by a briefing seminar. Their visit
In 2002, the professional and personal objectives lasts about two weeks during which they learn
were as follows: ‘Education, economic policy, about the vocational training system of the host
professional and personal matters with the country in a series of lectures, seminars and
following focuses: continuing vocational training; practical demonstrations in companies, voca-
familiarisation with education and vocational tional schools and other vocational training insti-
training systems and professional practice in tutions, etc. The briefing and debriefing seminars
other countries and their economic and political are held in Germany.
contexts; support for multipliers who pass on the
knowledge they acquire abroad to their own 5.2.3. Use of electronic media for preparation,
companies and others; a contribution to making implementation, dissemination and
vocational training systems more flexible and lasting effect
improving intercultural understanding’ (extract To prepare participants for their stays abroad
from BMBF guidelines). electronic media are used increasingly to facilitate
Further training abroad should ensure, among communication among them, to create lasting
other things, that: networks and to consolidate results (e.g. innova-
(a) companies and other institutions sending tion transfer and reports on special topics from
workers abroad on a IFKA visit profit from the the host country). CDG/InWent has developed an
international exchange and the establishment Internet platform for these purposes.
of national and international networks. This
applies particularly to small and 5.2.4. Partner countries
medium-sized enterprises; IFKA exchanges have been organised with the
(b) participants gain additional professional and following States: Argentina, Australia, Austria,
general skills, including international and Belgium, Brazil, Bulgaria, Canada, Chile, the
intercultural competences; Czech Republic, China, Denmark, Finland,
(c) experience of a foreign vocational training France, the former GDR (24), Greece, Hungary,
system enables participants to stimulate Indonesia, Ireland, Israel (25), Italy, Japan, the
innovation within the German vocational Republic of Korea, Luxembourg, Malaysia,
training system; the Netherlands, Norway, Poland, Portugal, the
(24) Independent programmes, which adhered to the same principles as IFKA until 1998.
(25) Idem.
138 Evaluation of systems and programmes
Russian Federation, the Slovak Republic, primarily targeted at improving project design.
Slovenia, South Africa, Spain, Sweden, Switzer- However, IFKA’s summative evaluations of
land, Taiwan, Thailand, Turkey, the UK (England annual programmes (e.g. 1999, 2000, 2001)
and Wales/Scotland), the US and Vietnam. have had a strong formative effect because
proposals from previous years’ assessments
have often been incorporated into exchange
programmes in subsequent years. This
5.3. Evaluation of the programme
approach of asking for proposals for struc-
as a whole, individual tural change as a major element of evaluation
programmes, particular was not merely the evaluators’ idea, but was
aspects, etc. also specified by the client. CDG/InWent
provides preparation for evaluation by
arranging joint workshops between organ-
5.3.1. Principles and focuses isers and evaluators and offering advisory
In order to achieve a comprehensive assessment discussions.
of IFKA, the basic questions asked in the evalua- For practical reasons, there has been no
tion focus on two issues affecting the impact of planned monitoring of individual projects as
IFKA (the transfer of results) on: part of an annual programme. A viable
(a) People scheme would involve evaluators supervising
(i) on programme participants (cause-and-effect up to 20 measures for 14 days each in many
model); host countries (Section 5.2.4). This is not
(ii) on other people or institutions connected practicable purely on grounds of cost. In
with IFKA in some way (dissemination of order to demonstrate to evaluators how a
results). programme is implemented in practice, they
(b) Systems should consider some ‘sample’ monitoring of
(i) on the vocational training structure of the a group, in the role of tutor for example.
enterprises where IFKA participants work; Several evaluators have taken advantage of
(ii) on vocational training policies outside this opportunity.
those enterprises;
(c) Programme organisation and its relationship 5.3.2. Chronology of the German Institute for
to and influence on ‘people’ and ‘systems’. International Educational Research
Until the mid 1990s evaluation focused on the (DIPF) evaluation of IFKA
classic programme objectives related to CDG/InWent and DIPF began working together in
personal development. Since then, evaluation 1982. They agreed to conduct an evaluation of
has increasingly concentrated on IFKA’s new IFKA participants’ reports from 1979 and designed
‘professional’ programme objectives oriented the study together. In 1984 a reader was published
toward vocational training policy. on vocational training abroad from the trainers’
A distinction has to be drawn between viewpoint. This focused on the four emphases of
summative evaluation as a global, summary subsequent evaluations mentioned earlier:
assessment after a year’s programme of (a) programme objectives (proposed and
outgoing projects and formative evaluation achieved);
Table 6: 1979-83: documentation of findings and formative evaluation
Time period/ countries Focuses Type of assessment Documented in:
Organisational framework;
Qualitative document
Until 1983: Outgoing career benefit;
analysis with question Lauterbach, 1984
and incoming, all countries personal benefit;
matrix
professional benefit.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 139
(b) programme implementation; to support the personal development of target
(c) programme yield (added value); groups via a tailored programme such as IFKA.
(d) proposals for continuing programme improve- Other important aspects of evaluation were the
ment. transmission of results to the appropriate special-
Proposals for programme improvement ists (in this case through publication) and the use
resulting from the qualitative evaluation of the of the experience gained in preparing exchange
participants’ reports (at that time freely formu- participants in subsequent years.
lated) initially concentrated on the preparatory
stage of programme implementation. They 1984-93 (selected documentation of results)
focused on: From 1984 to 1993 evaluation of certain parts of
(a) professional familiarisation with the vocational IFKA concentrated on vocational training collabo-
training system in the host country in the ration with countries regarded as significant
German partners, e.g. China, Turkey, and the role
context of the general background (the
model, Japan. The evaluation reports resulting
education system, politics, culture, the
from a quantitative hermeneutic text analysis
economy, etc.);
related to the three selected countries referred to
(b) the organisation of the programme and the
the previously mentioned ‘traditional’ programme
measure (especially guidance for participants
objectives of participants’ personal development.
from support staff, the match between
The findings also fed into country studies of
programme objectives and the actual
China, Japan and Turkey prepared at the same
programmes of the individual measures);
time by researchers.
(c) the structure of the programme (exploitation
By 1992, more than 1 500 German IFKA partici-
of opportunities to learn about the whole pants had submitted reports on their experiences
spectrum of the host country’s education and between 1983 and 1991 and the study was even-
training system). tually extended to include projects up to 1994.
In many cases programme objectives were not These reports were evaluated, complemented by
achieved, particularly those relating to ‘familiari- CDG/InWent material on the IFKA programme,
sation with education and vocational training CDG/InWent accountability reports to BMBW and
systems and occupational practice in other coun- internal CDG/InWent programme assessments.
tries in the context of economic and political The then BMBW, CDG/InWent and DIPF cooper-
conditions’. This was because the overwhelming ated on developing a design for computer-aided
majority of participants used the German dual hermeneutic evaluation. The evaluation focused on:
vocational training system as their reference. The (a) professional and personal added value
evaluation suggests this was mainly due to (including intercultural understanding) for
participants not being able to put things into participants;
context indirectly demonstrating the urgent need (b) benefit for the companies and other organisa-
Table 7: 1983-94: formative evaluation and documentation of findings
Time period/ Focuses Type of assessment Documented in:
countries
Computerised qualitative document
A section of Chapter 7, on
analysis with database of relationally
Organisational ‘transnational mobility and
linked main categories and keywords.
Since 1983: framework; international
All participants’ reports were
all outgoing career benefit; vocational training cooperation’
processed accordingly.
countries only personal benefit; in Internationales Handbuch der
Particularly the professional aspects
professional benefit Berufsbildung (IHBB),
of the foreign vocational training
Baden-Baden 1995-2002
systems were evaluated for the IHBB
140 Evaluation of systems and programmes
tions employing the IFKA participants and for of 40 country studies and a comparative intro-
international vocational training cooperation; duction totalling around 4 300 pages. Its function
(c) programme organisation and implementation within IFKA has led to fierce debate in the scien-
and programme improvement; tific community.
(d) transfer of results, with the following compo- The evaluation findings also provided input for
nents: their own operationalisation in programme plan-
(i) professional preparation of participants; ning, implementation and assessment. This took
(ii) availability of results to the relevant inter- the form of meetings and workshops for
ested specialist audience (enterprises, employees of CDG/InWent and BMBF, programme
educational authorities, etc.), e.g. multi- participants and DIPF evaluators. The outcomes of
pliers, networking, greater flexibility in the this complex and costly formative evaluation
German vocational training system; (approximately EUR 160 000 over 18 months) were
(iii) discussion of practical and theoretical not only those visible externally, such as new
results in the scientific community. reporting structures for participants and the IHBB.
The evaluators concluded that the programme They also included many changes to the
objectives had been achieved but that the programme itself and to personnel training with
programme could be greatly improved if changes people on the ground implementing the majority of
relating to programme planning, implementation and the recommendations from the bottom up.
assessment, reporting structure, training of support In 1997/98 a survey of IFKA participants from
staff, collaboration with partner organisations abroad 1983 to 1996 was conducted to supplement the
and dissemination of findings were introduced. qualitative formative evaluation primarily based
The Internationales Handbuch der Berufsbil- on participants’ reports. In order to permit an
dung (International Vocational Training Handbook, overall assessment of IFKA, the main question of
IHBB) was and is an important medium for the the research study, which focused essentially on
circulation of the results of participants’ experi- people, was worded as follows: ‘What impact did
ences of their host countries’ education and IFKA have on programme participants (a
training systems. It can also be used to prepare cause-and-effect model must be reconstructed)
participants in outgoing projects for their stay and on other people or institutions in some kind
abroad. Today the IHBB is a loose-leaf collection of contact with or in some way affected by IFKA?’
Table 8: 1983-96: summative evaluation, added value of IFKA, formative proposals for
programme redesign
Time period/countries Focuses Type of assessment Documented in:
Survey of all participants
Questionnaire data on
from 1983 to 1996
participants from 1983 to
via a questionnaire
1996
(47 % response rate)
The participants’ reports
professional/ stored in the database under
1995 and 1996:
career-oriented main categories and keywords
all outgoing countries,
continuing training; undergo second-level Findings are recorded in
detailed assessments
international skills; correlation and selection using Evaluierungsstudie
of comparative studies
international contacts; structural content analysis zum IFKA.
for Finland, Japan
networking within (WinMax program) based
and the US
the visit group. on categories developed
from the survey assessment.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 141
A complex model of the conditions for a were correlated with the findings on personal
‘successful’ exchange was developed on the benefit. This allowed conclusions to go beyond
basis of empirical analysis of available data. This personal added value and to make recommenda-
created a theoretical framework for the findings. tions for the ‘ideal’ IFKA project.
The research adopted a multimethod approach The questionnaire data were structured using
because there were several ways of accessing path analysis (See Figure 4) to obtain evidence of
the matter to be evaluated and a research project cause and effect. Cause-and-effect analyses
goal was to deal with the question from different require longitudinal investigations to ascertain
angles and at different levels. Various methods which projects have what impact (systematic
were therefore used to explore the facts. Since variation of projects). However these seemed
different sampling tools can produce the same or unlikely to occur in this case because of for
different results, the processing procedure tested example, the client’s future prospects, cost, the
the similarities and differences between the tendering system of public bodies and the design
various data-gathering devices. The convergent of the study. The plethora of potential relation-
and divergent findings of the research tools ships was therefore reduced and the model was
provided information on the validity of certain tested to see whether it was able to represent the
judgements and the possibility of performing an available data adequately.
internal validation of the assessment procedure. In order to demonstrate the potential of this
A multimethod approach was selected to evaluation procedure for key (retrospective)
examine the research questions and to compare results and future (prospective) design changes, a
and correlate the various types of information. number of results were quoted, with speculative
recommendations.
(a) Participant questionnaire data from 1983 to
1996 (a) Participants’ expectations from individual
projects
Former IFKA participants from 1983 to 1996
completed questionnaires on their attitudes, In general, participants had high expectations
opinions and judgements of IFKA, using of their stays abroad. They mainly hoped that
prescribed categories of answer. This made it they would become familiar with other voca-
possible to perform a correlate analysis incor- tional training systems, broaden their horizons,
porating some personal data. meet people from other cultures and learn to
reflect critically on their own work. The peda-
(b) Participants’ reports from 1995 to 1996
gogical aspect was also important to them.
Since participants committed themselves to Participants did not expect to be able to forge
writing a report before travelling to their host links with enterprises, learn new approaches to
country, valuable information about the visits interpersonal relationships with colleagues,
was available immediately a visit was over. improve language skills or develop friendships
CDG/InWent provided a precise report frame- in such a brief period of time.
work to which participants were obliged to Fulfilling expectations increases the chances
adhere. Their reports were independent, indi- of participants being multipliers and obtaining
vidual accounts. Utilising both data sources key skills, thereby profiting personally. These
in conjunction allowed valid judgements to be outcomes closely related to the professional
made on the quality of implementation and on benefits of gaining ideas for participants’ own
the benefits of IFKA. work. If this aim was realised, participants
In contrast to the annual evaluation which was were likely to assess the exchange very
expected to provide detailed information on both favourably.
the annual programme and the individual projects
(b) Personal benefit: gaining key skills
and their added value for people and systems
(Section 5.3.1), this summative evaluation placed Answers to the questions demonstrate clearly
the central focus on the effect taking part in IFKA that IFKA strongly inspired participants to learn
had on the individual participants. The individual about other cultures. They stated that their
target countries were of interest because condi- appreciation and understanding of the host
tions there (organisation and programme content) country’s culture and of aspects of daily life
142 Evaluation of systems and programmes
Figure 4: Structural analysis of evaluation (survey of participants) 1984-96
had increased greatly as a direct result of the cultures. It appears that IFKA encourages its
exchange. They also felt they were now better participants to reflect more on their own
equipped to deal with people from other culture and to be more open to foreign ideas.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 143
(c) Professional benefit: gaining ideas for one’s worked in companies with fewer foreign
own work connections. Consequently they were far less
involved in developing international contacts
Taking part in exchanges gave participants
than men.
new ideas for didactic, pedagogical, social and
cultural aspects of their work. Visits were not (h) Opinions of IFKA improved over the years
as productive in exchanging and implementing Participants’ opinions of IFKA improved over
specific plans or teaching and learning mate- the years. However, there was a decline in
rial. The majority of participants felt that the personal benefit in ideas for participants’ own
primary benefit of the visit was personal work through exchange of training material
enrichment. The stay abroad rarely had a and teaching and learning material and in the
direct impact on participants’ careers. Some multiplier effect in motivating colleagues and
reported increased status among colleagues trainees.
and superiors and sometimes envy.
(i) Previous experience of an international
(d) Contacts/networking exchange improved status within the company
Information and training trips enabled many Previous experience of an international
participants to make various contacts. 54 % exchange had a positive influence on the
established private contacts with people in status of IFKA participants within the
the host country with 33 % maintaining these company through increased competence and
contacts to the present day. 26 % of partici- respect as a result of taking part in IFKA.
pants had even taken private trips to the host
(j) Academic and vocational training was not a
country after the visit. It is therefore clear that
significant determinant of success in IFKA
these contacts were quite close and quite
remarkable since IFKA projects are Analysis of the questionnaire data analysis
short-term and the intense schedule leaves showed that academic and vocational
little time for private interaction. training had no influence on the success or
failure of participation in IFKA.
(e) Employers and superiors
(k) Essential added value
Employers and superiors must be actively
interested in IFKA if the multiplier effect is to Studies into the essential added value of IFKA
succeed and if participants are to gain ideas revealed that its benefits were not simply
for their own work and to implement these at derived from the directly programme-related
their place of work, fostering the development opportunities given to participants to famil-
of international contacts. iarise themselves with other vocational
training systems and to gain key skills. IFKA
(f) Attitudes toward the host country
also allows German vocational training
Positive attitudes toward the host country specialists to establish contacts with enter-
related to personal benefit in gaining key prises and institutions in the host country and
skills and new ideas for participants’ own to become acquainted with them in a way
work. that would not have been possible through
mere tourist activities. Participants believed
(g) IFKA projects affect men and women differently
that bringing together people with similar
The average age of women taking part in occupational focuses created the right condi-
exchanges was 40, five years younger than tions for future commercial and project coop-
the average male participant’s age. They had eration between IFKA participants and institu-
better initial training qualifications than men, tions and companies in the partner country.
had superior language skills and tended to The results of the qualitative and quantitative
have a more positive attitude towards the text analysis of participants’ reports from
host country. However, they had less experi- 1995 and 1996 corroborated, illustrated,
ence of international exchange and usually refined and complemented the findings from
144 Evaluation of systems and programmes
the questionnaire data (Table 8). The fact that research and comparative vocational training
the reports allowed participants to express research. Supplementing this more research-
themselves freely supports the plausibility of oriented study, the short book 20 Jahre IFKA
the statements made in the survey and the
(Göbel et al., 1998c) targeted a wider audience
proposals for programme remodelling.
The findings of evaluations from 1983 to 1996 and served a marketing purpose through a
were published in a monograph which also public relations event that supported IFKA
contains more general chapters on exchange public relations.
Figure 5: Schematic model of ‘programme success’ factors, taken from surveys of participants
and assessment of participants’ reports
Internal
Visit contacts External
selection contacts
Supervision Time
management
Orientation Exchange
seminar programme
implementation
Multiplier
Personal and professional consequences
effect in
Attitude Cultural profession
towards the
differences
Initial motivating factors
host country
to host country Personal
benefit:
Incentive key skills National and
through international
employer Professional
interest in networking
benefit: ideas
IFKA for own work
System
congruity
Fulfilment of Professional
expectations benefit: status
in company
Participant
composition
Sex
Academic and
Year of vocational training
participation Function of
in IFKA participant as
Company’s a trainer
international Foreign
contacts language skills
Age when Experience of
participating international
in IFKA exchange
Employment
status
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 145
Table 9: 1995-2001: annual formative evaluation
Time period/ Focuses Type of assessment Documented in:
countries
Computerised qualitative
organisational framework; document analysis with
1994-98: career benefit; database. All participant DIPF annual report on
all outgoing personal benefit; reports were processed programme evaluation
countries only professional benefit accordingly with relationally for CDG/InWent and BMBF
multiplier function correlated main categories
and keywords.
Participants’ reports
evaluated according to
all major IFKA focuses
Evaluation forms with
closed and open answers
from every participant
(SPSS quantitative and
organisational framework; DIPF annual report on
qualitative assessment)
career benefit; programme evaluation for
personal benefit; CDG/InWent and BMBF
Computerised qualitative
professional benefit; Education policy and
document analysis with
multiplier function; system dimensions of IFKA
1999-2001 database of professional
long-term effect of measures discussed in Innovationen
and transfer reports using
in vocational training policy nationaler
the WinMaxprogram
(dissemination and transfer Berufsbildungssysteme
of findings) and networking von Argentinien bis Zypern
Interview with coordinators
by participants (Hellwig et al., 2001)
Project-specific assessments
also possible because
of the 78 % response rate
Annual evaluations have been carried out for objectives, a restructuring of IFKA took place,
each year from 1995 to 2001, in the year together with a number of projects, under the
following, briefly overlapping with the ‘summative slogan ‘IFKA 2000’. Evaluation design was an
evaluation 1983-96’. The last was conducted important part of the transformation. Evaluation
between February and August 2002. tools such as opinion polls of participants on the
Consequently, evaluation practice has been organisation, or their statements on the profes-
constantly developed and refined in the manner of sional added value for the German situation were
formative evaluation. Advances have not only been jointly developed alongside the new programme
the result of evaluators’ recommendations. They structures, before programmes were imple-
have also originated in the detailed structure of the mented.
more qualitative, hermeneutic evaluations up until Integration of evaluation into the programme
1998, which were based on participants’ reports on design and programme implementation allows not
their experiences written according to the reporting only the evaluation of an entire year’s projects
framework devised in 1994. The programme organ- (which can be as many as 20 individual measures)
isers (CDG/InWent) and the evaluators (DIPF) main- but also the collection of statements on a specific
tained close communication throughout. outgoing project. Reporting by participants as the
When vocational training policy shifted in 1999 basis for evaluation was completely revamped.
and changes were made in IFKA’s programme When IFKA began, the basis for evaluation was
146 Evaluation of systems and programmes
participants’ freely formulated reports on tion of programme findings. In the past, partici-
prescribed themes, the content and structure of pant reports were a particularly valuable source.
which were often adapted to changed circum- They were the subject of several publications
stances (see above). Later, the basis for evalua- (Lauterbach, 1984, 1986, 1995; Henze, 1991;
tion was on professional reports concentrating on Georg, 1993). These standard works also served
the ‘added value’ of adapting the German voca- as introductory material for future participants.
tional training system to meet international chal- One major work is Innovationen nationaler
lenges and a questionnaire with open and closed Berufsbildungssysteme von Argentinien bis
questions which participants had while projects Zypern (Hellwig et al., 2001). But participants and
were running. publications are not the only disseminators of
In 1999 and 2000 these direct evaluation data programme findings. Special events also play a
were complemented by questionnaires sent to vital role. They have included IFKA’s 20-year
every participant a year after they had finished jubilee in 1999 and the International transfer and
(e.g. evaluation of the 2000 programme included vocational qualification conference in 2001, which
distributing questionnaires to 1999 participants) involved IFKA participants, companies, national
to ascertain the lasting impact of measures, and international educational authorities, partner
particularly with regard to networking, relevance organisations abroad and the team of evaluators.
to the vocational training policy of participants’ Annual evaluations have so far not considered
companies, and innovation transfer in the voca- these events or the published reports on evalua-
tional training system. The programme organiser tion and transfer. However, their role in making
also set up an Internet platform (CDG/InWent the programme better known, gaining new busi-
website) to document participants’ professional ness partners and inspiring numerous applica-
reports in order to support networking by partici- tions for the projects should not be underesti-
pants and record innovative projects relevant to mated.
vocational training policy resulting from individual
participants’ encounters with a foreign vocational
training system.
5.4. Summary
5.3.3. Interdisciplinary team of evaluators
Hybrid evaluation is a useful addition to qualita- IFKA evaluations have taken place since 1984.
tively hermeneutic and quantitatively empirical The programme reflects the revolution in evalua-
evaluation methods. It established itself as the tion culture and the most important principles of
norm between 1983 and 1996, synthesising the evaluation. Methods have been constantly refined
combined efforts of research teams consisting of and diversified. Hermeneutic text analysis domi-
vocational educationists, educational psycholo- nated initially, only to be supplanted by
gists and comparative VET researchers. This computer-aided text analysis complemented by
combination brings together academic VET skills participant surveys. Qualitative and quantitative
in the country of origin and the host country with social science evaluations have become an
proficiencies in evaluation research and interna- increasingly automatic part of annual evaluations
tional comparative research. and of the one evaluation of participants over
Involving different scientific communities 10 years carried out to date.
enables us to assume that evaluation research There is certainly also a case for making an
designed in interdisciplinary discussions is evaluation of IFKA evaluations, given the number
always current. It is, therefore, important that so far conducted and developments in evaluation
evaluators continue to perform active research in techniques. The standards of scientific associa-
their own fields and do not overspecialise in the tions specialising in evaluation research and
evaluation of international vocational training programme evaluation would lend themselves to
programmes. this endeavour. As mentioned above, the Joint
Committee on Standards for Educational Evalua-
5.3.4.Dissemination of findings tion (JCSEE) and the German Evaluation Society
As discussed above, since the programme began (DeGEval) have a series of evaluation criteria: (1)
one of its main features has been the dissemina- utility, (2) feasibility, (3) fairness and (4) accuracy.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 147
Figure 6: IFKA evaluation structures
IFKA Planning
Annual programme 1997, etc.
planners and implementers
dialogue with programme
Formative evaluation:
Individual ‘similar’ projects Evaluation
questionnaire
1 2 3 4, etc. and professional
report
Evaluation assessment Evaluation annual
complex quantitative and programme
qualitative document analysis Report
The utility of IFKA evaluations is undisputed in the because at the end of each measure most of the
context of increased integration of evaluation tools anticipated material is available for assessment
into the operation of the programme, changes in (except the results of surveys conducted one year
programme structures and programme outcomes later). Since feasibility is hardly possible without
such as conferences, publications and transfer of the stakeholders (organisers and participants), the
innovations to the German system. Given the use aim is to achieve a permanent dialogue. To ensure
made of the evaluation tools in ascertaining ‘true’ ‘fairness’, all stakeholders should have access
project and programme outcomes, their utility in wherever possible to evaluation results. Some
transferring innovations and further developing the shortcomings in this area have yet to be remedied.
programme is also a foregone conclusion. The The programme’s reliance on scientific methods is
integration of data collection into project imple- a step in the direction of ‘accuracy’. Meta-evalua-
mentation increases the ‘feasibility’ of evaluation tion would be of advantage.
6. Summary and recommendations for future practice
6.1. Four practical models been part of evaluation practice for years and are
a match for the DeGEval standards. Both of them
Selection of the four practical models was not broadly correspond in design to the examples of
limited to evaluations of EU mobility programmes EU programmes. DFJW is to be considered
because practical evaluation began with the comparable to LdV in its variety and complexity,
‘older’ international exchange and promotion whilst IFKA and CSV are similar in the clear struc-
programmes of other providers, such as indi- ture that they both exhibit.
vidual countries or foundations and more experi- The availability of documentation on results,
ence was available in such fields. Two of the access to appraisals and interview opportunities
examples considered (DFJW and IFKA) have were also central to the selection of examples.
Table 10: Mobility in VET – one of many ways of moving towards the principal objective
DFJW LdV
The various DFJW projects are designed to further the The LdV Programme was created to promote better
principal priority, the promotion of international under- mutual understanding of VET in the EU countries.
standing. VET furnishes one means of communication, Mobility is one of the LdV programme areas. The various
so that over the course of the years exchange activi- measures in this field serve the above-mentioned prin-
ties have evolved in this field. These activities differ cipal aim of LdV, but their forms may differ quite
greatly (i.e. apprentice exchanges, work experience for substantially (i.e. apprentice exchange, work experience
young people in initial training, exchanges between programmes for young people in initial training,
vocational schools, exchanges of trainers, long-term exchanges between vocational schools, exchanges of
and short-term measures). trainers, long-term and short-term measures, CSV).
Table 11: Mobility in VET — two concrete measures
IFKA CSV
IFKA set itself the goal of promoting VET in Germany CSV assists vocational training decision-makers from
by means of 14-day in-service training visits abroad Member States by organising group visits to EU coun-
that focus on particular themes to give specific target tries lasting three to five days and focusing on partic-
groups the opportunity of individual continuing training ular themes. The intention is to create a better under-
and to encourage innovation in VET in Germany. New standing of other VET systems (or aspects of systems
national and international networks are created and political decisions) so that what is learnt can be
through the process of group dynamics. used by participants in their own working environment
in the European context. CSV supports construction
of international networks.
6.2. Implementation of evaluation ating at a political level, want programme evalua-
tion to provide feedback and explain and justify
and practical examples
expenditure of public funds. They also hope for
Programme designers and organisers and perpetual improvement of actual programmes by
funders of programmes, particularly those oper- means of more precisely defined goals. Are these
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 149
expectations fulfilled by the four cases mentioned between professional researchers, educa-
above? The following results provide an interim tional psychologists and comparative VET
summary and combine theoretical and practical researchers within interdisciplinary research
aspects of the cases. The order in which points teams. This combination unites academic
(1) to (15) are placed refers to the order of an qualifications related to VET in the home and
evaluation based on the evaluation standards host country or countries with skills in evalua-
discussed in Section 1.3. tion research and international comparative
The evaluation team began with: research. This involvement in a discipline
(a) programme organisation, in order to examine enables researchers to keep pace with their
the focus, structure and organisation of the ‘own’ discipline.
programme; Specialisation in evaluating international VET
(b) programme goals were the next stage, programmes without this firm basis will not
concentrating on evaluable operationalisation; meet the high expectations made of evalua-
subsequent comments referred to: tions. Qualitative hermeneutic and quantita-
(c) quality of evaluation. Success is not solely tive empirical methods can be complemented
based on the methodological stringency of an by these interdisciplinary competences. In the
evaluation design in a narrow sense, but IFKA case study this combination of skills
relates to: was applied to the evaluation.
(d) transformation of evaluation results into the
practice of programme implementation, an 3. Harmonisation of evaluation goals, content
essential criterion for ‘successful’, ‘exem- and tasks between players
plary’ evaluation. Prior to an evaluation, the basic goals of
quality assurance measures should be agreed
6.2.1. Programme organisation, evaluation between those involved. Frequently there is
planning and preparation substantial disagreement on the application
1. Programme content and structure of evaluation findings. Programme designers
chiefly expect proposals for improvements in
A meaningful and practice-oriented
implementation practice, whilst evaluators
programme evaluation that draws on the
(and frequently funding bodies) are more
most recent scientific standards requires a
concerned with the individual and social
thorough knowledge of the programme
content and an appropriate analysis of impact of programmes.
programme structure. The structure of the A common understanding of evaluation goals,
individual elements of the programme should contents and tasks is an essential precondi-
be taken into account. Determination of the tion if those involved wish to work together to
suitable method for the evaluation should be improve quality.
based on programme structure and the 4. International evaluation projects and potential
programme budget and may depend on the misunderstandings
budget for activities. Consideration of suitable
methods of evaluation during the design The analysis of DFJW and LdV shows that
phase of a programme significantly heightens varied goal interpretation and evaluation
the effectiveness of formative and summative paradigms are to be expected in the case of
evaluation. For this reason evaluation special- differentiated international evaluation projects.
ists should be involved in the design phase of The risk of potential misunderstandings
VET mobility programmes. increases with growing interdisciplinary and
international involvement. Precise definition of
2. Highly skilled interdisciplinary research teams central concepts, procedures and analytic
The strict demands of evaluation require methods is necessary for this reason. Opera-
highly skilled personnel. A combination of tionalisation presupposes definition and limi-
such skills in one individual is extremely rare. tation of concepts. In German VET alone, the
In the programmes included here, a realistic literature offers a total of 654 variant defini-
approach would therefore be cooperation tions of ‘key skills’ during a 10-year span
150 Evaluation of systems and programmes
(Weinert, 2000). Evaluation can, therefore, desired changes. The role of the client is to
help to clarify research concepts. monitor the feasibility and viability of the eval-
uation.
5. Linking methods and concepts of evaluation
7. Evaluation of the long-term impact of
In the context of ‘skills for Europe’ (DFJW),
programme goals
Demorgon et al. draw attention to the close
meshing of tools and concepts in evaluation. Programme administrators’ stated goals go
They even go so far as to say that it is beyond mere concepts and include very
improper to transfer evaluation strategies broad contextual and causal relations
developed in other fields and that it is better affecting the effectiveness of measures.
practice to develop separate procedures for However, it is seldom possible to include
each individual topic. This amounts to a thor- complex contexts and long-term processes in
oughgoing constructivist approach. Irrespec- evaluation. Consequently, individual evalua-
tive of whether one shares this opinion, it tion projects need to refer to available find-
does demonstrate the need for tightly defined ings from theoretical and empirical research
concepts and evaluation methods. To date, to fill, by interpolation, gaps in explanations
this approach has seldom been applied in or forecasts not encompassed in a given time
evaluation practice. It is cumbersome and, frame or context and thereby to offer at least
although it promises excellent results, it is some hypothetical conclusions. Another
certainly more ‘expensive’ than a ‘standard option is to connect series of evaluations in a
evaluation’ applied to all manner of cases, programme. This requires not only access to
which can be restricted to counting survey previous evaluations and their data, but also
figures. presupposes their compatibility. It is then
possible to verify the stability of old findings
6.2.2. Programme goals, development of and to show medium and long-term causal
suitable methods and tools for chains. It would be better in many respects
evaluation (cost, results, benefit for the programme) if
long-term evaluation were incorporated in
6. Operationalisation of programme goals
programme planning and annual operationali-
Programme goals are usually expressed in sation. Moreover, assessment can be planned
very general, abstract and broad terms. It is in longer phases, e.g. every three years. The
essential that they be operationalised, i.e. duration of phases should be governed by
reformulated in a less abstract manner so that the parameters of the programme (e.g. dura-
their evaluatory basis consists of observable tion and sustainability of programme goals).
and, where possible, measurable phenomena.
This cannot be achieved without selecting 6.2.3. Quality of evaluation, optimisation
and prioritising variables. To eliminate arbi- potential
trariness in the reduction and operationalisa-
8. Relationship between evaluation design and
tion process, it is necessary to define the
evaluation results
precise programme goals in consultation with
the client during the design phase, possibly in Because of the extreme complexity and
a multistage procedure which includes provi- general lack of feasibility of control group
sion for subsequent correction and refine- design, there are always several alternative
ment. The role of the teams of evaluators explanations and prognoses which should not
consists of advising programme administra- be accepted solely on the strength of one
tors, in the light of thorough methodological evaluation. The implications are therefore
and theory-based expertise, particularly on always limited and related to the scope of an
the effects of decisions made during the evaluation design. This is especially true of
design phase on the effectiveness of evalua- evaluations whose results are expressed
tion projects. Tools that are to be used or vaguely. It therefore wise to regard the results
developed for a given evaluation must be of an evaluation as hypotheses which should
sensitive enough to register and reflect feed into further evaluation.
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 151
9. Match between an overall programme and based statements which include data from
individual programmes and measures the level of the individual participant up to
that of the system require highly complex
The study of the series of mobility measures in
evaluation designs and analysis procedures
the LdV programme has revealed many ways in
which surpass usual EU-level VET evaluation
which evaluation may be optimised. Although
projects in their scope and resource require-
most of the elements of programmes and
ments. Such innovative evaluations by
measures are covered by evaluations
TIMSS, PISA and the forthcoming DESI
conducted by the three implementation centres,
InWent, ZAV and DAAD, there is a need for project are based on expensive designs
coordination between individual steps and eval- incorporating several levels in the evalua-
uation of the entire programme in order to avoid tion (26). They aim to make statements on how
repetition and to improve the validity of small effects at individual (student) level relate to
steps that build one on the other. effects at system (school) level. Quantitative
multilevel analysis has shown that there can
10. Long-term impact of programme goals be surprises: causes of positive effects at an
The inability of the majority of evaluations to individual level will not necessarily have such
create a control group is a particular problem. effects on a system level and vice versa. More
Three possibilities may be imported from such level-specific effects are to be expected
scientific research methodology: when investigation involves large numbers of
– statistical control; levels as are found in surveys at a suprana-
– complex models; tional level. It is therefore recommended that
– internal comparison applying modern representative large-scale assessments which
qualitative evaluation strategies. incorporate several aggregate levels should at
These possibilities all have the same essential least occasionally be conducted within the
purpose: the essence is to analyse the fluctua- European framework too. All individual evalua-
tion of key variables to monitor differences in tion projects should be designed to assure
effect. Statistical control aims to concentrate data compatibility. It seems reasonable also to
on pertinent sets of variables by monitoring have regard to continuity in evaluation in order
and systematically segregating ‘disruptive’ to guarantee progress in evaluation practice
factors. Complex models such as path analysis and programme optimisation.
(see IFKA) include all variables which may be 12. Design and methods of evaluation projects in
regarded as disruptors or moderators of effects VET in the light of the OECD school perfor-
and make them calculable. The first two possi- mance study
bilities mentioned above assume a quantitative
method. The same purpose may be served by The advanced level of design and method-
a qualitative approach. Factors potentially ology in the OECD school performance studies
responsible for the success or failure of has set standards for further evaluations in
programmes can be identified by analytic education. For the first time the representative-
separation of successful and less successful ness of participants was a priority and proba-
programme components or projects. bility models which use state of the art survey
methods were applied. The fruits of this model
11. Meaningful evaluation results from several
survey are to be seen in the most recent eval-
aggregate levels and continuity of evaluation
uation application (DESI), which features new
Programmes are evaluated at many aggre- developments such as the inclusion of
gate levels. The success or failure of a given large-scale assessment of intercultural compe-
project as part of a programme contributes to tence (27). There is no reason why such evalu-
the success or failure of the programme at a ation projects should not also be conducted at
national or international level. Empirically a high methodological level in VET.
(26) In the case of DESI, 11 000 pupils and 200 schools are representative of all Länder in the German Federal Republic.
(27) DIPF acts as the secretariat for the Deutsch-Englisch-Studie-International project (DESI). The project module to develop tools
for intercultural competence is headed by Kerstin Göbel and Hermann-Günter Hesse.
152 Evaluation of systems and programmes
6.2.4. Transformation of evaluation results 14. Dissemination of results
and financial framework
The dissemination of results is of great impor-
13. Top-down or bottom-up evaluation? tance for the success of an evaluation. Evalua-
tors of VET international exchange programmes
It is possible to differentiate between two
aspects: the treatment of multiple programmes often consider their duties discharged once
such as LdV and DFJW, which are split into a data has been collated. The systematic
series of individual programmes, and hermetic dissemination and discussion of results do not
individual programmes. Individual programmes always receive due consideration. Equally,
(e.g. CSV as a component of LdV) may, public debate ensuing from a broad dissemina-
however, conceal many individual measures. tion of the stages and results of evaluation is
LdV and DFJW evaluate individual rare. Transparency and time for discussion of
programmes all of which together constitute the evaluation are, however, much more impor-
the overall programme. Evaluation should be tant for those directly affected and, in particular,
done at the individual programme level implementation centres. This is true of both
(bottom up) because of the variety of formative and summative evaluation. Electronic
programmes alone. As the example of LdV media are particularly useful in such cases. In
implementation in Germany shows, there is order to achieve the intended broad effect,
much worthwhile evaluation, but no agreed target groups should also be included in
design for formative evaluation has yet been dissemination.
developed and agreed by the implementing Apart from these internal optimisation consid-
agencies. Similarly, top-down evaluation still erations, the dissemination of scientific find-
dominates DFJW, where this process is still ings and experiences among a broad public
apparently in favour because of the estab- provides a platform that may benefit future
lished evaluation culture. evaluations.
Transparent programmes such as CSV and The funding arrangements for an evaluation
IFKA are also characterised by bottom-up project should allocate sufficient resources
and top-down effects. When external and for the transfer of results. The production of
political arguments are to the fore, as in the evaluation studies simply destined to gather
case of the CSV evaluation 2000/2001, there dust at the back of a shelf is not desirable
is less emphasis on programme organisers and the fact that a good plan for the dissem-
(Cedefop, NLOs) and participants. The ination of results (in printed, electronic and/or
abstract results of these top-down evalua- conference form) will incur costs is often not
tions make them more akin to consultation considered.
and political justification. The bottom-up 15. Financial framework, evaluation concept and
effect, advantageous to the long-term devel-
political mediation
opment of programmes, is replaced by a
top-down effect. Such summative evaluation Programme planners and funders, particularly
addresses neither participants nor those operating at a political level, want
programme organisers in the realisation of programme evaluation (a) to justify the expen-
programme goals via individual measures. diture of public funds and possibly (b) to opti-
Since evaluation results provide the input for mise programmes by means of precisely
operationalised programme planning, imple- defined goals. The availability of funding has an
mentation and appraisal, which can only be indirect effect on the scale of an evaluation and
successfully achieved with the cooperation of thus on evaluation procedure. To date, (a)
organisers and programme participants, the unequivocally dominates as it is easier to justify
integration of these stakeholders in design politically to the public. A complex ‘interven-
and operation is a significant bottom-up tion’ to ensure programme optimisation, based
factor in the effectiveness (intervention poten- on evaluation results (b) is less convincing
tial) of evaluations. Such a bottom-up proce- within such a context and also more costly.
dure has been adopted in CSV for partici- Equally, funding for evaluation designs which
pants’ reports. must clearly be regarded as falling under (b)
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 153
must be secured by programme organisers pants. Measuring programme output and its
through persistent political pressure and by documentation is a contribution to programme
appropriating arguments largely from (a) (see design which has internal consequences for
IFKA). The consequences of ‘optimising’ a programme organisation and its tools and
programme, such as lower organisational external consequences for political practice.
costs, satisfaction of participants, easier Evaluation is therefore a public affair rather
marketing or durability of programme goals than an activity of government monitoring agen-
achieved, seem to be of little significance in the cies such as an auditing commission. The
allocation of resources. If, however, the public dissemination of results to the (expert) public is
feels politically ‘alarmed,’ as is the case in an important instrument in creating transparency
large-scale assessments of school perfor- and in a public culture of evaluation. These
mance such as PISA, then financial support for results could also lead to evaluations no longer
evaluation is more readily forthcoming. These being motivated by political justification but to
results lead in turn to a direct demand for greater emphasis on the linking of evaluation and
greater dissemination of programme results to intervention in programme optimisation. This new
the politically aware public. situation would also mean that the costs of eval-
uation were regarded as an integral part of
programme costs.
Developing such an evaluation culture presup-
6.3. Prospects for a new poses a minimum of mutual confidence between
evaluation culture and stakeholders. ‘Fear’ of a bad report and an apol-
12 proposals for the ogist reaction to critical evaluation findings are
implementation of evaluation counterproductive to developing this new, inclu-
sive evaluation culture. Everyone can pursue the
The four case studies presented here demon- common goal of programme implementation and
strate very different approaches to the evaluation optimisation. Furthermore, those involved expect
of their programmes. The continual development this from evaluation. Only where a stakeholder or
of evaluation methods in EU mobility programmes a group demonstrably reacts negatively should
stands out. Evaluation has obviously received the question of their replacement be entertained.
more attention and thus greater financial support Complementary to these basic remarks on
so that, given the continuation of this trend, we evaluation culture, this report concludes with
can expect a continual rise in the quality of evalu- twelve brief general proposals containing essen-
ation of VET exchanges within the EU. tial rules for the conduct of evaluation in interna-
The other two programmes presented here tional exchange programmes and how evaluation
(IFKA and DFJW) were pioneers in the design and can support the successful implementation of a
content of evaluation. Today they are at risk of programme.
losing substantial elements due to budget cuts.
IFKA will only be evaluated every three years from Proposal 1
2003, which will prevent data generation immedi- The measures in a programme should be organ-
ately during or after an exchange (by InWEnt ised to enable formative evaluation making evalua-
GmbH), hinder the continuous advising of tion an integral part of programme implementation.
programme organisers by evaluators and call into ‘Formative’ means acquiring knowledge on how to
question the inherent validity of formative evalua- optimise programmes, i.e., on how to plan and
tion. There will, therefore, no longer be any basis implement interventions and disseminate knowl-
for re-interpretation through evaluation. The eval- edge and information to support the enhancement
uation practice of DFJW is equally subject to the of programme outcomes in line with goals.
funding provided by sponsors. Vacillation in this
respect precludes longitudinal investigation. Proposal 2
Today, evaluations are complex processes less Evaluation and scientific research should be interre-
dependent on political justification than on the lated. Scientific work means hypothesis-based and
dialogue between stakeholders such as funders, quasi-experimental work. It supports the acquisition
implementation centres, providers and partici- of knowledge on the relativity of impact factors.
154 Evaluation of systems and programmes
Proposal 3 tive and constructive criteria.) All ‘new’ criteria
No evaluation project can stand alone. Each eval- involve the interaction of all participants/stake-
uation project must refer to knowledge outside holders. Evaluation is no longer seen as an external
the project, that is, to theory. The programmes to approach to the programme being evaluated. Eval-
be evaluated are highly complex. They have uators are instead experts instrumental to the
multiple goals, not all of which harmonise. They programme. Evaluation is thus a continuous inter-
refer to different levels, individual, institutional, active process which can be adapted during all
systemic and cultural, and they offer different phases of programme implementation.
perspectives from different groups of stake-
holders. The consequences are: the same goal Proposal 7
can be accessed in different ways whilst the The paths which lead to a product’s evaluation can
same method may lead to different outcomes be reconstructed by way of an inductive strategy
with distinct effects and thus to possible conflict that should be grounded on theoretical foundations.
situations, with one goal achieved at the cost of This strategy includes the clarification of concepts
another. Fundamental disagreements over the and methods. Evaluations planned in this fashion
possible scale of the impact due to different theo- can contribute to the accumulation of knowledge
retical foundations are a good reason for clarifica- that is useful not only for the programme in ques-
tion of programme strategies. The theory-driven tion but also for future programmes.
approach is motivated by the need to bridge the
gap between description and the need for eluci- Proposal 8
dation in order to optimise the programme. Before the implementation of a programme, it is
important to estimate the intended scale of the
Proposal 4 impact. Measures and evaluation instruments have
Evaluation should be useful for stakeholders. to be selected and developed with this in mind. If
Evaluation must be designed so as to respond to they are not sensitive enough to record outcomes,
stakeholders’ questions about the matter to be they will furnish no information on impact.
evaluated. Because of the normative character of
evaluation, clarification, overt explication and Proposal 9
refinement (operationalisation) are a necessary Before the programme is implemented,
first step in the implementation of a programme. complexity, as reflected by the scale of the impact,
This aids precise definition of the goals, should be taken into account, not only the
measures and expectations of a programme and, intended effects but also side effects that could
ultimately, its practicability. alter or even prevent the success of the outcome.
What conditions must be fulfilled to ensure that
Proposal 5 evaluation is a learning exercise for all those
The theory-driven approach is the basis of gener- involved, including policy-makers?
alisation which draws on the results of evalua-
tions of different programmes. Evaluation, if theo- Proposal 10
retically based, thus supports the accumulation Evaluators should be experts in their specific
of programme implementation knowledge. field, with methodological and strategic knowl-
How can evaluation be used before, during and edge to ensure the usefulness of the evaluation
after a programme or revised implementation to process for those concerned.
feed back into the reform process or to alter the
characteristics of a programme or a reform which Proposal 11
do not appear (any longer) appropriate? Evaluations should be designed so as to guar-
antee interconnectability with other evaluations.
Proposal 6
The shift in priority from ‘objective’ criteria (reliability, Proposal 12
validity, objectivity) to ‘constructive/constructivist’ The practical realisation of evaluation results is to
criteria (communication, intervention, transparency, be regarded as a step in its own right. Interven-
relevance, developing alternatives) is concomitant tion in the form of ‘action research’ should be
with an increase in the applicability of evaluation. programmed and evaluated in an appropriate
(There need not be a contradiction between objec- manner.
List of abbreviations
BA Bundesanstalt für Arbeit / Federal Employment service
BIBB Bundesinstitut für Berufsbildung / Federal institute for VET
BMBF Bundesministerium für Bildung, Wissenschaft, Forschung und Technologie / Federal
Ministry of Education, Science, Research and Technology (from 1995)
BMBF Bundesministerium für Bildung und Forschung / Federal Ministry of Education and Research
(from 1998)
BMBW Bundesministerium für Bildung und Wissenschaft / Federal Ministry of Education and
Science (to 1994)
CDG Carl Duisberg Gesellschaft / Carl Duisberg Society
Cedefop Centre européen pour le développement de la formation professionnelle / European Centre
for the Development of Vocational Training
Comett EU academic and business technology cooperation programme
CSV Community Study Visits Programme for those responsible for vocational training
DAAD Deutscher Akademischer Austauschdienst / German academic exchange service
DeGEval Deutsche Gesellschaft für Evaluation e.V. / German Evaluation Society
DESI Deutsch-Englisch-Schülerleistungen-International / German-English Pupil Performance
International
DFJW Deutsch-Französisches-Jugendwerk/German-French youth foundation
DIPF Deutsches Institut für Internationale Pädagogische Forschung / German Institute for interna-
tional educational research
DSE Deutsche Stiftung für Internationale Entwicklung / German foundation for international
development
EEA European economic area
EFTA European free trade area
ESF European social fund
ETF European Training Foundation
Eurotecnet Initiative promoting vocational training innovation in view of the technological transformation
in the EU.
FORCE Formation continue en Europe. European Community initiative to promote vocational training
among young people in Member States and to prepare them for adult and working life.
GTZ Deutsche Gesellschaft für Technische Zusammenarbeit / German Society for technical
cooperation
IALS International adult literacy survey
IBE International bureau of education
IFKA Internationaler Fachkräfteaustausch (in der Berufsbildung) / International exchange of VET
specialists
IHBB Internationales Handbuch der Berufsbildung / International handbook of VET
ILO International Labour Organisation
InWent Internationale Weiterbildung und Entwicklung gemeinnützige GmbH / International contin-
uing education and development non-profit Ltd.
156 Evaluation of systems and programmes
JCSEE Joint committee on standards for educational evaluation
LdV Leonardo da Vinci (programme) Initiative for introducing EU vocational training policy
Lingua European Community programme promoting language teaching and learning
NLO National liaison officer
OECD Organisation for Economic Cooperation and Development
PETRA Partnership in education and training European Community initiative to promote vocational
training among young people in Member States and to prepare them for adult and working
life
PHARE Poland and Hungary, aid for restructuring of the economies
PISA Programme for international student assessment
SME Small and medium-sized enterprise
TIMSS Third international mathematics and science study
TVET Technical and vocational education and training
Unesco United Nations educational, scientific and cultural organisation
VET Vocational education and training
VmQ Verfahren modularer Qualifizierung / modular qualification system
WSF Institut Wirtschaft- und Sozialforschung (Economic and social research institute)
ZAV Zentralstelle für Arbeitsvermittlung (Central placement office)
References and bibliography
BIBB. Valorisation durch Evaluation. Ergebnisse Cedefop. Katalog – Catalogue 2003. Study Visits.
europäischer Innovationsprojekte in Leonardo Thessaloniki: Cedefop, 2002a.
da Vinci I (1995-1999). Impuls (Leonardo da Vinci), Chen, H.; Rossi, P. H. Evaluating with sense: the
2001, Vol. 1. theory-driven approach. In: Evaluation Review,
Busse, G.; Fahle, K. Lernen durch Mobilität. Unter- 1983, Vol. 7, pp. 283-302.
suchung zu längerfristigen Auslandsaufen- DeGEval – German Evaluation Society. Beywl, W.
thalten von Auszubildenden und jungen Beruf- (ed.) Standards für Evaluation. Cologne:
stätigen im Auftrag von Cedefop. Dortmund: DeGEval, 2002. Available from Internet:
CDG – Carl Duisberg Society, 1999. http://www.degeval.de/standards/standards.htm
Campbell, D. T.; Fiske, D. W. Convergent and [cited 11.12.2003].
discriminant validation by the multitrait- Demorgon, J.; et al. Europakompetenz lernen.
mutlimethod matrix. Psychological Bulletin, Frankfurt: Campus, 2001.
1959, Vol. 56, p. 81-105. DFJW – Deutsch-Französisches-Jugendwerk (ed.)
CDG – Carl Duisberg Gesellschaft. Kompendium Activity report 1993. Bad Honnef: DFJW, 1994.
2001: Leonardo da Vinci: Vermittlungs- und DFJW – Deutsch-Französisches-Jugendwerk (ed.)
Austauschmassnahmen der Antragsrunde Activity report 2001. Bad Honnef, Paris: DFJW,
2001 [Compendium 2001: Leonardo da Vinci: 2002a.
DFJW – Deutsch-Französisches-Jugendwerk (ed.)
placement and exchange measures from the
Richtlinien DFJW/ Directives OFAJ. Bad
2001 round of applications] / Bundesanstalt für
Honnef: DFJW, 2002b (Version of 1.1.2002).
Arbeit – BA. Bonn: CDG, 2001.
EMNID. Die 15–24 jährigen und der deutsch-
CDG. Berufliche Weiterbildung im Ausland:
französische Jugendaustausch: Repräsenta-
Programme 2001. Cologne: CDG – Carl Duis-
tivumfrage im Auftr. d. Dt.-Franz. Jugendwerks
berg Society, 2001.
(DFJW), October 1985; Documentation = Les
CDG; Federal Ministry of Education and Research,
15-24 ans et les échanges franco-allemands de
brochures, manuscripts, etc. (not published).
jeunes. Bielefeld, Paris: 1985.
Bonn, Cologne: 1997 ff.
Europabüro für Projektbegleitung GmbH. Gemein-
CDG; Federal Ministry of Education and Research.
schaftsinitiative Beschäftigung. efp-Leitfaden
Berufsbildung im weltweiten Dialog. 20 Jahre
Projektevaluation. efp-news 7. Special edition.
Internationaler Fachkräfteaustausch in der European Community National Coordination
Berufsbildung. Cologne: CDG – Carl Duisberg Office. Bonn: 1999.
Society, 1998. European Commission, Directorate-General for
CDG; Federal Ministry of Education and Research. Education and Culture. External evaluation of
Internationaler Innovationstransfer und beru- the European Centre for the Development of
fliche Qualifizierung am Beispiel des Interna- Vocational Training – Cedefop. Final Report.
tionalen Fachkräfteaustausches in der Berufs- Brussels: European Commission, 2001.
bildung (IFKA). Berlin, 18–19 October 2001. European Commission. Abschlussbericht über die
Documentation. Cologne: CDG – Carl Duisberg Durchführung der ersten Phase des Gemein-
Society, 2002. schaftlichen Aktionsprogramms Leonardo da Vinci
CDG; Federal Ministry of Education and Research. (1995-1999). Luxembourg: Office for Official Publi-
Weiterbildung ohne Grenzen 2002. Angebote cations of the European Communities, 2000
zur beruflichen Qualifizierung im Ausland. (COM(2000)863 final).
Cologne: CDG – Carl Duisberg Society, 1959. Fahle, K. Die Bedeutung von Auslandserfahrungen.
Cedefop. Annual report 2001. Luxembourg: Office Findings of the Leonardo da Vinci annual
for Official Publications of the European meeting on mobility, 24-25 September 2001. In:
Communities, 2002b. Impuls (Leonardo da Vinci), Vol. 2, 2002.
158 Evaluation of systems and programmes
Friedrich, W.; Schumacher, M. Evaluierung der Nomos Verlag, 2001 (Carl Duisberg Society
Umsetzung des Programmes Leonardo da Vinci series,Vol. 11).
in Deutschland 1995-1999. Study commis- Henze, J. Berufliche Bildung des Auslands: Volk-
sioned by BMBF. Kerpen: BMBF – Federal srepublik China. Baden-Baden: Nomos Verlag,
Ministry of Education and Research, January 1993 (Carl Duisberg Society series: Interna-
2000. tionale Weiterbildung, Austausch, Entwicklung,
Georg, W. Berufliche Bildung des Auslands: Japan. Vol. 7).
Zum Zusammenhang von Qualifizierung und ITB – Institut Technik und Bildung. Die Program-
Beschäftigung in Japan im Vergleich zur mevaluation. Manual. Der Programmträger.
Bundesrepublik Deutschland. Baden-Baden: BLK-Programm: Neue Lernkonzepte in der
Nomos Verlag. (Carl Duisberg Society series: dualen Berufsausbildung. Bremen: ITB Institut
Internationale Weiterbildung, Austausch, Technik und Bildung, 2001.
Entwicklung, Vol. 8). JCSEE – Joint Committee on Standards for Educa-
Göbel, K; Hesse, H. G.; Lauterbach, U. 20 Jahre tional Evaluation (ed.) Handbuch der Evalua-
IFKA. Cologne: Carl Duisberg Gesellschaft, tionsstandards (Second edition). Opladen:
1998c. Leske and Budrich, 2000.
Göbel, K; Hesse, H. G.; Lauterbach, U. Berufsbil- JCSEE – Joint Committee on Standards for Educa-
dung im europäischen Dialog. Studie über das tional Evaluation. Standards for evaluations of
EU-Studienprogramm: Ergebnisse und educational programs, projects, and materials.
Perspektiven. Cologne: Carl Duisberg
New York: McGraw, 1994.
Gesellschaft, 1999.
Jöreskog, K. G.; Sörbom, D. Advances in factor
Göbel, K; Hesse, H. G.; Lauterbach, U. Die beru-
analysis and structural equation models.
fliche Bildung braucht den Dialog mit dem
Cambridge, MA: Abt Books, 1979.
Ausland: der internationale Fachkräfteaus-
Kohler, A. Quality in mobility. Quality award for
tausch (IFKA) und seine Folgewirkungen.
Austrian Leonardo da Vinci mobility projects.
Research study. Baden-Baden: Nomos Verlag,
Presentation at Agora 13, Mobility as a didactic
1998a (Carl Duisberg Society series: Interna-
tool, 29-30 October 2001 in Thessaloniki.
tionale Weiterbildung, Austausch, Entwicklung,
Lauterbach, U. Berufliche Bildung des Auslands
Vol. 10).
aus der Sicht von Ausbildern. Carl Duisberg
Göbel, K; Hesse, H. G.; Lauterbach, U. Kurz-
Society (ed.). Stuttgart: Echo-Verlag, 1984.
Evaluierung des EU-Studienbesuchspro-
Lauterbach, U. Berufliche Bildung des Auslands:
gramms (EUS). Frankfurt: DIPF, 1998b.
Hager, W. Zur Wirksamkeit von Interventionspro- Türkei. Mit einem Vergleich der Gesetzgebung
grammen: Allgemeine Kriterien der Wirk- zur beruflichen Bildung. Baden-Baden: Nomos
samkeit von Programmen in einzelnen Unter- Verlag, 1993 (Carl Duisberg Society series:
suchungen. In: Hager, W.; Patry, J.-L.; Brezing, H. Internationale Weiterbildung, Austausch,
(eds) Evaluation psychologischer Interventions- Entwicklung, Vol. 3).
maßnahmen. Standards und Kriterien: Ein Lauterbach, U. Die Carl Duisberg Gesellschaft und
Handbuch. Bern: Hans Huber, 2000, die Entwicklungszusammenarbeit. In: Greinert,
pp. 153-168. W. D. et al. (eds) Vierzig Jahre Berufsbil-
Hager, W.; Patry, J.-L.; Brezing, H. (eds) Evaluation dungszusammenarbeit mit Ländern der Dritten
psychologischer Interventionsmaßnahmen. Welt. Baden-Baden: Nomos Verlag, 1997,
Standards und Kriterien: Ein Handbuch. Bern: p. 399-408.
Hans Huber, 2000. Lauterbach, U. et al. Jährliches Gutachten zur
Hegenbarth, G.; Stamm-Riemer, I.; Umbsen, P. Die Evaluierung der Maßnahmen des Interna-
Zukunft beginnt mit dem Lernen von heute. Ein tionalen Fachkräfteaustausches. Frankfurt am
Leitfaden zur Qualität transnationaler Aus- Main: German Institute for International Educa-
tauschpraxis. Berlin: Understandingbus, 2001. tional Research, 1995 (mimeo).
Hellwig, W.; Lauterbach, U.; von Kopp, B. (eds) Lauterbach, U. Vergleichende Berufsbildungs-
Innovationen nationaler Berufsbildungssysteme forschung. Baden-Baden: Nomos Verlag, 2003
von Argentinien bis Zypern. Baden-Baden: (Bildung und Arbeitswelt, Vol. 8).
Evaluation of EU and international programmes and initiatives promoting mobility: selected case studies 159
Lauterbach, U.; Göbel, K.; Lanzendorf, U. Finanzier- European Commission under the Leonardo
ung der beruflichen Bildung in Europa, Status da Vinci programme.) Cirius, Denmark, 2002.
Quo und Herausforderungen für die Zukunft. Sedlmeier, P. Jenseits des Signifikanztest-Rituals:
Vergleich unterschiedlicher Finanzierungs- Ergänzungen und Alternativen. In: Methods of
systeme in Europa. European workshop, 16-18 Psychological Research, 1996, Vol. 1, No 4.
June 1997 in Munich. Cologne: Carl Duisberg Available from Internet: http://www.mpr-online.de
Society, 1997. [cited 11.12.2003].
Nijsten, M. Mobilität: Qualität von Ausbildungssys- Stockmann, R. (ed.). Evaluationsforschung. Grund-
temen erhöhen und Austauschmöglichkeiten lagen und ausgewählte Forschungsfelder.
verstärken. Praxisnahe Beratung und Opladen: Leske and Budrich, 2000a.
umfassendes Angebot zum Erfahrungsaus- Thomas, A.; Abdallah-Pretceille, M. (eds) Interkul-
tausch. Bildung für Europa, 2002, Vol. 3, p. 9. tureller Austausch: deutsche und französische
Owen, J. M.; Rogers, P. J. Program evaluation. Forschungen zum interkulturellen Lernen.
Forms and approaches. London: Sage, 1999. Baden-Baden: Nomos Verlag, 1995.
Posch, P.; Altrichter, H. Möglichkeiten und Grenzen Thonhauser, J.; Patry, J.-L. (eds) Evaluation im
der Qualitätsevaluation und Qualitätsentwick- Bildungsbereich. Innsbruck: Studien Verlag,
lung im Schulwesen. Innsbruck: Studien Verlag, 1999.
1997. von Homeyer, E. Mobile Ausbilder schaffen flexible
Rossi, P. H.; Freeman, H. E.; Hofmann, G. Programm- Systeme. Auslandserfahrung für Personal-
Evaluation. Einführung in die Methoden ange- fachkräfte. 20 Jahre Austausch, heute mit 40
wandter Sozialforschung. Stuttgart: Enke, 1988. Ländern. Der Ausbilder, 1998, Vol. 46, No 8,
Rost, J. Allgemeine Standards für die Evaluations- p. 17-20.
forschung. In: Hager, W.; Patry, J.-L.; Brezing, H. Weinert, F. E. Lehren und Lernen für die Zukunft –
(eds) Evaluation psychologischer Interventions- Ansprüche an das Lernen in der Schule. Päda-
maßnahmen. Standards und Kriterien: Ein Hand- gogische Nachrichten Rheinland-Pfalz, Vol. 2,
buch. Bern: Hans Huber, 2000, pp. 129-140. 2000, p. 1- 16.
Sanders, J. R. Cluster evaluation. In: Chelimsky, E.; Widmer, T. Qualität der Evaluation – Wenn
Shadish, W. (eds) Evaluation for the 21st Wissenschaft zur praktischen Kunst wird. In:
century. A handbook. London: Sage, 1997, Stockmann, R. (ed.) Evaluationsforschung.
p. 396-404. Grundlagen und ausgewählte Forschungsfelder.
Schiffler, A.; Hübner, S. Allgemeine Standards für Opladen: Leske and Budrich, 2000, pp. 77-102.
die Evaluationspraxis. In: Hager, W.; Patry, J.-L.; Wordelmann, P. Evaluierung von transnationalen
Brezing, H. (eds) Evaluation psychologischer Praktika. Presentation at Agora 13, Mobility as
Interventionsmaßnahmen. Standards und a didactic tool, 29-30 October 2001 in
Kriterien: Ein Handbuch. Bern: Hans Huber, Thessaloniki.
2000, pp. 141-151. Wottawa, H.; Thierau, H. Lehrbuch Evaluation.
Schultz Pinstrup, A.; Winther, M. (eds) How to Bern: Hans Huber, 1998.
ensure the quality in foreign work placements. A Various electronic documents were also provided
handbook of best practice. (Supported by the by Cedefop (all dated 2002).
