Leonardo Partnership ‘ASK’
Comparative Study
Based on the Survey of Assessment of
non-formally and informally acquired
skills in vocational training across the
participating organisa-tions.
Produced by UK
September 2010
INTRODUCTION
"Knowledge and skills are the most valuable assets to present and future generations, as
governments seek to maintain global competitiveness, increase the flexibility and respon-
siveness of labour markets and deal with issues of population ageing.”
Angel Gurría, OECD Secretary-General, 2009
People are constantly learning everywhere and at all times. Not a single day goes by that
does not lead to additional skills, knowledge and/or competences for all individuals. For
people outside the initial education and training system and for adults in particular, it is very
likely that this learning, taking place at home, at the workplace or elsewhere, is a lot more
important, relevant and significant than the kind of learning that occurs in more formal set-
tings.
1
However, learning that occurs outside the formal learning system is not very well unders-
tood, made visible or, probably as a consequence, appropriately valued, or indeed subject to
any standardized form of assessment. Until the recent OECD’s activity on the recognition of
non-formal and informal learning involving 23 countries on 5 continents, it has also been
under-researched. Most research has focused on learning outcomes from formal education
and training, instead of embracing all types of learning outcomes, thereby allowing visibility
and portability of such outcomes in the lifelong learning system, in the labour market or in
the community. Similarly there is relatively little research into the diversity of assessment of
informally or no-formally acquired skills in the field of vocational training.
And that of course is the focus of our present Leonardo Partnership ‘ASK’, bringing together
the experiences of partner organizations from Bulgaria, Germany, Greece, Italy, Poland,
Slovakia and the United Kingdom.
In 1996, the OECD education ministers agreed to develop strategies for “lifelong learning for
all”. The approach has been endorsed by ministers of labour, ministers of social affairs and
the OECD Council at ministerial level. The concept of “from cradle to grave” includes formal,
non-formal, and informal learning. It is an approach whose importance may now be clearer
than ever and non-formal and informal learning outcomes are viewed as having significant
value. Policy-makers in many OECD countries, and beyond, are therefore developing strate-
gies to make use of all the skills, knowledge and competences that individuals may have,
however they have been learned at a time when countries are now striving against the ra-
1
This introduction has as its basis an article that appeared in the OECD website in 2009 adapted and extended
by KS. No author is cited.
Page | 2
vages of recession to reap the benefits of economic growth, global competitiveness and de-
velopment/advancement of their populations.
Within this context it is relevant to restate the basic definitions of the three different types
of learning that researchers now generally agree to, and that the ASK Project should also
bear in mind.
Formal learning is always organised and structured, and has learning objectives. From the
learner’s standpoint, it is always intentional: i.e. the learner’s explicit objective is to gain
knowledge, skills and/or competences. Typical examples are learning that takes place with-
in the initial education and training system or workplace training arranged by the employer.
One can also speak about formal education and/or training, or more accurately speaking,
education and/or training in a formal setting. This definition has very few detractors.
Informal learning is never organised, has no set objective in terms of learning outcomes
and is never intentional from the learner’s standpoint. Often it is referred to as learning
by experience or simply as experience per se. The idea is that the simple fact of everyday
living constantly exposes the individual to learning situations: at work, at home or during
leisure time for instance. This definition, with only a few exceptions also meets with a fair
degree of consensus.
Mid-way between the first two, non-formal learning is the concept on which there is the
least consensus, which is not to say that there is full agreement on the other two, simply
that the wide variety of approaches in this case makes consensus even more difficult. Nev-
ertheless, for the majority of authors, it seems clear that non-formal learning is rather or-
ganised and can have learning objectives. The advantage of the intermediate concept lies
in the fact that such learning may occur at the initiative of the individual but also happens as
a by-product of more organised activities, whether or not the activities themselves have
learning objectives. In some countries, the entire sector of adult learning falls under non-
formal learning; in others, most adult learning is formal. Non-formal learning therefore gives
some flexibility between formal and informal learning, which must be strictly defined to be
operational, by being mutually exclusive, and avoid overlap.
One of the first things that strikes the author in approaching the daunting task of a compar-
ative study of the assessment of informally and non-formally acquired skills in vocational
training is that unlike the assessment of formal learning which has been well established,
documented and institutionalized in the vast majority of our schools colleges and universi-
ties, the information that partners have shared in this project so far has been very diverse,
variably influenced by each country’s own initial education and training system and depen-
dent upon each country’s motivation and achievement in creating a more effective and pro-
ductive workforce.
Page | 3
THE EUROPEAN CONTEXT AS DESCRIBED BY THE ASK PARTNERS:
THE NATURE OF SKILLS DEVELOPMENT TODAY
Italy emphasised the importance of building on individuals’ skills, particularly if acquired in
an informal way but that would lead to formal qualifications.
Greece referred to the development of skills acquired in the context of family support lead-
ing to professional qualifications.
Poland talked about strength-based learning in their own organisation, where people were
encouraged to develop the skills in which they were already showing proficiency. However,
the country as a whole was still very traditional and formal in its approach to skills acquisi-
tion.
Bulgaria also cited strength-based assessment as being important, though as part of a range
of different approaches. It is the norm for the more informally acquired competences to be
taken to a formal outcome with the older generation, while for younger people there was
greater emphasis on them being encouraged to recognise their own strengths and skills and
taking greater ownership for developing such skills themselves.
Germany highlighted the goal for the majority as being the achievement of formal qualifica-
tions awarded by recognised bodies. However the application of terms such as informally
and non-formally acquired assessment usually depended on the sort of qualification. E.g.
the European Driving Licence (for Information Technology) is very much biased towards
formal learning and formal assessment. However it was far more important to ensure that
people are encouraged to ‘move forward’ rather than their journey always having to con-
form to a label.
UK attached high importance to the way in which outcomes are captured and then as-
sessed, and by what means individuals are given the opportunity to demonstrate their com-
petence in a range of situations. Vocational competency-based qualifications were available
across almost all sectors of the labour market, though in some quarters would still be
judged to be ‘second-class’ qualification, and not having the same value as more formally
acquired skills.
Slovakia remarked how important it is to develop skills in accordance with the labour mar-
ket, while at the same time to synchronize skills acquisition with what happens in the formal
education system.
Traditionalists in all partner countries seemed to suggest that there was still much credence
attached to the value of knowledge as opposed to the application of a combination of
knowledge, skills and values attitudes. However all countries expressed the opinion that
there was now a momentum building towards an acceptance of both informal and non-
formal learning. It would seem that this trend is in turn slowly having an impact on the
Page | 4
methods employed for assessing and verifying vocational skills. There is an undeniable shift
taking place towards greater informality in our schools, training centres and in the work-
place.
THE JOURNEY TOWARDS OPPORTUNITY READINESS AND EMPLOYMENT
The ASK partners discussed these trends at the UK meeting, where there was an eventual
consensus that the ideal journey towards employment or opportunity readiness would in-
corporate the following:
The development of soft skills leading to hard outcomes
A strength-based approach
A whole-person approach
The inclusion of self-assessment together with the need for the input of other
raters/observers.
The journey itself could then be described as beginning with learning, whether formal, in-
formal or non-formal.
The learning was also a combination of knowledge, attitudes, application together with the
relevant technical and social skills required for ‘the job’.
In converting those competences into any form of recognition or qualification, opportunities
then had to be created for the individual to demonstrate or evidence those competences.
This opportunity provided the context for the assessment process.
Certification would follow, which would be recognized or validated either internally by the
assessing organisation or by being further endorsed by an official awarding body – often at
the national level.
The validation process itself would normally be accompanied by an established quality as-
surance procedure, such as internal and external verification and moderation across
ASSESSMENT TOOLS IN USE
The partner organisations apply different assessment techniques while trying to validate the
non-formally or informally acquired competences of their different target groups. These
Page | 5
tools were described in detail by the partnership in a document paper called ‘Validation of
Competences – Survey of Methods and Tools in Use’2.
SIMILARITIES AND DIFFERENCES
When comparing the different assessment tools in use, the partners focused on the follow-
ing features of their own summary:
Germany: In the process referred to as ‘competency analysis’, students were quite happy to
do written tests in literacy and numeracy, though practical assessment was very popular es-
pecially when tasks involved working in a team.
According to the teachers in the Assessment Centres, further training should follow the as-
sessments, based on identified individual strengths and weaknesses.
Greece: Learners liked getting feedback after their assessment, saying it offered them a
sense of direction. It was also important that assessments be validated. This added to their
sense of achievement as well as offering added value to their skills in the form of accredited
certification.
Italy: Partners talked about the training courses that have been carried out in Sardinia for
carers. 90% of the participants were women, in their middle thirties, unemployed and 60%
of them had only a primary school education. The most important element of the assess-
ment process has been the participants’ motivation to carry out the complete process,
Moreover, because that method of learning has been perceived as the means to get a rec-
ognised qualification, they have then felt they would have better opportunities of finding a
job. All the Teachers agreed that a non-validated system or lack of certification would be a
major problem.
Poland: Partners noted that choices were still very much determined by what formal qualifi-
cations were demanded by employers and therefore offered at Further and Higher Educa-
tion levels. However there was now greater movement towards less formal assessment,
which students liked because they were treated as individuals. Tutors found work practice
to be particularly useful for job matching.
UK: Partners commented that responses from both learners and staff had been very posi-
tive, largely because of how the particular College of Further Education selected for the
Questionnaire managed the learning experience, and provided one-to-one feedback and
support throughout their courses. It was also significant that the responses were equally
2
cf. Document paper published on the partnerships website www.lp-ask.org
Page | 6
positive from both the 16 to 18 year old group on short programmes of work-related skill
building, as well as the more mature long-term unemployed group.
Slovakia: Partners reported that a very high percentage of respondents are not clear about
what they want to do in the future. Respondents were happy with the feedback they got,
but significantly a high percentage of respondents did not feel well motivated to participate
in the assessment process. This is probably because for almost 67% of respondents, partici-
pation in the assessment process is mandatory.
It can already be seen that not only are certain similarities emerging from taking the view
across all of ASK’s European partners, but that there are also apparent significant differ-
ences as well.
Perhaps it is that these differences arise through differences of focus, while in fact sharing
the very same intention of providing their learners/ students/trainees with the best possible
opportunities for acquiring the skills, knowledge, and attitudes necessary to go through
whatever ‘hoops’ their national system of qualifications deemed necessary. In this way they
get to be considered ‘worthy’ of an industry-related Certificate or Diploma or National Voca-
tional Qualification – meaning they can then enter the world of work.
Another factor is the very nature and purpose of each partner’s own organisation. The sig-
nificance of this cannot be underestimated because our project is not charged with survey-
ing national picture of each member country. The ASK project remit is simply confined to
partners’ own experience through their individual organisations. What is happening in their
home country is contextually relevant of course, but is not the main subject of this study.
So let’s take a look at the diversity of organisations involved:
In Germany, ZIB – Zentrum für Integration und Bildung GmbH is a coordinating organisation
working in a number of learning centres mainly with young people who require a ‘second
chance’ because of their social or educational circumstances. This client group seems to cor-
respond quite closely to what are referred to in the UK as NEETS – those who up to this
point have ‘not been engaging with education, employment or training’.
In Bulgaria, DBBZ – The Bulgarian-German Vocational Training Centre offers more main-
stream vocational training provision that then feeds into key trades and industries. From its
title we guess that the model adopted owes some features to its German influence and sup-
port.
In Greece, KMOP – Kendro Merimnas Oikoyenias kai Pediou is a Not-for Profit social NGO
working across the country with disadvantaged client groups including the elderly, migrants
and those with mental health issues.
Page | 7
In Italy, Anziani e Non Solo Societa Cooperativa, is a Not-for Profit Cooperative working in
the north of the country with older people, live-in care givers, carers and other vulnerable
people such as migrant women and those in debt. Here, even the definition of competence,
prescribed under Government Degree, is at variance with the definitions used in other coun-
tries, namely comprising technical professional knowledge, operative knowledge and behav-
ioural and relational abilities.
In Poland, The Development and Consulting Centre offers largely traditional training in high
demand skills such as ICT and Technology, whose students/trainees also demand paper
qualifications as a necessary passport into an increasingly competitative labour market
In Slovakia, The Komarno Office of Labour, Social Affairs and Family is as the name suggests
a government department, but has huge responsibility across all strands of the population,
that almost inevitably leads to shortfalls amongst certain client groups like the unemployed,
self-employed and those who are socially excluded. However we are also aware of a thriving
non-formal system of education in the country. It is also perhaps significant to reflect on the
fact that the latest OECD Survey of Higher Education outcomes throughout Europe placed
Slovakia in 5th place compared to the UK that has now dropped to 17th!
In the UK, The Rickter Company Limited is a private company, but one with a very strong
value base. Since 1999 it has worked almost exclusively with Public Sector and Not-for-Profit
organizations in the field of social inclusion. As a consequence it has had the freedom to re-
spond to government initiatives as it chooses, but to do so creatively, providing assessment
training services with products that focus totally on the unique needs of the individual. In
this way the company hopes to avoid the pitfalls of politics and the frenzy of financial mar-
ket volatility. Or at least that is the theory!
It is only fair to say that organisations that exist today in different sectors of provision,
namely the Public, Private and Not-for-Profit/Voluntary and Community Sectors have
evolved very differently, even though they might work with exactly the same client group.
They are likely to have very different organisational structures, are financed differently are
affected differently by government legislation and changing policy initiatives – often made
on the expedient of politics or finance, and often exhibit very different core values. E.g. a
typical private sector company is in business to make money, while the Public Service is a
usually a large scale provider using government/tax payers’ money, and the Not-for Profit
organisation is dependent on a continuous stream of short term grants, charitable dona-
tions and collections, and relies significantly on the altruism and good will of its staff.
No wonder that the learning opportunities these widely different models of provision de-
velop there own approached to offering effective learning opportunities to their clients. No
wonder that the methodology they choose for assessing their clients’ learning is so variable.
And no wonder that a project such as the Leonardo Partnership ASK is so significant in giving
its partners the opportunity to hear about each others work and share the good practice
that goes on - largely unknown outside their own region of operation.
Page | 8
THE STUDENT/TRAINEE/LEARNER AND TEACHER/TRAINER QUESTIONNAIRES
While conducting the survey on different assessment tools in use, the partnership applied
two different questionnaires, one addressed to the ‘Student/Trainee/Learner’, and the
other to the ‘Teacher/Trainer’. The aim was to get an overview of the satisfaction of stu-
dents as well as of the trainers with the specific assessment techniques in use.
These two Questionnaires have already been analysed and presented to us by the Bulgarian
partners, so we know the statistical breakdown of the information gleaned from that exer-
cise. Once again though, it must be stressed that the individuals who gave their responses
were from very different learner groups with varying ages, socio-economic backgrounds,
male and female, of all ages, varying ethnicity, specific needs and specific goals. It is ex-
tremely difficult to give more than a flavour of any patterns or trends that might suggest
themselves to the reader of the gathered data.
Of itself it is a significant data set from six European countries, though more significant for
its diversity than being capable of definitive interpretation.
Excerpt from the questionnaire for learners
Page | 9
However, let´s take an overview of the 329 completed Learner Questionnaires.
Broadly speaking we can catagorise the responses into either positive or negative, based on
the option set. For example, with the question about the respondent’s level of motivation to
participate in the assessment process, answers were to be chosen from ‘very much’, ‘OK’,
‘not really’ or ‘not at all’. We are assuming the first two have a positive bias, and the latter
two a negative one. This simple protocol can then be applied throughout the Questionnaire.
Filling the questionnaires in an interview and in the classroom
On this basis we can deduce that the majority of respondents in all countries agreed that:
The assessment was voluntary
They were motivated
They understood what they had to do
They were happy with the assessment
The feedback they got was sufficient and helpful
They were clearer about what they wanted to do in the future
They were clearer about their strengths and skills
They were clearer about the resources available to the
The assessment was free.
However the responses from Slovakian learners had a significantly different distribution
pattern from all other partners. This pattern can be seen across all questions where there
were four options to choose from. Bulgaria showed a similar but less exaggerated pattern
across the same questions. We do not have any evidence to offer any interpretation of this
fact at this time. Perhaps such detail can be discussed with partners at our forthcoming
meeting in Bulgaria.
Page | 10
The satisfaction levels were particularly high across these same questions for Italy. The Ital-
ian partners suggest a combination of reasons for this result:
a) The relationship between learners and their assessors was well nurtured
b) There was significant consistency in terms of preparation, support and quality of
feedback
c) There was a strong connection between skills validation and training, a peculiarity of
this particular skills validation system that enables learners to rapidly understand
and plug their skill gaps
d) Many of assessment tests are directly linked to the practical activities handled by
learners on the job.
Any of these influences would be significant in determining examples of good practice that
might then be shared and implemented across the partnership.
It was also heartening to note that the percentage of respondents choosing the most nega-
tive category only reach between 1.5% and 3.9% for any question and across all countries.
This suggests that being actively involved in an assessment process which we have heard
from all partners is based on what the learners what seems also to equate with what works
– even though in some countries the methodology is more or less formal.
Moving to an overview of the Trainers/Teachers responses to their questionnaire, there
was far more variation across partners.
The only justifiable comparisons that can be made are in respect of the following:
All countries with the exception of Slovakia said categorically that their assessments
could be validated.
All countries were happy with the resources that were available to them
Only Germany and Slovakia responded 100% that their organisations staff training
was sourced externally, while all other sourced staff training both internally and ex-
ternally
100% of Slovakian respondents felt their students were neither motivated nor en-
thusiastic. In contrast 100% of the UK respondents thought the opposite. How do we
account for this polarized type of response? Does it reflect the effort that is put into
creating a stimulating learning environment?
The only thing that can reasonably be said about the make-up of the assessment
process in every country is that it is a complete mix of elements from the very for-
mal essay writing to the greater informality of a group exercise.
Similarly the type of competences covered by partner organisations’ assessment
processes typically covered the full range of working with others, working with liter-
acy, numeracy, ICT, manual and technical skills, social skills, cross cultural compe-
tence and ‘other’.
Page | 11
All partners used a combination of assessment techniques, although Slovakia only
used two: profiling and competence analysis
Italy was typical of partners using multiple assessment techniques, in their case mul-
tiple choice tests to assess theoretical knowledge; specific written or practical exer-
cises to assess technical skills and the Behaviour Event Interview based on a case his-
tory to assess the inter-relationship skills.
Only Germany offered just the assessments that they themselves had developed. All
others used a combination of in-house materials and externally developed assess-
ments.
Similarly Germany is the only partner where 100% of its respondents said that their
assessments that are fully recognised nationally.
Surprisingly 100% of the Italian respondents said that their assessments were not
recognised nationally. This is due to the fact that in Italy, the validation system of the
skills acquired on the job is regulated at the regional level. Also certain regions have
not yet incorporated a skills certification system. For that reason the ANS assess-
ments aim to validate and recognize learners’ skills in order to strengthen their com-
petences for the job market and integrate their vocational training experience. This
validation and certification method does not aim to achieve a qualification because
in Italy domestic care work is currently not recognized as a vocational qualification at
the national level.
There was a mix of responses from all others.
CONCLUSION
It is clear that what partner organisations are currently offering in terms of assessment
processes seem to be perceived generally by students/ trainees/learners and teach-
ers/trainers alike to be ‘fit for purpose, in meeting both the expectations and needs of all
concerned. This is very good news in itself.
Beyond the relatively small and certainly diverse data set, we must still strive towards find-
ing what we all need to do not only in creating appropriate and motivating opportunities for
learning, but to match the effectiveness of that provision with excellence in the assessment
of learning. Indeed a good assessment can itself be a learning experience.
We perhaps need to think more:
about monitoring the individual’s journey of learning, application, experience and
reflection,
taking into account both the hard outcomes of competence achieved in specific skills
of even of exams passed as well as the softer indicators like confidence and self-
esteem,
Page | 12
that validation means finding the evidence of the individual’s journey and the meas-
urement of the distance travelled,
and finally, that such processes need to be moderated and quality assured.
But that is what we shall be moving onto next.
Keith Stead
The Rickter Company
September 2010.
Page | 13