Learning Center
Plans & pricing Sign in
Sign Out




More Info
									  Good Practices and Critical Success Factors in the
  Use of the Online Skills and Competence Analysis
                    Tool (OSCAT)


The e-Assessment of Learning Needs in SMEs project, aimed to develop an
online skills and competence analysis tool (OSCAT) (e-assessment software)
that can be used by trainers in SMEs, training developers and business

The prototype of the OSCAT was tested by a range of user target groups in a
number of the project’s partner countries; the Czech Republic, Hungary, Italy,
Lithuania, and Spain. As a means of learning from the testing process, good
practices and critical success factors in use of the OSCAT have been
analysed and can be considered within the context of approaches to learning
needs assessment.


Only about one third of those who are employed in SMEs actually work in the
profession for which they once trained. The broad range of tasks and
responsibilities in an SME often requires managers and employees to have
competencies in additional fields. A lack of skills and competencies can
quickly lead to quality problems in the company. Competencies are often
acquired through formal learning, a significant cost factor for companies,
however most people build knowledge and competencies through informal
learning methods. As we do not receive certificates and qualifications for
informal learning, it can be difficult for SME employers to gauge the skill and
competence levels of their employees, and therefore it becomes difficult to
identify their development needs. Bearing in mind the difficulties facing
SMEs, we can see that the OSCAT can help to address their problems
because it offers a low cost approach to direct and target-oriented
qualification and skills development.

Two different approaches to the assessment of learning needs are:
• the running of personal interviews within the company; and
• the use of pre-structured online-questionnaires.
These methods imply different critical success factors. The advantages of the
direct personal interview are its individuality and the possibility to adjust
interview questions or to discuss matters in greater detail if something is not
clear. On the other hand, interviews are time-consuming and tend to entail
high costs, which are disadvantages. The use of online-questionnaires on a
stand-alone basis can be an effective low cost option, which can be structured
and evaluated easily. The disadvantages, however, are that they can be quite
rigid, require IT competence, and do not allow the interviewer to intervene or
adapt the questioning approach.

The online skills and competence analysis tool may help to balance the
advantages and disadvantages of the approaches described above. Potential
users in the 5 partnership countries mentioned above tested the prototype tool
between November 2006 and January 2007. 53 people participated in the
testing process, 15 as authors and 38 as users.

Companies from the following locations were involved:
  • 3 companies from the Moravian-Selesian region, Czech Republic
  • 3 companies from Kaunas and Vilnius, Lithuania
  • 2 companies from Szeged, Hungary
  • 3 companies from Castellón, Spain
  • 3 companies from the Turin area, Italy

The companies (all SMEs, with up to 200 employees) represented the
following sectors:
    • Machine building and plant construction
    • Education
    • Union of independent companies
    • Transport (taxi company)
    • Finance
    • Medical Services (hospital)

The testers belonged to the following professional categories:

   Managers, general managers, HR managers, management consultants,
   administrators, professors and lecturers

  Managers, department managers, management consultants,
  administrators, tutors, IT experts, bookkeepers, technicians, nurses and

Analysis and questions

The main aim of the testing process was to assess the user-friendliness and
the functionality of the tool.
Once the testing process was complete, the partners who co-ordinated the
testing completed a questionnaire from which good practice and critical
success factors were evaluated.

The findings of the testing process were as follows:

   •   There had been good cooperation between the project partners and
       the test partners
   •   There had been flexible communication between the parties involved,
       especially concerning the online creation of the testing questionnaires
   •   There had been an immediate evaluation of the analysis
   •   The testing schedule was realistic
   •   There had been good cooperation with the responsible staff during the
       creation of the competence-related-questionnaires
   •   The companies’ levels of motivation had assisted in the success of the
       testing process.
   •   The software had proved to be reliable.
   •   The software ‘user guide’ was easy to understand.

Issues noted:
   • Managers were apprehensive about completing the test
      questionnaires. This may have been due to insufficient briefing before
      commencement of testing.
   • The managers had concerns that there might be resistance to the
      testing process from the users’ side
   • The managers in general were reluctant to fill in questionnaires.
   • The policies and the specifications concerning the use of the ICT in the
      various companies were sometimes a hindrance
   • Administrators and users need a level of IT competence to effectively
      use the tool.
   • Some involvement from IT experts was necessary (at least during the
      initial testing phase)

Conclusion and recommendations

We can conclude by commenting on an example of good practice from the
test phase of this project and making recommendations of practices which will
contribute to the successful use of the OSCAT.

The project’s partner from Hungary, the Chamber of Commerce and Industry,
Csongrad County, co-ordinated the testing process at an SME from which the
testing process and results can be considered an example of good practice in
use of the OSCAT. External consultants performed a competence evaluation
with the help of the OSCAT. The management board helped to develop an
online questionnaire which was then tested by managers and employees.
The self-assessment questionnaires were followed up with interviews during
which the answers given in the questionnaires were discussed. On the basis
of the data collected, the company created a human resource development
plan which defined the learning needs of each employee (including a
schedule and training plan). The management endorsed this plan. All of the
testing participants accepted the result. The company was satisfied with the
services offered by the tool. Moreover, it became clear that the tool could also
be used efficiently for human resource certification according to ISO. The
company has confirmed its intention to use the tool in the future.

Based on our findings during the testing phase, the partnership can
recommend a number of practices to increase the likelihood of successful use
of the OSCAT:

   •   Have IT support specialists available to assist with any internet
       access/ICT related issues that might arise when installing and first
       using the tool
   •   Users without ICT experience will use the OSCAT more effectively with
       an introduction into the tool’s navigation before completing a survey
   •   Administrators (i.e. those working with the back office of the OSCAT)
       will find it easier to use the tool if they have some previous experience
       of windows based applications.
   •   We recommend that users refer to the ‘user guide’ before and during
       use of the OSCAT.
   •   Employers/managers may need to take into consideration employees’
       attitudes to skills and competence evaluation. To gain most benefit
       from the use of the OSCAT, it may be beneficial to brief employees on
       the skills and competence analysis process, the use of the data
       collected, and the benefits the process can offer both the employee
       and the company.
   •   Time pressures create resistance to survey completion. It may be
       beneficial to allocate specific time periods away from employees’ or
       managers’ day to day tasks for them to complete the surveys.

The OSCAT may be used in conjunction with a system for competence
certification in Europe: the Leonardo da Vinci pilot project ESO-CSA
(CEMES). This project elaborated competencies, which are valid on a
European level, in order to assess a manager’s competencies. The
assessment and the analysis are carried out online on the basis of common
questions, case studies, and assessment criteria. Independent assessors
evaluate the competencies of the candidates (users). Both strategies (OSCAT
and CEMES) could be combined in a process for competence assessment
and analysis of learning needs:
 1. Research and analysis of the necessary
    workplace and task requirements

 2. Analysis of education requirements– gap
    between requirementsand competence level

 3. Use of further education opportunities           Competence development

 4. Certification of compentence levelsgained
    from formal, non-formal and informal

Personal interviews and online questionnaires may be combined in order to
analyse individual learning needs. Online questionnaires enable us to collect
data in a structured way and give the respondents an overview about the
subjects that could be relevant for further training. Using the OSCAT, the
competence areas are set by the administrators before the users answer the
questions. This means that we have developed a “qualified” online-interview.
This method is quick and comfortable thus providing the companies with an
economic and cost-efficient solution for the assessment of learning needs.

Source: “Compass for qualification. A guideline for the early assessment of qualification
needs in small and medium-sized companies in the German region of Brandenburg“;
Vanessa Franz, Hanne-Johe-Kelberg, Franz Seibert, Martin Zwick, (isoplan-Institut
Saarbrücken, Berlin, Brussels) July 2002

This project has been funded with support from the European Commission. This publication
reflects the views only of the author, and the Commission cannot be held responsible for any
use which may be made of the information contained therein.

To top