customersatisfactionreport by PlazaVietLong


									Sieu thi dien may Viet Long -


       Customer Satisfaction

                              A Centre of Excellence in ATM Training

                                                                                  Copyright Notice
              © 2006 European Organisation for the Safety of Air Navigation (EUROCONTROL). All rights reserved.
              No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic,
              mechanical, photocopying, recording or otherwise.
              Any other use is subject to prior written consent by EUROCONTROL. Request shall be addressed to:

              Director of the Institute of Air Navigation Services, 12, rue Antoine de Saint-Exupéry, L-1432 Kirchberg/Luxembourg.

The EUROCONTROL Institute of Air Navigation Services aims to provide the services that you want
and to make your stay in the Institute as enjoyable as possible. All Institute personnel are there to
ensure that your stay at the Institute is successful. However, if you do have a complaint (or a
compliment) please tell us. If you are not satisfied with the service we provide or you would like to
propose an improvement then please fill out the form at,
or contact directly.

 ii                                           © Copyright 2006 EUROCONTROL
Table of Contents
INTRODUCTION..................................................................................................................................... 4

ATM TRAINING ...................................................................................................................................... 5

ATC TRAINING .................................................................................................................................... 12

E-LEARNING ........................................................................................................................................ 13

STAKEHOLDER FEEDBACK.............................................................................................................. 14

COMPLAINTS ...................................................................................................................................... 15

CONCLUSION ...................................................................................................................................... 16

Printed on 30/11/2004                        Produced by the Business and Administration Unit                                                        3
        Customer Satisfaction Report

This is the 7th report that the Institute devotes to the     •   A Student Evaluation Form filled in by the
assessment of the level of satisfaction of its                   students at the end of a course. This evaluation
customers. The study on which this report is based               form is used for both the ATM course
has been conducted in the framework of the quality               participants and ATC course participants.
improvement programme implemented at the                     •   An E-Learning Evaluation Form filled in by a
Institute since end of 1996.                                     student upon completion of an E-Learning
Measuring customer satisfaction on a regular basis           •   A generic Stakeholder Feedback Form filled in
is one of the key elements of the quality strategy               by various stakeholders. The form is available
implemented by the Institute since 1997.                         at
Our objective is to understand exactly how our         
customers perceive the Institute and, based on their
level of satisfaction, to improve our services and
products.                                                    Customer Visits
This report proposes a summary and an analysis of            IANS visits a sample of its customers on a yearly
the data collected in 2005.                                  basis. The customer visits serve as a main source
                                                             of feedback on existing products and a source of
The customer segments                                        information for improvements and new products.
                                                             The customer visits include visits to Air Navigation
assessed in 2005                                             Service Providers and civil and military State
IANS applies a policy to measure the satisfaction of
its customers with all its products and services.
These include:
                                                             Complaint Procedure and
•       The ATM course participants
•       The ATC course participants
                                                             Complaint Form
•       The E-Learning Students                              IANS uses an anonymous form, available on its
•       Training Sponsors                                    web-site for the collection of complaints related to
•       Consultancy Bodies.                                  its products, services and staff. A reference to the
•       User Groups                                          complaint procedure and associated form is made
                                                             in all products and services. The complaint form is
Assessment Tools                                             available                                         at
The assessment is based on a set of assessment
tools and measurement methods. During 2005
further work has been done to increase the integrity
of the data collection.

Evaluation Forms
To measure the satisfaction of its customers with
the products and services IANS uses the following
evaluation forms:

    4                                       © Copyright 2006 EUROCONTROL
ATM Training
The Satisfaction of Students with the ATM Training Products is measured through the use of a Course Evaluation

At the end of each
(classroom) course,
students are presented
with an evaluation form
containing a number of
criteria to be assessed.
Each criterion can be
given a result between 1
and 4 meaning
unacceptable (1),
unsatisfactory (2), good
(3) an excellent (4).
The complete set of
criteria are grouped in
five main categories:
•   Training Content
•   Instructional Team
•   Documentation
•   Equipment
•   Administration

The first three categories
are directly related to the
course. The last two
categories are directly
related to the supporting
environment and

                                                  Figure 1 Student Evaluation Form

Printed on 30/11/2004            Produced by the Business and Administration Unit                            5
     Customer Satisfaction Report

From Forms to
Results                                             Measuring Satisfaction
We map the marks of the
students on a scale from
0 (all students mark           Marks from Students
Unacceptable) to +100
(all students mark                  Unsatisfactory         Satisfactory                      Good    Excellent
Excellent). When all
students mark Good                          1                     2                           3         4
then the result is 75 and
when students mark, on
average, between Good
and Excellent then the
result will be around 85.
Likewise, if students fill     Result
in 'all-Uncacceptable'                       0                 25                             75        100
then the result will be 0.
If students fill in 'all-Un-
                                           All 1               All 2                         All 3      All 4
Satisfactory' then the
result will be 25.

                                                               ©Copyright 2003 EUROCONTROL                        1

                                                   Figure 2 How we measure satisfaction

IANS aims to meet our objective of being a Centre
of Excellence in ATM Training. For us this means             Results
we aim to achieve results on any criterion between
75 and 100. As an example: the overall average (all          The results presented below are extracted from the
courses, all criteria, 3090 forms) for IANS in 2005 is       automated student satisfaction evaluation system.
87.5 – compared to 85 in 2004. The highest result            Results can be presented per category, per training
on a single criterion is 95 (Subject Knowledge of the        domain and per criterion. We have selected the
Instructor). The lowest result remains pre-course            most significant results to be included in this report.
information (76.5 – compared to 70 in 2004).

 6                                        © Copyright 2006 EUROCONTROL
                                                                                              Size of the
                          3500                                                                Population
                                                                                              This graph shows the
                          3000                                                                number of evaluation
                                                                                              forms received and
                          2500                                                                processed by IANS. In
                                                                                              2005 the Institute
                                                                                              continued to optimise the
                          2000                                                                processing of the forms.
                                                                                              This led to another
                                                                                              significant increase of
                                                                                              the number of processed
                          1000                                                                forms in 2005.


                                   2002            2003       2004          2005
         Number of Forms           1937            1915       2624          3090

                        Figure 3 Number of Evaluation Forms

Composition of
the Population                                                                         1298

Students indicate on the
evaluation form which
type of staff they are.
This chart shows that a                     2005
majority of Students at                                                                             693
IANS in 2005 are                                                     600
Operational and                  600
Technical Staff.
These results are                400                          345
consistent with the
results of the previous                            223
years.                           200

                                          Administrative     Managerial         Operational         Technical

                                   Figure 4 Composition of the population (number of students)

Printed on 30/11/2004               Produced by the Business and Administration Unit                                  7
    Customer Satisfaction Report

                       100                                                      This chart shows the
                                                                                satisfaction of the
                                                                                students with the ATM
                            90                                                  courses. The result is
                                                                                obtained by averaging
                            85                                                  the results obtained on
                                                                                every criterion in the
                            80                                                  categories Training
                                                                                Content, Instructional
                            75                                                  Team and
                                                                                A result of 87 indicates
                            65                                                  that, on average,
                                                                                students mark criteria
                            60                                                  between Good and
                                                                                The increase in
                            50                                                  satisfaction has been
                                   2002        2003        2004        2005     achieved through
                                                                                detailed analysis of the
         Course Results                83      84,5         86         87,5     results of each course
                                                                                and introducing
                                                                                corrective action
                             Figure 5 ATM Course Results                        wherever required.

                                                                                Segmented View
                                                                                This chart presents the
                      95                                                        results per measurement
                      90                                                        Category. These are
                      85                                                        Training Content,
                                                                                Instructional Team and
                                                                                IANS has improved the
                                                                                results on each of these
                      65                                                        three categories during
                      60                                                        the last four years.
                                                                                IANS will continue to
                      50                                                        analyse the results and
                                 2002         2003        2004         2005
                                                                                make improvements
         Content                 81,5          83          84,5        86,5     where possible and
         Instructor               88           89          91              92   practicable.
         Documentation            80          80,5         83,5            85

                 Figure 6 ATM Course Results: per Category

8                                           © Copyright 2006 EUROCONTROL
‘Best’ Results
Customers often praise                               100
IANS for its staff and
excellent infrastructure.                            90

This informal feedback is
confirmed by the course                              80
evaluation results. The
criteria ‘Instructor                                 70
Subject Knowledge’ and
‘Classroom Equipment’                                60
have consistently
received the highest                                 50
marks from the students                                       2002         2003         2004           2005
during the last three
                                       Instructor Subject     92,5           93         94,5           95,5
IANS will continue to                  Classroom              88,5           89         90,5            91
invest in the competence               Equipment
of its instructors by
continuation training and
competency assessment
                                                     Figure 7 Criteria with ‘Best’ Results
and it will maintain the
building at its high

                                                                                               Areas for
                        100                                                                    Improvement
                        90                                                                     IANS continuously looks
                                                                                               for opportunities to
                                                                                               improve. Criteria with the
                                                                                               less positive results are
                                                                                               the Pre-Course
                                                                                               Information and the
                                                                                               Course Duration.
                                                                                               In 2004 IANS has put in
                        50                                                                     place an action plan to
                              2002           2003           2004         2005                  improve the Pre-Course
          Pre Course          61,5             64           70,5          78                   information and
          Information                                                                          significant progress has
                                                                                               been made in this
          Course Duration      71              74            76           76
                                                                                               domain in 2004 and
          Canteen             73,5            72,5           75          77,5                  2005.
                                                                                               The optimum duration for
                                                                                               each course will be
                  Figure 8 Criteria with less positive Results                                 reviewed on a course-
                                                                                               by-course basis.

                                                                                               In 2005 IANS changed
                                                                                               the layout of the coffee
                                                                                               area and installed new
                                                                                               equipment in the kitchen
                                                                                               to improve the flow of

Printed on 30/11/2004                Produced by the Business and Administration Unit                                     9
      Customer Satisfaction Report

Focus on Safety
IANS is heavily involved
in the development and                  95
delivery of Safety                      90
Courses to support                      85
Safety Regulation and                   80
Safety Management.                      75
In 2005 the number of                   70
courses delivered in the                65
Safety domain increased                 60
from 36 to 50 and the                   55
number of participants                  50
increased from 614 to                               2003               2004            2005
844.                                 Safety          86                88,5             90
We are pleased to note               Overall        84,5                86             87,5
that the results in this
relatively new area for                        Figure 9 Safety Domain Course Results
IANS equal the average
obtained in other, more
mature, training
It is a clear indication
that the rigorous
development approach
at IANS leads to
products that meet
customer expectation.

        100%                                                                   Loyalty
                                                                               In 2004 we introduced a
         80%                                                                   new question on the
         70%                                                                   evaluation form: ‘Would
         60%                                                                   you recommend the
         50%                                                                   course to a colleague’.
         40%                                                                   In 2005, like in 2004,
         30%                                                                   more than 90% of the
                                                                               students would
                                                                               recommend the course
                                                                               that they attended to a
          0%                                                                   colleague.
                       Yes                     No             Blank
        2004           91%                     1%               8%
        2005           92%                     1%               7%

               Figure 10 Would recommend course to a colleague

 10                                     © Copyright 2006 EUROCONTROL
                  10,00%                                                             dissatisfaction
                   9,00%                                                             To measure
                                                                                     dissatisfaction IANS uses
                   8,00%                                                             the percentage of
                   7,00%                                                             ‘unsatisfactory’ marks on
                                                                                     the evaluation forms.
                                                                                     This chart shows the
                   5,00%                                                             proportion of un-
                                                                                     satisfactory marks on the
                   4,00%                                                             evaluation forms for all
                   3,00%                                                             criteria.

                   2,00%                                                             The excellent results in
                                                                                     2005 are spread over the
                   1,00%                                                             entire spectrum of criteria.
                   0,00%                                                             Note that that an
                               2002         2003           2004          2005        ‘unsatisfactory’ mark on
                               0,91%       0,90%          0,83%         0,59%        one criterion does not
         Percentage of                                                               necessarily mean that the
         'Unsatisfactory'                                                            student is not satisfied
         Results                                                                     with the whole product.

                        Figure 11 Level of dissatisfaction

Printed on 30/11/2004             Produced by the Business and Administration Unit                             11
         Customer Satisfaction Report

ATC Training
The satisfaction of ATC ab-initio students is                   •   Last but not least, the ATC ab-initio students
measured using the same evaluation as the system                    are evaluated at the end of the course. At the
used for measuring the satisfaction of the ATM                      end of the evaluation they either pass or fail.
students. Satisfaction figures of ATM and ATC                       The outcome of the evaluation tends to have a
students must however not be compared. The                          significant impact on the satisfaction of the
context of these courses, although delivered in the                 students.
same building, is completely different:                         IANS believes that the critical success factor for
•        The duration of ATM courses is maximum of              ATC students is their validation rate (i.e. the
         two weeks whereas ATC have a duration of               proportion of students that obtain an ATC license).
         several months.                                        We do however present the satisfaction data in this
•        The student population of ATM courses are              report as it is part of our quality management tools.
         staff members from service providers and
         member states.
•        The student population of ATC courses are ab-
         inito ATC students
                                                                                         ATC Student
                            100                                                          Satisfaction
                            90                                                           This chart shows the
                                                                                         satisfaction of ATC ab-
                            80                                                           initio students.

                            70                                                           The last few years ATC
                                                                                         ab-initio training has been
                            60                                                           characterised by high
                                                                                         rotation of staff and
                            50                                                           continuous changes to the
                                   2002       2003        2004            2005           training content and
              Training Content     79,5        79          75              78            structure as a result of
                                                                                         variations in demand and
              Instructors          76,5        73         72,5             77            new regulatory
              Training              78         69          76              78            requirements.
                                                                                         In 2005, IANS significantly
                                                                                         improved the
                                                                                         documentation and
                            Figure 12 ATC Student Satisfaction                           training of the ATC
                                                                                         Compared to 2004 the
                                                                                         satisfaction of the
                                                                                         students has increased for
                                                                                         each segment.

    12                                      © Copyright 2006 EUROCONTROL
IANS introduced satisfaction measurement for its e-            In 2005, students filled in 2774 forms (800 in 2004)
learning products in the middle of 2004. Students              evaluation forms were filled in. 90% of the students
are given an opportunity to evaluate each e-learning           identified to be ‘satisfied’ with the modules and
module at the end of the module. The figures                   more than 88% of the students indicated that the
presented in this report cover all the modules on the          module met the objectives.
EUROCONTROL IANS on-line TrainingZone,
including the CFMU modules.                                    These figures indicate that the overall satisfaction
                                                               with the e-learning modules is excellent.

1.   Why are you studying this module?
          Personal reasons (e.g. to further career, out of personal interest etc).
          My employer decided I should.                                                Evaluation Form
          It was a prerequisite for a class-room based course or programme.
                                                                                       A customised form was
2.   The module                              agree     not sure-n/a     disagree       developed to capture
     followed a logical sequence,              O             O              O          student satisfaction with
     was not too easy or too difficult,        O             O              O          the e-learning modules.
     was clear and easy to understand.         O             O              O          The form collects
3.   The module was                          agree     not sure-n/a     disagree       information related to the
                                                                                       ‘reasons for studying’, the
     easy to use,                              O             O              O          complexity of the module,
     interactive and interesting.              O             O              O          the objectives of the
4.   In order to successfully access this module, did you have to                      module and the impact of
          change your browser (Internet Explorer) settings?                            the module on the
          install new software (flash plug-in, java plug-in, internet explorer)?       student’s job.
          disable a popup-killer or anti-intrusion programme (e.g. Norton Internet
5.   The module                              agree     not sure-n/a     disagree
     met the objectives set at the beginning,O               O              O
     allowed me to check my understanding. O                 O              O
6.   Did you enjoy studying this module:                  O yes           O no
7.   Will this module
         Help you directly in your current job?
         Help you in a future job?
         Increase your understanding of the industry without direct impact on your
         current/future job?
         Help you personally (e.g. out of personal interest)?

                                                                                        Structure of E-
     100%                                                                               Learning Modules
                                                                                        This chart shows the
                                                                                        percentage of students
     70%                                                                                that answered ‘yes’ on the
     60%                                                                                following questions:
                                                                                        • Structure of the module
     40%                                                                                  meets expectations.
                                                                                        • Level of the module
     20%                                                                                  meets the expectations
                                                                                        • Module uses clear
                   structure                level              clarity of language        language.
     2004               92%                 73%                       86%               These results indicate that
     2005               95%                 80%                       87%               the structure and content
                                                                                        of the E-Learning modules
                                                                                        meet the needs of the
                 Figure 13 Structure of E-Learning Modules                              students

Printed on 30/11/2004              Produced by the Business and Administration Unit                              13
     Customer Satisfaction Report

                                                                      Origin of E-
                        45%                                           Learning
                        40%                                           Students
                        35%                                           It is IANS’ strategy to
                        30%                                           integrate e-learning in
                        25%                                           well defined training
                                                                      programmes – defined
                        20%                                           and implemented by the
                        15%                                           student’s employer - and
                        10%                                           as pre-requisite modules
                                                                      for classroom courses.
                                                                      These results indicate
                                    2004                    2005      that the proportion of
                                                                      students registered
          Self-driven               43%                     28%
                                                                      through their employer or
          Employer-driven           30%                     39%       studying as a pre-
          Prerequisite              27%                     33%       requisite module has
                                                                      significantly increased
                                                                      compared to the number
                                                                      of students that study out
           Figure 14 Why did students study e-learning modules        of their own initiative.
                                                                      The number of students
                                                                      that see a direct impact
                                                                      on their job from
                                                                      studying e-learning
                                                                      modules has increased
                                                                      from 56% in 2004 to
                                                                      68% in 2005.

14                                     © Copyright 2006 EUROCONTROL
Stakeholder Feedback
                                                               improve our communication with the members of
Training Consultation Group                                    the consultation group.

The Stakeholder Feedback Form was distributed to               Other stakeholders
the meeting participants at the end of each Training
Consultation Group Meeting. The majority of the                The stakeholder        evaluation   form   was   also
feedback indicates that the overall perception of the          distributed to:
Institute lies between Good and Excellent (on a
scale unsatisfactory – satisfactory – good –                   •    Participants of the training focus group.
excellent).                                                    •    The training partners at IATA.
                                                               The results indicate an equally high level of
Compared to 2004 the results related to                        satisfaction of working together with IANS.
communication (content, format and timeliness)
have improved. We will continue to review and

Customer Visits
The Customer Visits are a process in which IANS                •    ATSEP Training: Many of the visited
meets stakeholders in member states on a bi-lateral                 stakeholders are currently in the process of
basis. It is a mechanism to collect qualitative                     defining basic training for Air Traffic Safety
feedback from various stakeholders. The countries                   Electronics Personnel (ATSEP). Some
visited in 2005 are: Austria, Belgium, BIH, Bulgaria,               collaboration with EUROCONTROL and other
Greece, Hungary, Spain, Sweden and Turkey                           member states could avoid duplication of effort.
                                                               •    On-Site Courses: In general the demand for
During the meeting of the Training Consultation
                                                                    on-site course (i.e. courses at stakeholder
Group in September 2005 it was agreed that IANS
                                                                    premises) exceeds the capacity of IANS.
would include in its Customer Satisfaction Report
                                                               •    Although most stakeholders are aware of the
information gathered during the Customer Visits.
                                                                    E-Learning products of the Institute many of
During these customer visits, stakeholders have in                  them have not yet been able to assess the
general expressed a high degree of satisfaction with                feasibility to use the products available.
IANS, its staff and the services it offers. The                •    ATC Training Notes: IANS should develop a
following points have been raised:                                  policy to distribute CCC aligned ATC Training
•   OJTI: The delivery of OJTI courses through                      Notes.
    partnerships is often perceived as an
    expensive solution (by stakeholders receiving
    the services). The solution is not perceived as
    entirely satisfactory.

Every training note distributed to students and all            compliment) please tell us. If you are not satisfied
pages on the On-Line Trainingzone contain a                    with the service we provide or you would like to
clearly visible reference to IANS’ complaint                   propose an improvement then please fill out the
procedure. The procedure enables stakeholders to               form                                               at
submit complaints in an anonymous way or an          ,   or
identified complaint through e-mail. The text is as            contact directly.
The EUROCONTROL Institute of Air Navigation                    Although the procedure is clearly marked in all
Services aims to provide the services that you want            documentation, IANS did not receive formal
and to make your stay in the Institute as enjoyable            complaints in 2005.
as possible. All Institute personnel are there to
ensure that your stay at the Institute is successful.
However, if you do have a complaint (or a

Printed on 30/11/2004              Produced by the Business and Administration Unit                               15
      Customer Satisfaction Report

IANS aims to be recognised by its stakeholders a           satisfaction of the ATC Ab-initio students with the
Centre of Excellence in ATM Training. We                   ATC training started increasing again in 2005. In
continuously improve our products and processes.           2005 we have further exploited satisfaction
We measure the satisfaction of our customers with          measures for the E-Learning Products and for the
the products that we provide.                              Stakeholders that we work with. The results in these
                                                           domains show a very high degree of stakeholder
This document presented the results of the                 satisfaction.
satisfaction measurement in 2005. The satisfaction
with our ATM Training Products shows an upward
trend in all domains for the fourth year in a row. The
                                                                                            END OF DOCUMENT.

 16                                       © Copyright 2006 EUROCONTROL

To top