Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management The STCW 95 amendments cover three important areas Responsibilities of shipping companies Uniform standards of competency – For the first time uniform standards of competency have been spelt out in the various maritime functions. The phrase “To The Satisfaction Of The Administration” has been replaced by more specific terms. Each element of competency contains specific criteria for standards of knowledge, understanding and proficiency. Methods of demonstrating competency and the criteria for assessing the same has also been discussed. ASSESSMENT : Assessment is an evaluation or an estimate of a Performance Measure as compared against a Performance Standard. STCW 95 determines the competency required at the various levels of operation and management. Competencies are then broken up into training outcomes or tasks for which a performance objective has been set. Against these objectives a minimum performance standard has been defined for each task and subtask. If a seafarer can prove that his/her knowledge or performance meets the minimum standard then that seafarer is deemed to be competent “To acquire that outcome or to perform that particular task”. When the seafarer proves to be competent in all the competencies required to be carried out by him/her under his/her function, that seafarer will be entitled to a Certificate of Competency for the given level. It is the aim of STCW95 that every in-service skill based competency should be divided into tasks and subtasks to such an extent, that the assessment of the same becomes very specific and unique, and is thus a measurable quantity leaving no element of doubt as to the seafarers competency. IT IS NOTED THAT TILL DATE IMO HAS IDENTIFIED THE COMPETENCIES FOR THE VARIOUS FUNCTIONS HOWEVER THE PERFORMANCE OBJECTIVES, PERFORMANCE MEASURES AND PERFORMANCE STANDARDS FOR EACH COMPETENCY HAS YET TO BE DETERMINED UNDER A STANDARD FORMAT.
Page 1 of 11
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management In order to carry out training and assessment the qualifications required by the instructors, supervisors and assessors has been adequately spelt out. On board training and assessment is necessarily to be carried out via a structured training programme approved as meeting the requirements of section AII/1 and AIII/1 under close supervision and monitoring by qualified officers aboard the ship in which the sea going service is performed. All activities must be adequately documented in a training record book or similar document, which needs to be backed up by a structured assessment package. Similar assessment must be carried out by training providers by comparing training objectives against training outcomes and evidence of the same will be maintained. FORM OF ASSESSMENT: Right up till the recent past a sailor went on board a vessel as an apprentice and learnt the ropes and rudiments of navigation and ship handling from his seniors. He was then formally assessed and awarded a certificate of competency by a government officer far remote from his place of work, who in fact had no means except by viva voce to establish if the candidate was competent or not. In effect he set certain conditions and based his questions on the same and the answers received from the candidate had necessarily to suit a pattern. Most candidates learnt the answers to the questions by rote and gloriously passed the viva and attained their certificates. Could or would these officers actually perform as mentioned in the answers was something the surveyor had only to hope for. Knowledge and skill in anti collision was namely judged by using wooden ship models, the magnetic board and picture cards. Once again a real life scenario could not be depicted by the surveyor and the surveyor was then left to judge the candidate only by gut feeling and instinct as to weather the person would actually perform as required. On board training and assessment has always been a good method of enhancing skill and judging competency but falls short due to the certain natural constraints. The officer is dependent on his luck to get a benevolent senior who is willing and able to teach and assess skills. In most cases the seniors themselves have not had any formal training and thus what they impart would still be subject to much variance due to different levels of understanding and varied perceptions.
Page 2 of 11
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management A senior officer may not be willing to put a junior in charge of the actual ship handling of the vessel since the consequence of an error not rectified in time could be phenomenal and the officer cannot bear that risk. Weather and other external conditions may not always be conducive to training. Opportunities would always be limited due to time constraints and persons own busy schedules. Till date there is no formal authority being given to a master nor any official weight to his assessment and this could be a de-motivating factor. Some administrations have in the recent years found it fit to insist on mandatory post sea training and assessment by approved colleges and institutes prior to Assessment by the Administration – however the onboard training and assessment which STCW-95 has broadly defined, has not been given its due importance. This is the point at which the use of simulators becomes most effective. FUNDAMENTALS OF ASSESSMENT: Assessment can be done by five different methods. The table below will enlighten us with some salient features along with the strengths and weaknesses of the individual methods. Method 1. Written examination. Strengths Valuable tool for assessment in areas where knowledge forms a key component of competent performance. Well designed questions can be standardized. Elicits key knowledge / understanding in short timescale. Weakness Only assesses ability to write and construct written material. Key issues Danger of assumption that “knowing” means “able to do”
2. Multiple choice questions
Always a 25% possibility of correct answer being chosen at random. (Given four choices)
Time and skill needed for design, delivery and marking. Supplementary evidence only – not direct evidence of actual performance.
Page 3 of 11
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management 3. Observance of performance Provides high quality evidence of competency. Provides continuous assessment. A useful tool for generating evidence where opportunities for assessment across the full range are limited or prohibited by health, safety commercial reasons. Test conditions can be standardised for skill tests. Valuable tool for collecting evidence across full range of activities. Opportunities to demonstrate competency across full range of activities may be limited. Removed from realistic working conditions. Individuals react differently under test conditions. Structure of assignments and projects often not adequately planned. Evidence collected by this method alone would not be sufficient to assign competency. Least likely to reflect or represent real working conditions. Need for trained assessors. Need for clear lines of communication and quality assurance measures. Need for planning and structure.
4. Specially set tasks: skills, tests, simulations, projects, assignments.
5. Oral questions – Viva
Need for trained assessors with effective questioning techniques. \Requires largest inferential jump to assigning competency.
STCW mentions simulators as one of the means to prove competency. Chapter II, III and IV of Code-A list down the competencies in the forms of tables required by deck, engine room and radio personnel at management and operational levels. These competency tables enumerate multiple means to prove competency and approved simulator training is mentioned at numerous occasions in parallel with in-service experience and training ship experience. This parallel between real ship and simulator, puts a responsibility on the simulator instructor to ensure that simulator based training is designed and conducted in such a manner that it gives real time experiences to the trainees. Simulator training is required to put the trainee in as close a working environment, mental scenario and physical stress as onboard the real ship.
Page 4 of 11
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management PREPARE FORMAL ASSESSMENT FOR NAVIGATION TASK SHORTFALLS: STCW-95 has defined the functions for all levels of competency at support, operational and management levels. The operational level competencies have been elaborately divided into tasks and subtasks of in-service training, for which the formal assessment process has been given above. Unfortunately the above process is only a sample format and STCW-95 has not detailed the performance measures and performance standards for each individual competency/tasks. There is an acute need to create this bank for proper assessment. DIFFICULTIES: It appears that the IMO at this juncture has left the basic standards to be written by each administration. Here lies the lacuna, which may render different states to have different standards of performance. This Herculean task of writing the performance objectives, performance measures and performance standards, is now left to be done by either the administration or to be left to the individual owners / companies or training institutes in the case of non mandatory simulator courses, in which case the standards will be market driven. In either case there lies the grave danger of having different standards thus defeating the very essence of STCW-95. WHO ARE THESE PERSONS? There is an acute need for a unified body to take on the job of preparing the above assessment package for the maritime world. This unified effort will render great advantage in maintaining uniform standards as well as in saving time, money and valuable human resources. There is a need for the administrations and the training providers to understand their grave responsibility of training and assessment in order to give the STCW-95 convention a full and complete effect. We are thus getting closer to the fact that simulator training and assessment on simulators with proper approval, creditability and validity from the administrations can be relied upon to provide this uniform base. The above argument has also been put forward by Prof. Peter Muirhead, former professor from World Maritime University, Malmo, Sweden. STCW Requirements On Simulator Based Training.
Page 5 of 11
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management STCW discusses simulators under three important headings; a. Training and assessment. b. Use of simulator. c. Minimum standards of competencies. Regulation-I/6 - Training and Assessment. This regulation demands all parties to ensure that training and assessment of seafarers is in accordance with STCW Code-A and all instructors and assessors are appropriately qualified and competent to carry out their task. It may also be of relevance here to mention that STCW convention lists down seven functions of seafarers at three different levels of responsibility. The three different levels are as follows: 1. Performance Standards - physical realism, behavioural realism, minimum errors, instructor control, suitability for training objectives and man-machine interface. 2. Training Procedure - briefing, familiarization, exercise stimuli, monitoring, debriefing, peer assessment. 3. Assessment Procedure - performance criteria, assessment criteria, briefing, grading methodology. SIMULATION TRAINING Simulators are extremely powerful tool for initial training, promotional assessment, and for on-going training. They offer an excellent interactive environment to acquire the needed skills and the necessary attitudes for the job. Scenarios can be introduced that would be difficult to encounter in the normal everyday situations. Certain pressures can be applied in order to determine how an individual reacts. Likewise, emergencies can be introduced. In the end, the performance standards of students can be assessed under different conditions. Any weaknesses or errors must be discussed at debriefings, these being the most important sessions during training. Repeated exercises can enhance any particular skill that may be lacking. SIMULATORS AND EMERGENCY PREPAREDNESS More often than not, accidents happen, not as a direct result of equipment malfunction nor for that matter ignorance of the seafarer, but more due to lack of situational awareness, poor communication, poor teamwork and interaction, poor judgment and decision making capability, inadequate leadership and incapacity to handle stress. P&I claim statistics clearly show human error accounting for more than 70% of accidents
Page 6 of 11
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management How does one train a chief mate due for command to handle a steering failure in a narrow channel with a pilot on board - through text books or chalk and talk? No he must be made to stand in the Master’s shoes, feel the pressure and live the moment. In the safety of a simulator’s bridge, navigators can experience highly authentic reproductions of emergency conditions. In most fields where on-the-job training is difficult, dangerous or expensive to implement, Emergency Simulation Programs (ESP) have been developed. The major task at hand though is to devise methods of evaluating and assessing candidates who are training on these simulators. All of the above factors need to be assessed separately and then summarized to form a total appraisal of the candidate. Simulators provide a practical method for measuring or testing levels of competence and proficiency and the ability to continue to prioritize tasks. Yet few training institutes fully address measuring, evaluating, and assessing individual mariner performance and it is a far cry for administrations to start using simulators for ship handling assessment. On this ground I must mention that some administrations have made Ship Manoeuvring imulator training (SMS) mandatory for obtaining a Master’s certificate of competency. The approved institutes conducting the courses have a grave onus for training and assessment since this assessment forms part and parcel of the competency certificate.
DEVELOPMENT OF A SIMULATION PROGRAM Phase 1 - Identify And State Need - In most training programs, the training need is implied, but never stated. The purpose of a simulator program could be stated as : Initial Training through various problems and / or emergency scenarios. Candidate Performance assessment Refresher training Phase 2 - Develop Goals And Performance Objectives - Based on the commitment to focusing on end results throughout the development of any training package, our next step (and one of the major activities required ) is to carefully determine the goals and behavioural objectives to be satisfied. Goals are statements of the things the training program is to achieve from the perspective of the program or the deliverer. Objectives, on the other hand, are stated from the perspective of the individual learner and describe the specific observable behaviour that the learner should be able to exhibit in order to satisfy the objective.
Page 7 of 11
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management Phase 3 – Content This would define the details of the program, what should be taught to the trainees. The content could be divided into three parts : 1. Theory / 2. Practice / 3. Simulation Theory: Manoeuvring data and ship handling characteristics, Route and Passage Planning, Ship safety procedures, Considerations of draught, Intact and Damage Stability, Effects of Grounding and collisions Practice: Manoeuvres – eg. Short turn around, constant radius turn, low frequency rudder cycling, Communication within the bridge team and with the outside world, Reporting to Vessel Traffic system (VTS), Interpretation of the Radar and data displays, Usage of all available resources on the bridge Simulation: Navigation in Dense traffic and /or reduced visibility, Live Communication within bridge team and on telephone / VHF, Response to unforeseen events, Obstacles in the channel/harbor, Various failures in machinery, steering gear, thrusters, etc., Pilot disability and over-riding of the pilot, Increased work load, Man overboard, Human error, Post accident scenario, Grounding, Collision, Pollution, Fire, Flooding A high level of stressful elements and intensity helps bring up the urgency level so very crucial in maintaining the realism required so as make the trainee build up a sweat – after all we would like to assess his behavioural response to situations. Phase 4 – Validation The effectiveness of the simulator to achieve the desired goals and objectives must be evaluated. Validation would generally be achieved through a 3 step process : Assess the characteristics of the simulation against set parameters and standards Eg : Sea trial data for the manoeuvring characteristics of a ship. Assess the sequence of events to ascertain they are unfolding in real time. Induced malfunctions and failures should produce the desired effect. Experienced operators would be required to go through this process and validate their correctness. Assess the fidelity (realism) of the simulator. The visual model, the motion model and the hardware-to-software interface should produce the real world situation and feeling. The definition of fidelity is “physical and functional similarity to the actual equipment.”
Page 8 of 11
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management Phase 5 – Debriefing This is the process of informing the learner, at the end of a simulation exercise, the evaluations of actions performed in a sequential order and whether the outcome was as desired or could have been better. This correctional input helps the learner to know his weaknesses and strengths and what was done right and wrong. Exercise debriefings are carried out immediately after each exercise. The instructor may re-play the exercise at fast or real time, stopping where necessary to highlight particular objectives or problem areas. The Course participants play an active role in the debrief sessions, highlighting their own negative and/or positive points. This could also serve as informal peer assessment. Voice / video tape recordings are particularly useful with respect to learning communication and improving on behavioural skills. Case studies on a relevant subject are used to enhance such discussions and help reinforce certain concepts. Phase 6 - Develop Criterion Testing Evaluation systems should assess the ability of learners to satisfy the objectives developed. One of the biggest weakness of many training programs is that the evaluation of performance is drawn from the content and not from the objectives. This results in evaluations which often fail to accurately reflect the ability of learners in the desired skill and knowledge areas. The evaluation cycle continues with the examination of syllabus changes to validate the desired improvements. To be able to assess student actions, there should be a clear understanding of what constitutes correct actions or correct goal states in the simulated environment. The training system can direct the simulation to carry out a set of steps that constitutes a demonstration of how a procedure is to be performed. It can direct the simulation to provide instructive feedback in the simulation context, and it can assess the correctness of a student's actions in a behaviorally realistic context. A master can have more confidence in assigning an officer to carry out a new task if that officer has correctly performed analogous tasks or at least components of the task correctly in a simulated environment. Hence evaluation should be based on whether the student has carried out a certain task required in those circumstances in a responsible and correct manner. Our experience has shown that though the student has the requisite knowledge, he is unable to perform the tasks at the right time when it is required, bucking under stress and work pressure thus leading to spontaneous incompetence. Simulators are particularly useful in assessing whether an officer is ready to be given a managerial position on board a ship. Will he be able to take the right decisions in crisis situations? Master Assessment and Proficient Training (MAPT), is intended for
Page 9 of 11
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management masters and chief officers. Besides training objectives, the program is de-signed to supply the ship owner with in-depth feedback about a candidate’s suitability and working style as master. This information is intended to form a valuable contribution to a ship owner's decision regarding employment, promotion or designating personnel for new tasks, e.g. high-speed craft or other advanced technology. After the completed MAPT course, the ship owner receives thorough feedback on the assessment of each participant’s performance, both written and oral, along with recommendations concerning the candidate’s suitability for a master’s position.
Rating RATING SCALE ALWAYS MOSTLY OFTEN USUALLY SOMETIMES OCCASIONALLY RARELY 7 6 5 4 3 2 1 Average Rating Descriptor Excellent Very Good Good Average Below Average Acceptable Not Acceptable
* Rating less than 4 in any particular element within a competency shall require a specific comment by the assessor.
Behavioural Marker System (Worksheet) No Competency Description of Behavioural Objective Motivates and leads by example Reflects on suggestions of others Gives personal feedback Involves team members in operation and monitors team performance Total Establishes atmosphere for open communication and participation Encourages inputs and feedback form others Takes notice of the suggestions of others Helps other crew members in demanding situations Offers assistance Sensitive to people's needs Total Page 10 of 11 Rating 6 6 4 5 26/5 5 4 5 4 5 5 36/8 4.5 5.2 Average Rating
1
Leadership
2
Teamwork
Maritime Pot-pourri 2007 “ASSESSMENT OF SKILLS” By Capt. Tescelin Almeida Anglo Eastern Ship Management
Verbalises and requests for a challenge and response on bridge Responds appropriately to a perceived challenge Issues challenges when a concept is breached Sets limits for various perceivable concepts Total Recognises and states clearly the development of a critical situation. Uses all available resources to deal with the situation Takes charge of the situation immediately and takes decisive action 5 6 6 5 35/7 6 5 4 5.0 5.0
3
Challenge & Response
4
Emergency Handling
Total 15/3 * Comments:_____________________________________________________ ________________________________________________________________ ________________________________________________________________
Page 11 of 11