Docstoc

Conducting-an-Evaluation-of-a-Training-Programme---The-University-

Document Sample
Conducting-an-Evaluation-of-a-Training-Programme---The-University- Powered By Docstoc
					Evaluation importance of Conducting an Evaluation of a Training Programme setting and The importance of setting and evaluating standards ofevaluationg training telemedicine The importance of training is evident in medicine. The faculties of medicine and nursing colleges ensure that students are trained to a high professional level in order that they are proficient in their chosen speciality and future career. Clinicians are trained to graduate and post-graduate level. Postgraduate training and life-long learning are important to maintain as they ensure that the certified doctor or nurse becomes an expert in their field, providing quality assured health care to the general public. Emphasis on theoretical and practical skills and evidence based practice is essential for clinical practice. Post graduate education has been provided for several years by the Royal Colleges, some of which is delivered at a distance by videoconference. A regular weekly education programme is broadcast to western Europe by satellite 1. However, at present, this highly proficient training is not carried into practice in telemedicine. Since telemedicine is medicine practised at a distance, there should be an increased emphasis on training clinicians in its use. Health professionals engaged either full-time or parttime in telemedicine and telehealth require to use special skills and

1

knowledge. At present they require these skills on the job and this experience may not be recognised. There should be performance standards and telepractice guidelines for professionals operating in the fields of telemedicine. Furthermore, there is a case for the development and implementation of education and training standards, enabling professionals who practise in the field to obtain suitable skills, knowledge and recognition for telepractice 2. Yellowlees states that to achieve good life expectancy with a telemedicine programme it requires nurturing. This includes the provision of money, ideas, education, training and innovation. It requires experience, which calls for the achievement of long and wide patterns of usage, the development of updated policies and procedures and the involvement of multiple disciplines. The importance of appropriate training in the use of videoconferencing technology is often underestimated when telemedicine projects are established. All technology requires training and simply installing videoconferencing equipment in a hospital or health centre and leaving the manufacturer‟s manual nearby is not sufficient to encourage or maintain its use for telemedicine
3

. For telemedicine to move from

experimental single use application to accepted routine practice, the training and support needs of users should be recognized. A paper by Barry, reports the findings of a survey of telemedicine usage, technical support and staff training in six health board regions across Scotland. It focuses on three

2

problems associated with training: a lack of familiarity with the technology, a lack of training, and a lack of knowledge of the availability of technical support for the system. The paper argues for the creation of a set of protocols that would specify support and that would be distributed through all health boards and authorities 4. Protocols for minor injuries have been suggested in a paper by Benger for the following reasons:  To maintain the speed of the tele-consultation  To ensure that all steps in the consultation process are followed  To serve as an aid-memoire for emergency nurse practitioners and doctors The tele-consulting protocol outlines the following steps in the consultation process: explain to the patient what is going to happen, establish contact with the specialist, make the introductions, summarise the problem, review the history, perform the examination; review any test results, discuss the diagnosis; discuss the management, discuss referral and follow-up and complete the clinical records 5. Telemedicine equipment is often referred to by the clinicians as „kit‟ and is often installed into health centres and hospitals without any future planning for staff education on its purpose or training in its use. This results in some cases in staff apathy and often fear of technology. Following initial

awareness raising about telemedicine and its potential, staff must be properly

3

trained, not just in how to turn on the equipment but in how to use it effectively for consultation, education and administrative purposes. Training needs to be extensive and repetitive. The nurse practitioners and consultants require initial lectures on optics, sound transmission, digitisation, transfer of images, use of software, ethics and confidentiality. Tachakra et al, found this to be essential as their first attempt at a tutorial in the videoconferencing room was ruined by “people asking a bewildering variety of questions.” When the lecture format is followed by tutorials in the rooms, the learning becomes more structured and ends with each person having an opportunity to conduct a telemedical consultation from both ends. In a study carried out by Gerrard, there were several factors identified which influenced the successful implementation of telemedicine: early involvement of all groups affected, prior consideration of the practical implications, thorough initial testing of equipment, good technical support, imaginative training and clarity of the purpose of the service. The study showed that all those involved must be prepared for the rapid learning that is implicit in the implementation of change. The challenge is to maximise the potential for the primary operator to use new techniques. “In respect of nurse-centred services, the nurse must not be viewed as a technician, nor should telemedicine be seen as a substitute for an available doctor 6. Training needs to be continual, especially where staff turnover is high. It should be practical, pitched at different levels,

4

incorporated into mainstream activities such as orientation courses and staff development days wherever possible and supplemented with clear, straightforward protocols and user friendly instruction manuals. In the paper by Brebner et al, the training of the telemedicine users was taken very seriously. However, problems were still encountered when new members of staff were used without the author‟s knowledge and no formal telemedicine training was given. This aspect was found to be so important that the author plans to introduce a telemedicine license without which people will not be able to practice. The problems are, lack of education, lack of practical skills training, lack of information technology skills, lack of practice and lack of skills retention. This leads to fear and technological phobia in some instances. Questions must be asked:What kind of training is required to produce a good learning outcome? Theoretical, involving formal lecture style? Passive learning, reading from a manual, or watching a video? Training should be interactive and hands on, utilising practical skills with information on how to use the equipment for the particular clinical task. Training by utilising video-conferencing is found to be an acceptable and effective method of learning
7,8

. In a study by

Haythornthwaite, comparisons of the improvements made by rural participants, who accessed training via videoconferencing and metropolitan participants, who accessed training face to face, revealed few significant

5

differences. Rural participants reported high levels of satisfaction and decreased feelings of professional isolation 9. In an innovative teaching project, medical undergraduates in six United Kingdom universities were connected by a high speed communications network for teaching in surgery
10

. In teaching medical students further afield, staff from the University of

Aberdeen employed a public communications network (ISDN) to teach pathology to students in the United Arab Emirates 11. Rural nurses in Wales experience professional isolation and lack of access to training and education. Low-cost videoconferencing equipment was used successfully in training sessions concerning asthma and travel immunisation, preferring videoconferencing to long journeys 12 . Protopapas et al, identified two goals in a PACS training programme. The first was to teach physicians how to retrieve images and reports from current as well as prior studies and display them on a computer workstation. Secondly, the training should include instruction on the use of various workstation tools to enhance image interpretation. Imaging requirements and usage by different physician groups vary and PACS training should be tailored accordingly. Difficulties in the scheduling of training sessions during working hours and the widespread use of a "generic" log on identification, have contributed to the low (22%) compliance of non-radiologists with the formal training programme. Although the authors believe that one on one the participants

6

training is the most effective and can be best tailored to the needs and computer expertise of an individual physician, computer based training (both on and off line) may provide an acceptable and, in some cases, a preferred alternative 13. Although many studies include an evaluation of user satisfaction effectiveness of training is often ignored
15 16 14

, the

. This can result in patients

becoming inadvertent experimental subjects for “on the job” training. If telemedicine is to be regarded as highly professional then all health care professionals should be appropriately trained in advance of service establishment 17.

Evaluation Methodology Specific training content depends partly on the equipment being used. In the present study the equipment used was a videoconferencing system (1600P, Sony) by Scotland and a Migra and Tanberg system by Sweden all sites operating at 384 kbits/second on ISDN lines. A document camera (AVP750, JVC) was used for wound and X-ray image transmission by Scotland. The content of a training course teleconsultations was established by an expert group consisting of doctors, nurses, educationalists and telemedicine researchers. The evaluation conducted in Finland had an identical

methodology questionnaire to that used in Scotland.The following aspects of

7

user competencies were identified and agreed upon as components for successful training:  adjusting the near-end camera angle (pan,tilt,zoom)  adjusting the near end brightness  making a call  use of picture in picture  use of the mute button  adjusting the volume  controlling the far end camera  saving a still image  viewing a saved image  use of the document camera for clinical image and x-ray transmission  disconnecting a call The videoconferencing equipment was set up at a base site and at remote training sites. A training manual was produced and training sessions were given. These training sessions were delivered from the Telemedicine

Laboratory, University of Aberdeen, the Geriatric Centre at Umea University Hospital and the Vannas Health Care Centre, Sweden to peripheral remote sites.

8

A quantitative and qualitative questionnaire was developed for the evaluation of user satisfaction and the effectiveness of training .The questionnaires were checked for face validity and a pilot study carried out before training began. A training booklet was also provided. In Scotland a database (SPSS, Statistical Package for the Social Sciences) was used for data entry and analysis. Ethical permission was deemed unnecessary by the local ethical committee. Results from Scotland A total of 102 (79%) fully completed questionnaires were returned. In the qualitative section of the questionnaire the one main area of concern reported was of insufficient opportunity to practise. An analysis of the results from Finland showed that there was no significant difference from the Scotland study

9

Figure 1

Expectation/Satisfaction Levels

5 4.75 4.5 4.25 SCORE 4 3.75 3.5 3.25 3

expectation level 3.84 3.98 4.12 satisfaction with training programme satisfaction with training booklet

Figure 1 shows that the satisfaction level for the training programme (3.98) and the booklet (4.12) both exceeded student expectation (3.84). The scores were obtained using a five point semantic differential scale (1=very poor 5=excellent).

10

Figure 2

User Competence

80 70 60 50 % 40 30 20 10 0 34.6 56.2

73.2 TRAINING ONLY

TRAINING AND BOOKLET TRAINING BOOKLET AND PRACTICE

Figure 2 shows that only 34.6% of students felt fully competent after the training programme but after studying the training booklet this increased to 56.2%. After they had the opportunity to practise user competence reached a maximum of 73.2%.

11

Figure 3

Competence Level

100 90 80 70 60 % 50 40 30 20 10 0

100

100 DAILY PRACTICE 60 43 WEEKLY PRACTICE MONTHLY PRACTICE <MONTHLY PRACTICE

Figure 3 shows that two months after training a user competence level of 100% was achieved by students who had the opportunity to practise on either a daily or a weekly basis. For those students who only had the opportunity to practise on a monthly basis the user competence level dropped to 60%. Those students who only had the opportunity to practise on a < monthly basis the user competence level dropped to only 43%.

12

Questionnaire
The purpose of this short questionnaire is to establish whether the teleconsultation training session has been effective. Please take a few minutes to study the questions below which refer to a range of procedures related to conducting a teleconsultation. All data will be treated in confidence and not related to any individual. Please tick only one box for each question.
1. When did you feel competent in adjusting the camera angle?

After receiving training

After receiving training and reading booklet

After receiving training reading booklet and practice

Never felt competent

2. When did you feel competent in adjusting the brightness?

After receiving training

After receiving training and reading booklet

After receiving training reading booklet and practice

Never felt competent

3. When did you feel competent in making a video call?

After receiving training

After receiving training and reading booklet

After receiving training reading booklet and practice

Never felt competent

4. When did you feel competent in adjusting the „picture in picture‟ during a call?

After receiving training

After receiving training and reading booklet

After receiving training reading booklet and practice

Never felt competent

5. When did you feel competent in muting the sound?

After receiving training

After receiving training and reading booklet and practice

After receiving training reading booklet

Never felt competent

6. When did you feel competent in adjusting the volume?

After receiving training

After receiving training and reading booklet and practice

After receiving training reading booklet

Never felt competent

13

7. When did you feel competent in controlling the far end camera?

After receiving training

After receiving training and reading booklet and practice

After receiving training reading booklet

Never felt competent

8. When did you feel competent in saving a still image?

After receiving training

After receiving training and reading booklet

After receiving training booklet and practice

Never felt competent

9. When did you feel competent in viewing a saved image?

After receiving training

After receiving training and reading booklet and practice

After receiving training reading booklet

Never felt competent

10. When did you feel competent in using the document camera to send an x-ray?

After receiving training

After receiving training and reading booklet and practice

After receiving training reading booklet

Never felt competent

11. When did you feel competent in disconnecting a call?

After receiving training

After receiving training and reading booklet and practice

After receiving training reading booklet

Never felt competent

What expectations did you have for the usefulness of the teleconsultation teaching session before it took place? Please circle only one number

1 low

2

3

4

5 high

How would you rate your level of satisfaction for the teleconsultation teaching session ? Please circle only one number

1 low

2

3

4

5 high

14

How would you rate the usefulness of the instruction booklet that you received?

Please circle only one number
1 low 2 3 4 5 high

Since completing the training session have you used the videoconferencing machine On average: (Please tick ) DAILY WEEKLY MONTHLY < ONCE PER MONTH

Do you still feel competent in using the videoconferencing machine? (please tick) YES Any other comments related to the training?

NO

15

Results from Sweden Evaluation of a Migra Training Programme Introduction As part of the REMEDY project, the staff involved at the Geriatric centre, Vännäs health centre and Vännäs municipality received training in the use of video conferencing equipment for co-ordinated care planning within the framework of the project. This report is an evaluation of the training based on the responses to the questionnaire filled in by the participants at the end of the programme. The questionnaire consisted of questions with close-ended response alternatives (often “yes”, “partly” and “no”) as well as more open-ended questions, with space for the respondents to add any comments they might have. The questions dealt with the education and training in the use of the Migra equipment as well as the skills that the participants had acquired as a result of the programme.

The demographics of the participants, such as age, sex, and place of work are presented, followed by the results of the questions on skills and training. The report is concluded with a short summary of the findings.

16

Respondents Over 60 persons appear to have completed the questionnaire, but due to partial drop out, the response to individual questions is often somewhat or – in some cases – much lower. Figure 1 shows the distribution of the respondents according to age and sex.

35 30 25 20 15 10 5 0 Female 18

31

3 4 Age > 25 Age ≤ 25 Male

Figure 1 Distribution of respondents according to age and sex. The respondents are mainly female. The age distribution shows that there are slightly more respondents over the age of 25.

17

Figure 2 shows the respondents‟ place of work. Approximately 60 % worked at the Geriatric centre, approximately 30 % at Vännäs municipality and 10 % at Vännäs health centre. The most common professional categories are registered nurses, assistant nurses and occupational therapists.

10%

Geriatric centre Vännäs health centre

59% 31%

Vännäs municipality

Figure 2 Distribution of respondents according to workplace. Skills The first two questions in the questionnaire concerned skills related to setting up a video conference and positioning the camera. Figure 3 shows that approximately a third of the respondents answered “yes” and a third “partly” to both questions. A slightly larger number reported that they are able to position or partly position the camera compared to setting up a video conference. A majority of the respondents, however, considered that there is a need for a specially designated person to position the camera during video conferences.

Regarding knowledge of how to set up a video conference, differences appear to exist in relation to age and workplace, which is also the case in many of the questions dealing with the skills acquired. A comparatively large

18

proportion of the younger age group (≤25 years) answered “yes” to the question “Are you able to set up a video conference?” (Figure 4), while a majority of the older age group answered “no” to the same question.
100%

80%

60%

40%

20%

0% Yes Partly No Able to position the camera

Able to set up a video conference

19

Figure 3 Skills in setting up a video conference and positioning camera.
100% 80% 60%

40% 20% 0% Yes Partly Age ≤ 25 Age > 25 No

Figure 4 Question: “Are you able to set up a video conference?” The columns show the proportion of the respective age group that has given a particular answer.
100%

80%

60%

40%

20%

0% Yes Geriatric centre Partly Vännäs health centre No Vännäs municipality

20

Figure 5 Question: “Are you able to set up a video conference?” The columns show the proportion at the respective work place that has given a particular answer. As can be seen in Figure 5, a comparatively large proportion of those working for Vännäs municipality stated that they were able to set up a video conference. A majority of Geriatric Centre staff stated that they partly possessed these skills, while a majority of those working at Vännäs health centre considered that they were incapable of setting up a video conference. It should be mentioned here that a comparatively large proportion of the respondents from Vännäs municipality belong to the younger age group, while those working at Vännäs health centre are mostly over 25 years of age. Thus, the younger age group seems to have acquired more knowledge than the older participants. On the other hand, there do not seem to be any great differences between men and women with regard to the level of knowledge.

The questionnaire also includes two questions about the ability to set up a video conference and position the camera by following the instructions contained in a manual describing the procedure. The answers to these two questions are presented in Figure 6. In this case, a larger number of respondents reported that they wholly or partly possessed the skills required (cf. Figure 3). Those who considered that they lacked the skills required to perform these tasks with the aid of the manual were given an opportunity to

21

explain why in the open response alternatives to the two questions. A common answer was not having tried.
100%

80%

60%

40%

20%

0% Yes Partly No

Able to set up a video conference with the aid of the manual Able to position the camera with the aid of the manual

Figure 6 Skills in setting up a video conference and positioning the camera with the aid of the manual. Two other competence-related questions dealt with whether or not those who received training knew what to do and who to contact in the event of sound and image problems. Here, there was a comparatively high proportion that knew exactly who to contact, although they were less certain when it came to what they themselves could do. However, quite a few stated that they had at least some idea of how to try to correct the problem (Figure 7).

22

Figure 7 Knowledge of what to do and who to contact in the event of sound and image problems. On the question of who to contact in the event of sound and image problems – as opposed to many other knowledge-related questions – only minor differences were found between different age groups and work places (Figure 8 and Figure 9).
100%

80%

60%

40%

20%

0% Yes Partly Age ≤ 25 Age > 25 No

23

Figure 8 Question: “In the event of sound and image problems, do you know who to contact?” The columns indicate the proportion of the respective age group who gave a particular answer.
100%

80%

60%

40%

20%

0% Yes Geriatric centre Partly Vännäs health centre No Vännäs municipality

24

Figure 9 Question: “In the event of sound and image problems, do you know who to contact?” The columns show the proportion of staff at the respective workplaces who gave a particular answer.
50

40

30

20

10

0 Yes Partly No

Training Few of the respondents considered that they had been able to practice sufficiently on their own after completion of the programme (Figure 10). As can be seen in Figure 11, only a few respondents spent more than 30 minutes on practising on their own. On the open-ended question about obstacles to training, the most common response was lack of time and – to some extent – motivation. On the question of how the respondents considered that such training should be organised, many stated that they would like it to form part of their scheduled tasks.

25

Figure 10 Question: “Have you been able to practice sufficiently on your own after completion of the training programme?”
25

20

15

10

5

0 ≤ 30 min. > 30 min. > 45 min. > 60 min.

Figure 11 Question: “How much time do you estimate you have spent on practising on your own?” On the whole, the participants seemed to be satisfied with the training programme. A majority could not find any fault with it (Figure 12), and the overall comments were positive. Of those who criticised the programme, the most common complaint was that more training was needed.
30

20

10

0 Yes Partly No

Figure 12 Question: “Does the instruction lack anything?”

26

Summary Of those who took part in the Migra training programme, a majority had, wholly or partly, acquired skills in setting up a video conference and adjusting camera positions. Knowledge of how to handle the equipment is especially good in the younger age group. However, most of the respondents considered that a specially appointed person is required to operate the camera during the video conferences. Even if many of the respondents did not know exactly what to do in the event of sound and image problems, a majority knew who to contact. In general, the participants were satisfied with the training programme. However, few considered that they had been able to practice sufficiently on their own after completion of the programme mainly due to lack of time

27

Questions to be answered after completion of the Migra training programme Female Male Age 25 years > Occupation....................... 25 years <

1. Are you able to set up a video conference? yes partly 2. Are you able to position the camera? yes partly

no

no

3. Do you think that a specially designated person is needed to position the camera during the video conference? yes partly no 4. In the event of image and sound problems, do you know a) what to do? yes partly no b) who to contact? yes

partly

no

5. Have you been able to practise sufficiently on your own after completion of the course? yes partly no If yes or partly a) When and how have you been able to practise on your own?

if partly (practising setting up the video conference and positioning the camera) b) When and how have you been able to practise on your own?

28

c) estimated time spent on practising on your own <30 >30 >45 >60 >90 >120 min min min min min min If more than 120 min, please state number of hours and minutes......... If no (practising setting up the video conference and positioning the camera) d) Did you encounter any obstacles to private practice. If so, please specify.

e) What, in your opinion, would be the best way to allocate time for private practice? Please provide examples:

6. Does the instruction lack anything? yes partly

no

a) If yes or partly, please state what you think has been omitted.

If you have not used the equipment for a considerable length of time 7. Are you able to set up a video conference with the aid of the manual? yes partly no If no a) please explain why you are unable to set up a video conference with the aid of the manual

29

8. Are you able to position the camera with the aid of the manual? yes partly no

30

If no a) please explain why you are unable to position the camera with the aid of the manual

9. Please list your comments, if any, regarding the Migra training course

10. Please state your own views and tips, if any, for improving the education and instruction

31

Joint Discussion Trainees had already heard about the intended training programme and the results show that their expectations were quite high. The training programme and the booklet were successful and both exceeded student expectation. If a high level of user competence is to be achieved then it is essential to back up the training programme by written instruction and to allow extensive practice with the equipment. If regular practice is not undertaken at least weekly then a substantial fall in user competence can be expected. When a telemedicine service is being evaluated it is of paramount importance that as high a level of user competence as possible is achieved and maintained. Current evidence suggests that if this is being done then it is certainly not being measured or reported 18 .It would be a simple task to modify the training programme and evaluation for any clinical or educational application. Picot has suggested that there is a case for the development and implementation of education and training standards, enabling professionals who practise in the field to obtain suitable skills, knowledge and recognition for telepractice 19.

Joint Recommendations Before establishing a telemedicine service the following steps appear to be important, 32

 identify the required training competencies  deliver a “hands on” training programme based on the required training competencies  measure the level of user competence after initial training  back up the training programme with an instruction booklet  ensure that trainees have at least weekly practice  re-measure the level of user competence after one month  ensure that there is a contact person available in event of technical problems

33


				
DOCUMENT INFO
Shared By:
Tags: Condu, cting
Stats:
views:11
posted:11/29/2009
language:English
pages:33
Description: Conducting-an-Evaluation-of-a-Training-Programme---The-University-