"replication of a clinical learning environment survey for junior medical officers"
Replication of a Clinical Learning Environment Survey for Junior Medical Officers: a Study of Medical Students in an Indonesian Hospital Ova Emilia*, Leah Bloomfield**, Arie Rotem** *School of Medicine, Gadjah Mada University, Jogjakarta, Indonesia; **School of Public Health and Community Medicine UNSW, Sydney, Australia Please address correspondence to Leah Bloomfield, email@example.com. Abstract Background: Monitoring medical students’ hospital learning experiences requires an instrument that probes organisational and educational aspects of the environment. Purposes: The study investigated the validity, reliability and practicality of a survey tool for probing the clinical learning environment used reported on the clinical learning environment in a major teaching hospital in Indonesia. Methods: A cross-sectional survey of 209 clinical clerkship students was conducted in an Indonesian hospital. Methods of analysis included factor analysis, reliability studies, cross-tabs and univariate analysis of variance. Results: The survey tool revealed departmental differences in the clinical learning environment, as perceived by senior medical students. The clinical learning environment was described by three factors, which together account for 61.8% of the variance. The major factor, which we named the Conditions for Learning, is the organisational and psychosocial aspects, which consists of seven inter-correlated scales. Structural aspects of the environment are described by two factors, which cover opportunities to practise clinical skills, assessment, learning activities and resources. Conclusion: The instrument is valid, reliable, practical and shows promise as a culture-neutral instrument for monitoring medical students’ perceptions of clinical learning environments. Indexing terms: Clinical learning environment; medical student; survey; methodology Ethics approval was obtained for this research INTRODUCTION Consideration of the learning environment is essential to any discussion about curriculum, learning outcomes, quality improvement or change in medical education. Students’ perceptions of the learning environment or climate (Genn, JM, 2001) influence their behaviour and achievements (Lizzio, A, Wilson, K, & Simons, R, 2002). Data on students’ perceptions of the learning environment are needed in order to plan and monitor changes in curriculum. Since (Hutchins, EB, 1961) study a number of instruments for characterising the medical school environment have been published. The Medical School Learning Environment Survey (Marshall, RE, 1978; Rothman, AI & Ayoade, F, 1970) (MSLES) has been used to track longitudinal changes in a single medical school (Clarke, RM, Feletti, GI, & Engel, C, 1984; Clarke, RM & Henry, RL, 1986) and to make comparisons between schools (Lancaster, CJ et al., 1997; Lieberman, SA, Stroup-Benham, CA, Peel, JL, & Camp, MG, 1997). More recently, the Dundee Ready Education Environment Measure (Roff, S et al., 1997) (DREEM) has been used to compare medical schools in developed and developing countries (Al-Hazimi, A et al., 2004; Bassaw, B et al., 2003; Roff, S, S, M, Ifere, OS, & Bhattacharaya, S, 2001) and adaptations of DREEM have been used for specialty training (Cassar, K, 2004; Holt, MC & Roff, S, 2004). Much of this work has been concerned with campus-based learning, with its orderly program of lectures, tutorials and practical sessions. In contrast the complex social, cultural, organisational, interactions in the hospital workplace make it a daunting educational environment for students (Seabrook, MA, 2004) . Learning is strongly influenced by the clinical workload, patient turnover, supervisor availability ((Rotem, A, Bloomfield, L, & Southon, G, 1996; Seabrook, MA, 2004) and from the students’ perspective, the curriculum-in-action is defined by the patients and the procedures they are able to observe and practise. Previously we have used concepts from organisational theory to develop a clinical learning environment survey for junior doctors in Australian hospitals (Hart, G & Rotem, A, 1995; Rotem, A, Godwin, P, & Du, J, 1995). Studies using the survey demonstrated that hospital departments vary widely in the social and organisational aspects of the learning environment, at least in the views of junior medical staff. In the present study we wished to test whether this survey tool contributed to understanding how medical students learn in a major medical school in Indonesia. The study was part of a larger investigation of approaches and achievements of students learning in clinical settings. METHOD We conducted a cross sectional study to test the psychometric properties of the instrument and to form an overview of the learning environment in a major Indonesian teaching hospital. Instrument Development. Small adaptations were made to the junior medical officer survey (Rotem, Godwin & Du, 1995) to make it relevant to senior medical students’ perceptions of the learning environment of their most recent clinical rotation. Section 1 included 28 Likert scale items about the psychosocial and organisational climate of learning. Section 2 comprised Likert scale questions about access to generic learning resources (eg library facilities) and generic learning activities (eg lectures, ward rounds, bedside tutorials, outpatient clinics). We also asked about frequency of performance of required rotation-specific clinical skills and procedures. Inventories of specific clinical skills and procedures were developed in consultation with departmental coordinators. For example, the items in the internal medicine rotation included assessment of nutritional status, vital signs, hydration status, auscultation of lung, heart, abdomen etc. Section 3 included Likert scale items about the methods of assessment (eg, viva, written exams, projects, clinical exam) and students’ views about them. Sections 2 and 3 contained some items intended as a cross-check on the reliability of responses to Section 1. Demographic information including, age, sex, type of rotation, GPA, and career intentions was gathered. The instrument was translated into Indonesian language and back-translated before pre- testing with 68 students, none of whom participated in the main study Subjects and procedures. The survey was given to all students doing clinical clerkships at a major medical school in Indonesia. The surveys were completed anonymously in the final week of the clerkship. Qualitative data were collected by participant observers, from students' self- report diaries and through student focus groups. These results are not reported systematically but are discussed where they provide insight into the meaning of the quantitative findings. Statistical analyses. All data were analysed using SPSS version 10 for Windows. Frequencies were computed for all scales, after reversing the responses for the negatively worded items. Factor analysis was performed using principal component factor analysis with oblimin rotation and Kaiser Normalization for each scale. Chi-square was used to test for differences in demographic variables. For univariate analysis of variance tests, we assigned numerical values to the Likert scale items. Psychosocial and organisational learning climate items and assessment items were scored from 1 (strongly disagree) to 5 (strongly agree). Levels of provision of learning activities and resources were scored as from 0 (nil) to 3 (high). The sub-scale scores were summed to give scale scores. For opportunities to perform clinical skills and procedures, we computed standardised scores that allowed us to compare rates in departments with different numbers of skills. RESULTS Students The surveys were returned by 209 of 255 students (82.0%). Students were unevenly distributed across 13 different departments. The great majority (90%) were in the main teaching hospital; the remainder were in a peripheral hospital. The number of students per department ranged from 2 (1.0%) to 39 (18.7%). There were almost twice the number of men (64.1%) as women (35.9%) and approximately equal proportions of students in 4th year (45.4%) and 5th year (54.5%). Chi- square analysis showed that neither sex nor year of study was related to career intention. Female students had a slightly higher GPA (p<0.05). The responses to the cross-checking items about resources, opportunities to practise and assessment indicated that students responded consistently throughout the survey. Factor analysis - validity and reliability testing Factor analysis revealed eleven scales, which segregated as three factors with eigenvalue higher than 1.00 (see Tables 1 and 2) and together accounted for 61.8% of the variance. Inter-scale correlations were performed using Pearson's correlation (see Table 3). Factor 1 explained 42.2% of the variance. The items segregated into the same seven scales as in the junior medical officer instrument: autonomy, supervision, social support, workload, role clarity, emphasis on teaching and learning and variety. All scales were represented by 4 items, except role clarity (3 items) and emphasis on teaching and learning (5 items). The seven scales were positively correlated; coefficients ranged from 0.29 to 0.64 (p=0.01, 2-tailed). This result validates our earlier finding that the seven scales constitute a holistic construct, which we named Conditions for Learning (CL). Factor 2 consisted of general learning activities and resources and accounted for 10.2% variance. Factor 3 explained 9.4% of variance and consisted of opportunities to perform rotation-specific clinical skills and assessment. Analysis showed an acceptable level of reliability. Cronbach’s alpha was 0.86, 0.49 and 0.38 respectively for Factors 1,2, and 3. The reliability of the Conditions for Learning (CL) factor was high (0.96) and the reliability of the seven component scales (from 0.60 to 0.85) was comparable to the junior medical officer instrument. These results suggested that the survey was valid and reliable for medical students in this teaching hospital in Indonesia. Larger, multi-site studies are needed to more thoroughly examine the psychometric properties. Students’ perceptions of the learning environment Conditions for Learning: Overall, perceptions of the conditions for learning were mildly positive. More than half of the students agreed that there was an emphasis on teaching and learning (56.9%) and sufficient variety of patients and procedures (65.7%). Twice as many students had positive perceptions of supervision (46.6%) as negative (26.3%). For social support, autonomy and role clarity, positive and negative perceptions were evenly balanced. About half the students (52.3%) perceived the workload as being reasonable. Around 20 to 25% of students had generally negative perceptions of the conditions for learning and a further 13 to 26% were non-committal. Looking at each scale in more detail, for emphasis on teaching and learning , about two thirds (62.7%), felt that staff saw teaching as a major aspect of their role, provided ward rounds and tutorials (63.2%) and encouraged students to ask questions (69.9%). Fewer students thought staff were keen to demonstrate and explain (42.6%) and that clerkship coordinator was educationally active (46.6%). For social support, about half (57.4%) felt relationships were friendly, felt supported in their attempts to learn (48.8%) and felt part of the team (49.3%). Fewer students felt that senior staff were interested in how they were coping (18.2%). About half rated supervision positively overall (46.6%), found staff accessible and willing to provide guidance (53.6%) and were given feedback when doing procedures for the first time (52.6%). Fewer (32.5%) found the general level of feedback was sufficient and 21.1% were left on their own in situations when they were not sure what to do. In terms of autonomy, almost half (45.5%) felt they had reasonable freedom to set their own priorities regarding their clinical activities and felt their autonomy increased as their skills increased (42.6%). About two thirds (63.6%) felt the workload was quite reasonable and clerical duties were not burdensome (71.3%) although fewer (39.7%) felt that their time was used productively. Around one third ((34.4%) had enough time for reflection on their learning experiences. In terms of role clarity, more students were given clear instructions about their roles in relation to staff (65.1%) than about their responsibilities (43.5%). Almost half (42.6%) felt they were given mixed messages. Two thirds of students agreed that there was enough variety in types of clinical procedures (63.6%) and types of patients (67.0%). Learning activities and resources: This factor comprised generic activities, such as orientation sessions, lectures, tutorials, ward rounds, bedside teaching, meetings with advisers, direct supervision and feedback, access to the library. The majority (74.4%) stated that these types of activities were provided at a low to moderate level; very few students (4.8%) reported a high level and 20.8% reported that they were not provided. Opportunities to practise skills: This factor comprised the department-specific skills, which related to the specific objectives of each rotation. Students’ experiences varied widely. Taken across all departments, 67.8% of students performed at least one skill on at least one occasion; few (12.7%) neither observed nor performed any skills. Looking at individual departments (Table 5) there were three departments where more than 20% of students neither saw nor performed any skills and six departments where more than a third (35%) of students saw skills demonstrated but had no opportunity to perform them. The most frequent methods of assessment were orals (90.1%), assignments (86.2%), case presentations (80.3%), skill performance (69.3%) and written exams (50.2%). Less than half (40.2%) were satisfied with their assessment or considered it fair (42.6%). About one third (36.8%) had not been told how they would be assessed and felt that their grade did not reflect their level of competence (33.0%). The influence of gender, seniority and career intention. Univariate analysis (see Table 4) showed that personal characteristics had a minor influence on perceptions of the learning environment. Women perceived autonomy, supervision, opportunities to practice, and resources slightly more positively than men. Career intention and seniority interacted with perception of activities. Fifth year students and those who intended to become specialists perceived that there were more learning activities than fourth year students and those who wanted to be general practitioners. Departmental differences Univariate analysis of variance showed that differences between departments were highly significant on all CLE scales (see Table 4). We excluded departments with fewer than 10 responses. Tukey-HSD post-hoc analysis showed that the Conditions for Learning scales tended to band together, that is, when emphasis on teaching and learning was high, scores on autonomy, supervision, social support, role clarity, variety were also high, and conversely. We used multiple pair-wise comparisons for each scale for a deeper insight into the departmental scores. Departments tended to score high on many scales, or low on many scales. For example, the two highest-scoring departments had significantly higher scores than other departments on five and six respectively of the seven conditions for learning scales. The same occurred for the three lowest scoring departments. Discussion The survey emerged as a valid, reliable, rapid self-report tool for characterising the clinical learning environment in a tertiary hospital in Indonesia. As the seven scales of Factor 1 (the conditions for learning) showed similar psychometric properties to the scales in the study of junior doctors learning in Australian hospitals, we tentatively conclude that the survey can be used to portray the social and organisational aspects of hospital learning generally, although this conclusion should be validated using larger studies in other countries. We do not regard self report as a limitation but as appropriate for a study of learning, as it is students’ perceptions of their learning environment that determines the way they approach learning tasks (Ramsden, P & Entwistle, NJ, 1981). In this way the instrument differs from some more ‘objective’ types of hospital audit tools (Callaghan, IH & McLafferty, H, 1997; Shailer, B, 1990). As the scales are positively correlated, it is technically possible to arrive at a global score for the conditions for learning, however we feel, as do (Feletti, GI & Clarke, RM, 1981) that there is value in examining the individual scales. The survey revealed the clinical learning environment in the study hospital as moderately favourable. Between 46.2% and 65.7% of students perceived the Conditions for Learning scales positively. Around 20 to 25% of students had negative perceptions and 13 to 26% were non- committal. Generic learning activities and resources (Factor 2), such as outpatients clinics, ward rounds, bedside teaching lectures, tutorials, access to the library, were provided at a low to moderate level. About one third of students overall did not perform any of the skills that were considered appropriate for their rotation (Factor 3). This was disappointing but not unusual (Dolmans, DH, Wolfhagen, HA, Essed, GG, Scherpbier, AJ, & Van Der Vleuten, CP, 2001; Remmen, R, 1998). We are confident that there was minimal confounding by personal characteristics. Women in our study perceived certain attributes slightly more positively than men, an interesting finding, given the mixed findings on gender-related perceptions and responses to clinical learning (de Saintonge, DMC & Dunn, DM, 2001; Robins, LS, Gruppen, LD, Alexander, GL, Fantone, JC, & Davis, WK, 1997) and the current debate about the ‘chilly climate’ perceived by women in higher education in some western universities (Prentice, S, 2000). Our study did not find deterioration in the environment as students progressed through medical school (Pololi, L & Price, J, 2000) however we did not expect this as they were in the senior years. The survey detected clear differences between departments and has considerable potential for suggesting specific ways to improve the quality of the learning environment, although we caution that sufficiently large numbers of responses are needed if it is to be used for this purpose. We hypothesise that the favourable conditions for learning defined by Factor 1 are necessary but not sufficient for a positive learning experience. In other words, organisational and socio-cultural interactions enable or inhibit students’ access to the formal, technical elements of the environment, such as the organised activities, resources and opportunities for practice that are represented by Factors 2 and 3. We were unable specifically to test this hypothesis in this cross sectional study, however there is support for a “gateway” hypothesis from the qualitative data and from a second arm of the study in which we followed a subset of 39 students through a series of rotations. These findings, which are not reported here, suggest areas for future research. Nursing research indicates that the preceptor is a key figure and gatekeeper in creating the conditions for learning (Hart, G & Rotem, A, 1994; Saarikoski, M, Leino-Kilpi, H, & Warne, T, 2002). Our study indicates that the role of the individual preceptor is only one element in a web of social and organisational interactions that determine the conditions for learning. As medical students usually have a variety of people acting in a supervisory role, it may be more important that all staff in the department share an understanding of staff and student roles, relationships, expectations and the limits of student autonomy (Seabrook, MA, 2003). Conclusion The Clinical Learning Environment survey instrument is a valid, reliable and practical survey for monitoring students’ perceptions of learning environments. It has similar psychometric properties to the tool developed for Australian junior medical officers and so shows promise as a widely applicable, culture-neutral instrument, although additional confirmatory studies are needed. The seven scales of the conditions of learning are not readily separated and appear to constitute a holistic construct. Autonomy, role definition, social support, workload, variety, feedback and supervision all contribute to a department’s emphasis on teaching and learning. When adequate, these conditions appear to enable students to make best use of activities, resources, and opportunities to practise clinical skills. The survey instrument allows differences in departmental culture to be quantified. We are not suggesting that it be used to rank or compare departments; rather, we prefer to use the instrument as a source of evidence to guide review and discussion of a department’s educational policies, practices and resources. It could be used to monitor trends within a department, for example, to ensure that successive batches of students are offered comparable experiences. It could be used to monitor the impact of curriculum change, to contrast actual and preferred environments and to compare staff and students’ perceptions of the same environment. The instrument can also be used for research, for example, to test hypotheses about the relationship between environmental variables and the development of professional competency. REFERENCES Al-Hazimi, A., Zaini, R., Al-Hyiani, A., Hassan, N., Gunaid, A., Ponnamperuma, G., Karunathilake, I., Roff, S., McAleer, S., & Davis, M. (2004). Educational environment in traditional and innovative medical schools: a study in four undergraduate medical schools. Education for Health: Change in Learning and Practice, 17(2), 192-203. Bassaw, B., Roff, S., McAleer, S., Roopnarinesingh, S., De Lisle, J., Teeluci, S., & Gopaul, S. (2003). Students' perspectives on the educational environment, Faculty of Medical Sciences, Trinidad. Medical Teacher, 25(5), 522-526. Callaghan, I. H., & McLafferty, H. (1997). An audit tool for assessing the learning environment for Project 2000. Nurse Education Today, 17, 338-342. Cassar, K. ( 2004). Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees. Medical Teacher, 26(3), 260-264. Clarke, R. M., Feletti, G. I., & Engel, C. (1984). Student perceptions of the learning environment in a new medical school. Medical Education, 18(5), 321-325. Clarke, R. M., & Henry, R. L. (1986). Student perceptions of the learning environment. Medical Education, 20(4), 371. de Saintonge, D. M. C., & Dunn, D. M. (2001). Gender and achievement in clinical medical students: a path analysis. Medical Education, 35(11), 1024-1033. Dolmans, D. H., Wolfhagen, H. A., Essed, G. G., Scherpbier, A. J., & Van Der Vleuten, C. P. (2001). Students' perceptions of time spent during clinical rotations. Medical Teacher, 23(5), 471-475. Feletti, G. I., & Clarke, R. M. (1981). Review of psychometric features of the medical school learning environment survey. Medical Education, 15, 92-96. Genn, J. M. (2001). AMEE Medical Education Guide No. 23 (Part 1): Curriculum, environment, climate, quality and change in medical education - a unifying perspective. Medical Teacher, 23(4), 337-344. Hart, G., & Rotem, A. (1994). The best and the worst: students' experiences of clinical education. Australian Journal of Advanced Nursing, 11(3), 26-33. Hart, G., & Rotem, A. (1995). The clinical learning environment: nurses' perceptions of professional development in clinical setting. Nurse Education Today, 15, 3-10. Holt, M. C., & Roff, S. (2004). Development and validation of the Anaesthetic Theatre Educational Environment Measure (ATEEM). Medical Teacher, 26(6), 553-558. Hutchins, E. B. (1961). The 1960 medical school graduate: his perception of his faculty, peers and environment. Journal of Medical Education, 36, 322-329. Lancaster, C. J., Bradley, E., Smith, I. K., Chessman, A., Stroup-Benham, C. A., & Gwendie Camp, M. (1997). The effect of PBL on students' perceptions of learning environment. Academic Medicine, 72(10), S10-S12. Lieberman, S. A., Stroup-Benham, C. A., Peel, J. L., & Camp, M. G. (1997). Medical students' perception of the academic environment: a prospective comparison of traditional and problem- based curricula. Academic Medicine, 72(10), S13- S15. Lizzio, A., Wilson, K., & Simons, R. (2002). University Students' Perceptions of the Learning Environment and Academic Outcomes: implications for theory and practice. Studies in Higher Education, 27(1), 27-52. Marshall, R. E. (1978). Measuring the medical school learning environment. J Medical Education, 53, 98-104. Pololi, L., & Price, J. (2000). Validation and use of an instrument to measure the learning environment as perceived by medical students. Teaching and Learning in Medicine, 12(4), 201- 207. Prentice, S. (2000). The conceptual politics of chilly climate controversies. Gender and Education, 12(2), 195-207. Ramsden, P., & Entwistle, N. J. (1981). Effects of academic departments on students' approaches to studying. British Journal of Educational Psychology, 51, 368-383. Remmen, R. (1998). Unsatisfactory basic skills performance by students in traditional medical curricula. Medical Teacher, 20(6), 579 - 582. Robins, L. S., Gruppen, L. D., Alexander, G. L., Fantone, J. C., & Davis, W. K. (1997). A predictive model of student satisfaction with the medical school learning environment. Academic Medicine, 72(2), 134-139. Roff, S., McAleer, S., Harden, R. M., AlQahtani, M., Ahmed, A. U., Deza, H., Groenen, G., & Primparyon, P. (1997). Development and validation of the Dundee Ready Education Environment Measure (DREEM). Medical Teacher, 19(4), 295-299. Roff, S., S, M., Ifere, O. S., & Bhattacharaya, S. (2001). A global diagnostic tool for measuring educational environment: comparing Nigeria and Nepal. Medical Teacher, 23(4), 378-382. Rotem, A., Bloomfield, L., & Southon, G. (1996). The clinical learning environment. Israel J. of Medical Sciences, 32, 705-710. Rotem, A., Godwin, P., & Du, J. (1995). Learning in hospital settings. Teaching and Learning in Medicine, 7(4), 211-217. Rothman, A. I., & Ayoade, F. (1970). The development of a learning environment: a questionnaire for use in curriculum evaluation. J. of Medical Education, 45, 754-759. Saarikoski, M., Leino-Kilpi, H., & Warne, T. (2002). Clinical learning environment and supervision: testing a research instrument in an international comparative study. Nurse Education Today, 22(4), 340-349. Seabrook, M. A. (2003). Medical teachers' concerns about the clinical teaching context. Medical Education, 37, 213-222. Seabrook, M. A. (2004). Clinical students' initial reports of the educational climate in a single medical school. Medical Education, 38(6), 659-669. Shailer, B. (1990). Clinical learning environment audit. Nurse Education Today, 10, 220-227.