CMVM – UG External Examiners Reports
Shared by: sol61731
CMVM – UG External Examiners Reports 1) BVM&S Summary of External Examiner Reports for 2006-07 as received at 15-11-07 Course 1. Adequacy of 2. Types of 3. Assessment 4. Quality & 5. Administration 6. General information assessment criteria standards comments The animal Would like more Concerns about All satisfactory Course appears All satisfactory None body (GEP) information consistency in to be unbalanced beforehand on the oral exams in favour of course content reproduction Integrated All satisfactory Narrower range All satisfactory Low pass rate All satisfactory Standard Pathology & of assessments could be due to comparable Infectious compared with admissions with other vet Diseases (GEP) I&I might have criteria and lack schools in UK put some of experience students at a with easements disadvantage methods Body Systems I All satisfactory All satisfactory All satisfactory All satisfactory All satisfactory Educational provision & assessment excellent All satisfactory All satisfactory All satisfactory All satisfactory All satisfactory None Course 1. Adequacy of 2. Types of 3. Assessment 4. Quality & 5. Administration 6. General information assessment criteria standards comments I&I All satisfactory All satisfactory All satisfactory All satisfactory Initial problems due Standard to new staff but did comparable not affect the with other vet process schools in UK Body Systems All satisfactory All satisfactory All satisfactory Minor point All satisfactory None II suggesting the use of second internal examiner a means of moderation All satisfactory All satisfactory All satisfactory All satisfactory All satisfactory Standard comparable with ( probably exceeds) that other vet schools in UK Cells, tissues & All satisfactory All satisfactory All satisfactory All satisfactory All satisfactory Excellent Development course, high standards of teaching and pastoral care Course 1. Adequacy of 2. Types of 3. Assessment 4. Quality & 5. Administration 6. General information assessment criteria standards comments Animal All satisfactory All satisfactory All satisfactory Concerns about Unclear about how Concerns that Husbandry opportunities for disabled students some students students to are dealt with. arrive without practice AH Scaling of marks any farm outwith teaching should be animal time considered in future handling experience Cat and Dog All satisfactory All satisfactory All satisfactory All satisfactory All satisfactory Standards course comparable and probably better than other institutions Integrated All satisfactory Not answered Difficulty No answered Not answered Would like Pathology& following more Clinical course marking scheme communication IV & V for practical between examination School and examiner All satisfactory All satisfactory All satisfactory Some essay All satisfactory Standard is answers satisfactory indicating lacking in breadth of knowledge Course 1. Adequacy of 2. Types of 3. Assessment 4. Quality & 5. Administration 6. General information assessment criteria standards comments Integrated All satisfactory Some concerns Consistency of All satisfactory All satisfactory Spot and oral Pathology & about the marking of oral exam on same Clinical Course structure of the exam improved day could VI (final year) oral since 2005-06 disadvantage examinations but tendency not students. to use full 0-10 System of marking scale observers at with marks oral exam centring on 5-6 applauded Integrated All satisfactory All satisfactory All satisfactory All satisfactory All satisfactory Students Pathology & showed a high Clinical Course level of VI (final year) confidence, knowledge and communication skills All satisfactory All satisfactory All satisfactory High proportion Special None of credit awards Circumstances surprising procedures considered confusing “All satisfactory” indicates that the examiner was satisfied with all aspects of the course relating to this section of the report. Comments are included only if examiners indicated that they were not entirely satisfied with one or more aspects of the course relating to that particular section of the report. 2) MBChB Summary of key points from External Examiner’s Reports 2006/2007 Course Comments Year 1 Molecules to Society Assessment types generally good, but consider introducing more MCQ/EMQ (one best 1 answer from a list) objective marking. Very few A (highest) grades does not reflect your students’ abilities. Suggest review process for translating scores to grades so that outcome gives similar proportion of A grades as achieved in other medical schools. Happy to help with this review and it is important matter for future medical careers, given MTAS and FY1-2 applications. Difficulties with computerised mark handling – seems to be a resource issue, and one that must be addressed urgently otherwise an unreliable complex computer-based system is much worse than a paper-based one. Year 1 Molecules to Society High quality course 1 Train markers to improve consistency of marking. Not convinced that the exam should be basis for deciding progression to the next year AND for award of distinction and honours. Year 1 SSC 1 and 2 Excellent quality of work Good input and hard work from SSC tutors Year 2 Molecules to Society Scope for increasing MCQ or EMQ-type questions in assessment as these are more 2 reliable than SAQs, remove discrepancies between markers and are less arduous for staff, who can focus on writing questions rather than marking them. Currently some SAQs are being remarked by Year Director because some markers are hawkish while others are dovish in their marking. Impressed by care and effort devoted to the exam by the Year Director and his team. However, a few academics are carrying a very heavy burden and some academic staff seem not to regard assessment as a priority. May be helpful to have a large spreadsheet available for inclusion of secondary assessments. Year 2 Biology of Disease Year 3 OSCE Good standards and OSCE was well-conducted with well-briefed examiners. Was asked at short notice (as a Y5 examiner) to help out in Y3 OSCE as no Y3 external examiner was available on 30 May 2007. Two students were very challenging for students in the time allowed – demands might be reviewed as the CVS station “non-cardiac chest pain” would be challenging for a consultant cardiologist to complete. Similarly the chest examination station. Year 3 and 4 SSC Edinburgh is at forefront in developing and assessing SSC. High standard and excellent organization by a dynamic group Year 4 Impressed with the College’s processes and standards. Care – pastoral, academic, administrative – exercised by the examining board towards candidates is exemplary. Year 5 Clinical Finals and MBChB final assessment is well considered, balanced and rigorous. It is applied fairly resits with appropriate attention to special circumstances. The team should be congratulated. One element, prescribing, merits further consideration as some differences in views between individual markers about what the station is designed to assess, and this was reflected in the variability of marks (some hawks, some doves). May need guidance, feedback and/or training of examiners. Not clear what the process for external review of the written exam is, and what QA is applied to this aspect of final assessment. Year 5 Clinical finals Well-conducted exam with appropriate range of clinical scenarios and appropriate assessment of competencies. Standards comparable with other medical schools in Scotland. Excellent Y5 examiners’ handbook. Year 5 Portfolio Viva and Favourably impressed by Y5 portfolio viva. Final examination Board Suggest some minor adjustments which the Board might wish to consider for next year. Congratulate the course organisers of the portfolio in general and the viva in particular. 3) BMTO External Examiner Reports and Responses of Chair of Exam Board (CEB) or Course Organiser (CO). Medical Biology 1: The standard setting process for the MCQ paper needs to be clarified. There was some confusion as to whether students who had submitted extenuating circumstances forms should have “S” appended against their mark in the university records. CEB: The difficulty of each (MCQ) question will (now) be determined from previous usage or opinion of the question setter. The composition of the paper will be such that scaling of marks will not be necessary. The results for all students deemed to have Special Circumstances will be flagged in the list returned to Registry. Anatomy and Pathology 2: Thorough, fair and rigorous. The course is an exciting and novel link between anatomy and pathology. Chemical Pharmacology 2: I am of the firm opinion that the Chemical Pharmacology 2 course is effectively managed and provides chemistry students with a broad perspective of many important areas of this discipline. Neuroscience with Pharmacology 2: We were all disappointed with the high number of fails and spent a long time in the exam board discussing reasons and outcomes for this. I must say that in my opinion the staff are not to blame for this in the least. A whole day to look at all of the written material before the board meeting might have been useful – the morning only was a little tight. CEB: We will endeavour to send Prof Robertson a provisional copy of the spreadsheets of results. We will offer him the opportunity to come to Edinburgh the day before the Board meeting. Physiology 2: This year, the number of A grades was slightly down on the previous year but the overall performance of the students was consistent with what might be expected from what I think is a fair but relatively demanding course. Mechanisms of Brain Development 3: I was very disappointed to hear that the university will be changing the management of the examiners meeting in favour of a much larger grouping where a wide range of examinations would be discussed. I strongly believe that such a move would undermine the quality of your assessment procedures. The small examiners groups present until this year have the time to discuss individual issues at length in a relatively informal atmosphere. I experienced exactly what you propose to do while working as an external examiner at a London Medical School. The changes introduced an apparent air of efficiency and an assumption that standards were levelled across examinations. The reality was that we did not have the time to deal with many issues in the way that we should have and an air of formality was introduced that stifled discussion CEB: The Board agrees with the External Examiner Medical Microbiology 3: I was surprised to see that non-staff members are being used to mark the Practical Reports given that these form an important part of the final degree mark. I would think that is would be safer to restrict the marking of work that counts to the final degree classification to the core staff group. The Course Convener and the Chair of the Board of Examiners are separate individuals and the Director of the BMTO sits on the board. Perhaps as a result, there did not appear to be one individual taking responsibility for the organisation of the process and leading the board in its dealings. CO: Restricting (marking) to one or two core staff would possibly ensure greater uniformity but this becomes increasingly difficult as staff numbers continue to dwindle. The solution is to remove this from the course assessment altogether. This has already had the approval of the teachers involved. The examiner may have been confused because Dr Stewart was present and some important issues were directed to him, as he is the expert in this area. However, this confusion may not exist in future as Dr Stewart will no longer sit on the exam board. Pharmacology 3, The mean marks for each component were acceptable, though it is clear that there is a long tail of weaker students as is seen in other institutions with which I am familiar. The quality and standards of the course and assessment are comparable to those of my own department. There are no apparent shortcomings or differences. Physiology 3: The assessment procedures are very fair and are applied consistently to all students. Practical Skills in the Biomedical Sciences 3: This year, the modal grade distribution was “B” which is in line with expectations though there were rather few A grades. CEB: The standards set were the same as in previous years in which more ‘A’ grades were attained. The examining of this course is multifaceted, so students will have to do very well in most components to obtain an overall ‘A’ grade. Consequently, the marking of each component will be carefully monitored this year. Epidemiology Honours 4: These students are studying for a BSc but their coursework is assessed as if they were studying for an MSc. I think it is worth considering increasing the credits on this course and converting it to a Masters course. CO: A very small number of students (usually 1 to 3 per year) taking this programme are integrated (and assessed anonymously) alongside the MSc in Public Health Research. This is felt to be beneficial to both sets of students. The BSc students take fewer taught courses than the MSc students and also two of the courses they take, which together make up 50% of the overall credits for the programme, are assessed at BSc level. Overall the BSc students are found to perform extremely well when compared with the MSc students, usually coming within at least the top 50% of the overall student group. Experimental Pathology 4: No problems. Transparent and fair. Medical Biology Honours 4: Where model answers were supplied this was of tremendous help. I would recommend that this procedure of providing the essentials of the expected answer be consistently applied to all those who set exam questions. The one issue still remaining to be solved is the discrepancy between the manner in which medical students and the students from other courses are dealt with in terms of final marks and grades. I highlighted this issue last year, and while some leeway has been given in that borderline cases, between grades, can now be considered in light of which stream they belong to, it is still a rather iniquitous situation. Indeed it lends itself to legal challenge one day, in my view. CEB: This year, ‘essentials of expected answer’ were obtained and sent to the External Examiner. Physiology Honours 4: Science and intercalating medical students take the exact same level 4 assessments. It is possible for a science student to outperform a medic across the range of level 4 assessments and yet obtain a lower class of degree. This is because of the carry forward of 50% from level 3 for the science student. CEB: This problem is outside the control of individual courses as the decision to apply this split between 3 rd and 4th year marks is made at University level. Thus this exam board believes the impact of the University decision to retain the 50/50 split for science students should be re-evaluated. Pharmacology Honours 4: The weighting of level 3 results (50%) is in my opinion excessive, but I understand this to be a University-wide policy and not an issue specifically to this degree board. My sole concern relating to the students’ work related to the retention of core knowledge from previous years. It is possible that the carry over of marks from level 3 encourages students to believe that core knowledge has already been assessed and is no longer of great concern in the Honours year. CEB: It has been a particular problem over the years of students carrying over core knowledge from previous years. This is why a paper on general Pharmacology was introduced several years ago which was subsequently transformed into the synoptic exam. The Programme Organiser and I welcomed the view that the external examiner was very keen on this form of exam. Neuroscience Honours 4: 1, Dissertations: a). The marking scheme should take more account of library projects (perhaps by having a separate scheme). b. It would be helpful to incorporate a short report from the project supervisor… and dedicate a minor component of the overall mark (e.g. 10%) to the supervisor’s assessment. c. The criteria for a first class mark were very steep and may have reduced the spread of marks. d. Experimental projects involving use of animals (or animal tissue) should state in the methods that the work is conducted within the Animals (Scientific Procedures) Act 1986 and give the project licence. 2. Synoptic Paper: There was evidence of students being very selective in choosing to answer question 1 or 5. It seems they were making the best of a failed question spotting exercise by adapting a prepared answer to these two general questions. Some polarisation of the marking suggested that some markers were putting more emphasis on “general knowledge” while others focussed on whether the “questions had been answered”. One way to focus the marking would be to present a list of around 10 bullet points of expected components to an answer at the time of question setting. Final degree classification of medical and science students: The inclusion of 50% of the third year mark to assign final degree classification has a negative effect on the grading of some students. Consequently some medical students received a First classification while other science students with higher absolute fourth year marks received a 2:1 classification. CEB It is envisaged that the following modifications will be implemented: 1. Dissertation. a Separate marking schemes will be issued for library project and lab projects. b A short report from the supervisor will be obtained. Whether the supervisor should contribute to the assessment will be considered again when the exam is set. c The criteria for a first class mark will be modified and markers encouraged to use the entire mark breadth. d A statement of compliance with animal legislation will be required. 2. Synoptic Paper: This issue will be addressed at question setting (reducing the potential for regurgitation) and by guidance to students prior to exam. Where appropriate a bullet point/specimen answer will be provided to markers. The board of examiners share with the external examiner concern about the fairness of the 50% mark split. Microbiology and Infection Honours 4: There was some inconsistency in the way that the (marking) criteria were applied to some aspects of the course. For example there were instances where an individual marker had used a single right/wrong marking scheme. A number of students were significantly held back by their 3 rd year marks. On the other hand, class averages where somewhat high in certain components of the honours examination. While these factors balanced out to give a good spread of marks, care needs to be taken to ensure that future imbalances do not emerge. The course convenor, the chair of the exam board and the Director of the BMTO sit on this board. There did not appear to be one individual taking responsibility for the organisation of the process and leading the board. The timing of the research project in the final year should be considered. The students are often applying for PhD positions having not yet started the project. CO: All questions set for the Current Topics Exam and Research Methods Exam should have an element of interpretation to allow the agreed marking criteria and descriptors to be applied. Dr John Stewart provides valuable information and advice to the Chair and the Exam Board. Care will be taken to explain the workings of the Board to future External Examiners, so that confusion does not arise. The timing of the research project is common to, and is determined to a large extent by, all the other biological and biomedical honours courses. It is therefore not possible to alter the timing of the research project. Junior Honours Progression Board: It is likely that in order to review the assessment procedures in detail, external examiners will be required to spend considerably more time reviewing scripts (if they are to be responsible for cognate groupings of subjects). In practice, this may mean that external examiners may need significantly more time in order to fulfil their commitments. All courses should be encouraged to present assessment data in a relatively consistent format to include the means and SDs of each assessed item of the course etc. A frequency distribution, means and medians might also be useful to enable the final course mark and grades to be compared across all the JH modules. My understanding is that, in future, the number of external examiners is to be reduced by re-allocating cognate subject groupings to a smaller group of externals. Whilst I would feel reasonably confident of reviewing pre-honours examination papers and procedures I would feel much less confident in reviewing specialist Honours courses. I would be concerned about determining the standard of assessment and comparability with degree programmes at other institutions in subjects areas that I am unfamiliar with. Director of BMTO: All spreadsheets will be produced to the same format and show statistical analysis of results. External Examiners will continue to scrutinise exam papers and perform all other activities required to fulfil university regulations and guidance. External Examiners who are experts in specific subject areas will continue to be recruited.