REPORT ON EXAMINATION REVIEW The Australian College of Veterinary Scientists has since its inception in 1971 continued to monitor, review and refine the examination system in order to ensure that the examinations for membership and fellowship of the College are fair and rigorous. As part of this commitment to continual improvement, the College in 2007 commissioned Dr Gabrielle Matters, an independent educational expert from the Australian Council for Educational Research (ACER), to complete a review of the examination system. During the course of the preparation for and delivery of the 2007 examinations, Dr Matters was provided with full access to College activities. Dr Matters reviewed all of the steps in the process including: written documentation and procedures about the examinations; preparation and editing of examination papers; conduct and assessment of written, oral and practical examinations (including attendance at a number of oral and practical examinations at the Gold Coast International Hotel in July 2007). Dr Matters then delivered a report that incorporated an appraisal of College procedures against our own objectives and guidelines as well as against accepted standards for professional assessment. A brief summary of the conclusions is presented here. The report provided a ringing endorsement of the College, concluded that there were no serious deficiencies in College procedures. as well as identifying a number of areas with particular mention of: The general literature on professional assessment in other disciplines is transferable to the veterinary profession; College policies and procedures are well documented and the College was congratulated for its attention to detail and consideration of every possible contingency. College examinations have high face validity 1 and high levels of inter-rater agreement 2 which are critical aspects of good examinations. The College was congratulated on its ability to manage a large number of examinations in a mainly paper-based system to produce final results in a very short time frame.
There were a number of areas identified where further improvements may be made and the College has taken these recommendations on board and will be implementing a number of changes in the future. These included: The algorithms applied for calculating final scores, although mathematically accurate, could be made simpler. This refers to methods for aggregating and averaging examination scores. The College is working towards simpler methods of recording and aggregating scores including increased use of computer-based systems. This is an operational issue. Dr Matters noted that examiners were applying two different assessment approaches ranging from objective assessment (eg applying part marks based on detailed breakdowns of the expected or ideal components in an answer to a question) and holistic judgements (referred to as the wine judging model) of whether an answer was deemed to be above a pass threshold or not. Holistic assessment was described as a valid and acceptable approach for assessment provided that threshold standards are clearly defined and performance against these thresholds recorded. There is a need for ongoing routine analysis of examination performance including monitoring of inter-rater agreement on an annual basis and monitoring issues such as
1 2
validity refers to whether the examination actually measures what it is intended to measure agreement between different examiners who are examining candidates in the same subject
effect of gender and background on performance. The College will implement systems for more efficient electronic recording of examination performance with a major anticipated benefit being the facilitation of routine monitoring of a range of measures of examination performance. Dr Matters made a number of very useful comments about a range of issues including training for examiners, whether the current practice of offering choice in examinations (answer 3 from 5 for example) is actually fairer than requiring all candidates to answer the same questions, the editing process and the use of clearly understood common terms (describe, list, define, discuss etc)
The recommendations arising from the report have all been accepted by the College and will feed into the ongoing evolution of the examination system. The report can be downloaded in full from the College web site and the College welcomes feedback from members on the report. All feedback will be considered by the College a s part of our commitment to continual review and improvement. On behalf of the College I would like to extend my gratitude to Dr Gabrielle Matters for her interest and enthusiasm in completing this important review. The report is a wonderful endorsement of the efforts of a large number of people who make the examinations work year after year. In addition the report has provided the College with a number of very useful recommendations to ensure that we continue to improve and provide a fair, valid and rigorous examination system into the future. Nigel Perkins 14 August 2008