Guidelines for Writing the Evaluation Summary
For questions contact: Curriculum Evaluator Julie Foertsch, Ph.D., foertsch@wisc.edu
1. Keep the goals of the Evaluation Summary and the Committee in mind: The purpose of the Evaluation Summary is to clearly and concisely identify the aspects of a course that, if adjusted or improved, would lead to a more effective learning experience for a greater number of students. The document needs to be well-written, constructive, and persuasive if it is to succeed in encouraging course directors to listen to students’ perspectives on how they learn best and make improvements to their courses that are mindful of students’ needs. As an Evaluation Committee member, you have a chance to represent the views of your peers and make a real difference in how a particular medical school course evolves over time. The faculty, the administration, and your fellow students appreciate your giving this important task your time, your thoughtfulness, and your problem-solving skills. By participating in an Evaluation Committee, you will be developing your own competency in the ACGME areas of Systems-Based Practice, Problem-Based Learning and Improvement, and Professionalism, skills that will serve you well throughout your medical career. As an Evaluation Committee member, your job is to be a course reviewer who: a) uses the available resources to gain perspective on a course’s strengths, weaknesses, and logistical constraints; b) accurately represents the varied opinions of the students, not just your own opinions; c) is respectful and constructive in your writing and commentary; d) is responsive to requests from your fellow committee members and timely in meeting your committee’s deadlines. 2. Review all the information available and summarize the balance of what you have learned: You will be receiving a complete course evaluation report about two weeks after the course’s final exam that summarizes the ratings for each rating question and lists all the responses to each text question, ordered by topic (see Appendix for a complete list of the questions). Although you will only be responsible for attending to the two sections of the report that match your subcommittee assignments, you will find it useful to look at the ratings in other sections and perhaps even skim the comments in sections that seem related to your sections. After you have an overall sense of how the course was evaluated, you should focus on the two sections to which you have been assigned to write a summary. The comments will already be ordered (to the extent that they can be) by the topic they cover so that comments which say something similar are clustered together. A suggested process for writing the section summaries is: Read all the comments for a section, jotting notes about the theme or main point of a given cluster of comments. This gives you a sense of the main issues you will need to discuss. Note how many students felt a certain way, giving the numerical ratings somewhat more weight than the written comments, since not all students will choose to comment on the same topic, but most do give as rating. When summarizing comments, your summary should prioritize the opinions of the majority over comments made by only a small number of students, but it is not unusual to have fairly large numbers of students with opposing opinions about the same issue. For example, you may have 25 students who thought the course book was way too detailed and found themselves overwhelmed and unable to discern the key points, 15 who liked 1
having the details but wanted more guidance on which parts to focus on for an exam, and 5 who loved the book and relied upon it more than any other resource. Your summary should reflect this difference of opinion and look for a resolution that will help the greatest number of students get what they need. For example, if the course director used highlighting within the coursebook or a clearer, more concise set of course objectives to draw students’ attention to key sections, the needs of all three groups of students could be met. Draft some sentences about the key issues and approximately how many students felt that way (almost all, the vast majority, the majority, some, a few). You can state the actual number of students who commented on an issue if you think that is helpful, but keep in mind that not all students who feel a certain way bother to comment. (Note: If you are on the Overall Course Effectiveness Committee, Question 28 is the one for which listing the actual % of commenters who made each suggestion is most useful.) Comments that are made by a very small number of students shouldn’t make it into your summary unless the points are particularly good ones or the suggestions especially effective and easy to do. In those cases, you should make it clear that was a minority opinion by saying something like “A few students thought it would be helpful if” or “one student suggested.” Write objectively and in the third-person. Remember, you are representing all students who rated or commented on this course, not just stating your own opinions. Write a one-paragraph summary of each section you are reviewing, keeping in mind that a paragraph can be as short as a 1-2 sentences if students had few suggestions for improvement or were all in agreement on what to do. Give enough detail so that course directors know why a particular aspect of the course is problematic, but there is no need to belabor a point when they can read the comments for themselves. If many students comment about how strong that aspect of the course is, you should mention that and why students think it is helpful, but focus your summary on areas where improvements can be made. For each statement you make about something that students had trouble with, offer their suggestions for improvement. You can offer a suggestion of your own if it aligns with the majority opinion and would address the issues of most students. Reread your summary and edit it for grammar, constructive word choice, and concision. A well written, carefully worded summary is much more persuasive than a haphazardly written, harshly worded one. Share your summary with your 1-2 subcommittee members by the end of the week. Your subcommittee then has a week to read the summaries of each member, see where they align and misalign, and choose/construct a summary that incorporates the key issues or best suggestions from all the summaries. This paragraph-long consensus section summary should be emailed to the Committee Chair by the section summary deadline.
Here’s an example of what a section summary should look like: Course Materials: Many students commented on how helpful it was to have all of the course materials available online. Of the 55 students who commented on the coursebook, the majority felt it contained too many details whose importance was unclear. They said they had a hard time keeping up with the reading and thought the coursebook’s key points were lost among the details. A significant minority said they enjoyed having those details available, but even they wanted more guidance on which parts to focus on for exams. Suggestions for how to improve this included highlighting key concepts in the coursebook or clarifying through the course objectives or lectures which processes were the most important to memorize in detail. 2
3. Use the responses to the final evaluation question and the section summaries to discuss and write the 1-3 priorities for course improvement. During the Course Evaluation Committee Meeting (or “meeting period” if the committee cannot actually meet and this work is done online), the committee convenes to discuss what was learned by reading the course evaluations. All committee members should have read the compilation of section summaries and Medical Advisor’s notes sent out by the Committee Chair. The Chair then facilitates a discussion about the top priorities for course improvement. Each section summary will be a part of the final report, so committee members needn’t be concerned that the suggestions from their section won’t be seen if they are not considered one of the top priorities. Nonetheless, the group does need to discuss and decide upon 1-3 suggestions for improvement that could have the biggest impact for the greatest number of students. Generally, these suggestions will address issues that interfered with learning for a significant number of students, but they can also be suggestions that, while mentioned by only a minority, are simple, effective adjustments to make. The job of the Evaluation Committee is not only to draw attention to the course aspects that were the most problematic for learners but to offer reasonable and well-considered suggestions for how to address those issues. Hence, when writing the priorities for course improvement, briefly describe the problem and why it’s a problem, but then focus on one or two ways that problem could be solved. After the Evaluation Committee has reached a consensus as to what the priorities for improvement should be, the Committee Chair will draft a clear statement of those priorities and put it at the end of an Evaluation Summary that follows the format described below. 4. Follow a standard report structure for the Evaluation Summary: The Evaluation Summary written by each course committee should use a similar structure and formatting in order to maintain consistency across courses and make it easier for course directors to find the information they need for their course review. On the next page is the outline and formatting that all the Evaluation summaries should use. Like this guideline, your report should be a Word document in 12-point Times New Roman. An electronic copy of these guidelines and a report template that you may copy and paste is available on the Medical Students Website under Academic Resources/Course Evaluation/Guidelines for Writing the Evaluation Summary.
3
Evaluation Summary for [Name of Course]
Fall [or Spring] of 20XX This report, provided by the course’s 1X-student Evaluation Committee, summarizes the evaluation comments received from XX students [the total N stated in the evaluation report] who took the course in the Fall of 20XX. The purpose of this summary is to provide guidance in making course improvements that will have positive impacts on student learning and the overall effectiveness of this course. First we provide summaries of the comments received on eight key aspects of the course, followed by a list of the most popular suggestions for the one change that would have the biggest impact on overall course effectiveness. We conclude with the Evaluation Committee’s consensus on the top X priorities for course improvement, which should be taken into account when making adjustments for next year. Summaries of student suggestions regarding various course aspects: Learning Objectives: One paragraph summaries submitted by each subcommittee Content: Course Materials: Learning Activities: Assessments: Schedule: Faculty: Integration: Overall Course Effectiveness: Top Priorities for Course Improvement: After discussing the strengths and weaknesses of the course and the suggestions made above, the Evaluation Committee feels the following changes would have the most positive impact on the effectiveness of this course: 1) Reduce the number of… 2)
4
UWSMPH COURSE EVALUATION QUESTIONS To be completed online via OASIS between the day of the final and 7 days later. Students will not be able to see their final exam grade online until it is completed. The scale for all non-text questions: Strongly Agree 1 2 3 4 Text questions are in bold. LEARNING OBJECTIVES 1) The learning objectives clearly stated the important concepts that I should learn. 2) The learning objectives were at an appropriate level of detail. 3) How could the objectives be more helpful in directing your studying and learning? CONTENT 4) The content was presented in a logical sequence that facilitated learning. 5) Each topic was allotted an appropriate amount of time. 6) I understood the relevance of the content to medical practice and/or research. 7) How could the presentation and organization of course content be improved? COURSE MATERIALS (includes all written materials like coursebooks, texts, online references) 8) The course materials clearly explained important concepts. 9) The course materials are well-edited, with few errors, good formatting, and logical organization. 10) Please suggest improvements in the course materials. LEARNING ACTIVITIES 11) Taken as a whole, the course’s learning activities helped me to understand the important concepts. 12) This course actively engaged me in the content through learning activities such as problem-solving, hands-on learning, and peer discussions. 13) The learning activities gave me insight into how I will apply that learning to clinical practice and/or research. 14) How could the learning activities be made more effective? ASSESSMENTS 15) The assessments measured my understanding and application of important concepts. 16) To be successful in this course, I needed to demonstrate an understanding of the content that went beyond the memorization of facts. 17) I was given a reasonably clear idea of what to expect on the assessments. 18) Please suggest ways that the assessments could promote more effective learning of course content. SCHEDULE 19) Course activities were scheduled in a way that facilitated my learning. 20) My workload during this course was manageable. 5
5
6
Strongly Disagree 7
21) Please suggest improvements in the course schedule. FACULTY 22) The teaching in this course enhanced my understanding of course concepts. 23) Questions were welcome and help was available if I had difficulty with course material. 24) Please suggest ways for improving the effectiveness of the teaching in this course. INTEGRATION 25) This course was well integrated with related curriculum offerings. 26) What topics in this course were too redundant with other parts of the curriculum or not linked effectively to related topics in other courses? OVERALL COURSE EFFECTIVENESS 27) Overall, this course provided a good learning experience. 28) What one improvement would have the biggest impact on the effectiveness of this course?
6