Best Practices in Student Learning Outcomes Assessment

Best Practices in Student Learning Outcomes Assessment Assessing Prior Knowledge Assessing Learning 2/26/03 Knowing What Students Think 1 Knowing what Students Think Techniques for Assessing Prior Knowledge 2/26/03 Knowing What Students Think 2 “It’s not what we don’t know that gives us trouble. It’s what we know that ain’t so.” Will Rogers 2/26/03 Knowing What Students Think 3 Why Assess Prior Knowledge?  Create class/individual knowledge profiles  Identify missing knowledge  Reveal and skills  Provide appropriate guidance to students remedial materials for unprepared students  Help students evaluate their current likelihood for success  Students  Provide misconceptions  Reveal fragile knowledge  Serves as an advanced organizer Evaluate value-added  Pre-Test learn what information and skills they will be expected to know and Post-Test Knowing What Students Think  2/26/03 4 Assessment Techniques  Inventories  Conceptual  Visual Representations Performance-based Activities  “Auditions”  Mini-Projects  Content Maps  Mind Maps  Concept  2/26/03 Knowing What Students Think 5 Inventories  Concept Inventories Identify how students reason about critical concepts, principles, etc.  Frequently used to uncover systematic but incorrect “theories”  Distractor items are common misconceptions or errors  Pattern of responses across questions indicative of underlying “beliefs” or “theories”  Goal: 2/26/03 Knowing What Students Think 6 Mechanics Baseline Test (Hestenes & Welles, 1992) MBT Two pucks are on a frictionless table. Puck II is four times as massive as puck I. Starting from rest, the pucks are pushed across the table by two equal forces. 20. Which puck will have the greater kinetic energy upon reaching the finish line? A) I B) II C) They both have the same amount. D) Too little information to answer Finish 21. Which puck will reach the finish line first? A) I B) II C) They will reach it at the same time. D) Too little information to answer I m 4m II 22. Which will have the greater momentum at the finish line? A) I B) II C) They will reach it at the same time. D) Too little information to answer Knowing What Students Think F F 7 2/26/03 The diagram depicts a block sliding along a frictionless ramp. The eight numbered arrows in the diagram represent directions to be referred to when answering the questions. 4. The direction of the acceleration of the block, when in position I, is best represented by which of the arrows in the diagram? A) 1 B) 2 C) 4 D) 5 E) None of the arrows; the acceleration is zero. 5. The direction of the acceleration of the block, when in position II, is best represented by which of the arrows in the diagram? A) 1 B) 3 C) 5 D) 7 E) None of the arrows; the acceleration is zero. 1 8 2 7 6 5 4 3 I III II 6. The direction of the acceleration of the block, (after leaving the ramp) at Knowing What 2/26/03 position III, is best represented Students Think by which of the arrows in the 8 Impact on Instruction  Goals:  To  Method  Divide  focus attention on important concepts  Stimulate active student thinking in large classes class hour into 15-minute chunks     Results 10-minute lecture - ends with qualitative question 1-minute to think about question 2-minutes to justify answers with a neighbor. 2-minutes instructor guides class to resolution of question  Greater gains on MBT, higher exam scores, fewer failures  2/26/03 Greatest impact at the low end of the distribution Knowing What Students Think 9 Statics Fundamentals Inventory Paul Steif (CMU)  Goals determine whether fundamental concepts and methods have been acquired.  To determine the relationship between performance on a concept on the SFI and performance on multifaceted problem.  To 2/26/03 Knowing What Students Think 10 For each question, circle your best answer. If you are uncertain of the answer, put a P next to any other answers that you think are somewhat likely to be correct. 1. Consider the configuration shown. A free body diagram is to be constructed which includes two of the weights (W2 and W3) and the cord connecting them. Which is the correct free body diagram? 2/26/03 Knowing What Students Think 11 Statics Inventory  Interesting Features understanding  Distractors are conceptually-based  Can distinguish “strong” vs “weak” misconceptions (possibility rating)  Qualitative vs Quantitative reasoning  Identifies conceptual 2/26/03 Knowing What Students Think 12 Content Inventories  Measure of student’s declarative & procedural knowledge.  What facts, procedures, concepts do students possess?  What level of expertise do students possess?  Techniques test (Direct Measure)  Self-Report (Indirect Measure)  Baseline 2/26/03 Knowing What Students Think 13 Content Inventory (indirect) How familiar are you with the concepts of “Karnaugh maps”?  Have never heard of them  Have heard of them but don’t know what they mean  Have some idea what they mean, but not too clear  Have a clear idea what they mean, have used them & can explain them 2. Have you designed or built a digital logic circuit?  I have neither designed nor built one  I have designed one, but have never built one  I have built one but have not designed one  I have both designed and built a digital logic circuit 1. 2/26/03 Knowing What Students Think 14 Ideas for Content Items  Procedural Knowledge  Operating  type of equipment, using tools Using a lighting board, designing in Illustrator, analyzing data in matlab  Performing specific  tasks  Declarative/Conceptual Knowledge  Within category  directing a play, gene splitting, writing a narrative, giving an impromptu speech, pitching a product  Number of programming or foreign languages, styles of architecture, musical styles Across categories Knowing What Students Think 2/26/03 15 How to Use Inventories  Identify the critical concepts and skills for the domain Formulate tasks/questions addressing the kind of knowledge you are interested in  Prerequisite vs Target Skills (or both)?   Consider your evaluation criteria  What level   Declarative, procedural, contextual, strategic, etc of performance is “acceptable”?  How will you use the results? Knowing What Students Think Norm-referenced vs criterion-referenced? 2/26/03 16 Concept Maps  Visual, schematic representation concepts  Relationships among the concepts  Hierarchically arranged  Inclusive to less inclusive  General to specific  Key 2/26/03 Knowing What Students Think 17 Concept Map on Reconstruction 2/26/03 Knowing What Students Think 18 Why use concept maps?    Externalize learner’s knowledge:  Structural Knowledge (types & number of links)  Can see how students integrate & represent information  Conceptual Knowledge (via links and concepts)  Makes misconceptions explicit (wrong links).  Allows for finely targeted remediation.  Declarative Knowledge  Makes knowledge gaps explicit (missing nodes or links). Helps students develop meta-knowledge. Facilitates encoding, recall, and higher level reasoning. 2/26/03 Knowing What Students Think 19 Concept map on Taste A) 2/26/03 Knowing What Students Think 20 Concept map on Taste B) 2/26/03 Knowing What Students Think 21 How to Use Concept Maps   Create an “Expert” map Determine your goal How will you assess the maps?  Number are you looking for?  What kind of map activity will best meet that goal? of correct concepts (absolute vs relative)?  Number of correct links (absolute vs relative)?  Number of incorrect concepts or links  What information   How will you use the results? 2/26/03 Knowing What Students Think 22 2/26/03 Knowing What Students Think 23 2/26/03 Knowing What Students Think 24 Mind Maps    Visual, schematic representation Key concept Relationships to key concept or characteristics  Major components  Steps or sequence information vs non-causal  Hierarchical vs Flat 26  Properties  Concept vs Mind Map  Causal 2/26/03 Knowing What Students Think Sample Mind Map Misconceptions Prior Knowledge Missing concepts Fragile Not Accessible Disconnected 2/26/03 Knowing What Students Think 27 Cautionary Note knowledge  Surface vs deep knowledge  Sophistication of links  Underestimating student knowledge  Understanding concepts vs knowing labels  Students vs expert terminology  Overestimating student 2/26/03 Knowing What Students Think 28 Performance-based Activities  Types of Performance Activities  “Auditions”  Live in-class performance task (act, sing, dance, draw, write, speak foreign language, etc) Apply knowledge & skills to solve problem, create artifact, etc.  Mini-Projects   Assessing Performance using Rubrics  Key   Dimensions Speech: eye contact, voice projection, use of visual aids, etc; Music: tempo,accuracy, interpretation  Qualities   of performance Rating scales: Excellent- Poor, Acceptable-Unacceptable, etc Annotations: description of problems, ideas for instruction, Knowing What Students Think 2/26/03 30 Assessment & Learning What’s the connection? 2/26/03 Knowing What Students Think 31 A coin is tossed into the air and follows the path shown in A. Disregarding any effects of air resistance, in which diagram (B or C) do the arrows correctly represent the forces acting on the coin on the a) upward and b) downward parts of its trajectory? a a b b A B C 2/26/03 Knowing What Students Think 32 What’s your best guess?  What percentage of high school physics graduates get the problem correct?  12%  What percent of students successfully completing their first semester of college physics get the problem correct?  28%  What percent of students successfully completing a second semester of college physics get the problem correct?  30% 2/26/03 Knowing What Students Think 33 What do you do when your class performs poorly on a test?         Adjust the scores upward. Grade on a curve. Adjust the difficulty of the next assessment. Give advice on study strategies. Long for the “good old days” Provide more practice problems, review. Spend more time on the material in the future. Change your teaching materials or approach. 2/26/03 Knowing What Students Think 34 How do you know what to change in your instruction? • Integrating assessment into the instructionlearning cycle can help us determine: • • • • If learning failures have occurred Why learning failures have occurred, What an effective course of action might be Whether our action has been effective. 2/26/03 Knowing What Students Think 35 What does assessment have to do with learning?     Good assessments provide opportunities for students to both learn and reveal their learning. All learning requires feedback but the feedback must be informative.  - Feedback must provide guidance to the student and the instructor on what to do to improve performance. Learners must learn how to use feedback to improve performance Grades or scores help professors and students monitor learning, but they do little to promote learning 2/26/03 Knowing What Students Think 36 What does Assessment have to do with Instru Instruction-Influenced Assessment Objectives Instructi on Assessment Often we don’t have a clear vision of what or how we will assess until after we have designed or completed instruction. 2/26/03 Knowing What Students Think 37 Assessment-influenced Instruction Formative Assessment Objectives Instruction Assessment 2/26/03 Knowing What Students Think 38 Advantages of the Assessment-influenced Instruction • Accurate task analysis • Helps us pinpoint the knowledge and skills that need to be addressed. • Relevant practice activities • Helps us to design guided and independent practice activities that are better aligned with desired outcomes. • Lucid Expositions • Helps us provide clearer explanations to students regarding the purposes and goals of the instructional and assessment activities. 2/26/03 Knowing What Students Think 39 The Assessment Triangle Observation Tasks, situations for students to practice & demonstrate their knowledge & skills Interpretation Methods & tools to reason about observations (both formal and informal) Cognition Theories of learning Knowledge & skills of domain 2/26/03 Knowing What Students Think 40 Observation Interpretation Cognition    What skills and knowledge are required and valued in my domain? How do the knowledge and skills develop?  What is the trajectory from novice to expert?  Are there popular misconceptions students possess?  Are there consistent points of difficulty  Are there prerequisites that students may not have How are these domain skills combined with general skills?  Problem solving skills (planning, decomposition, result checking, etc) when faced with a complex project or assignment. Knowing What Students Think 2/26/03 41   What activities will enable students to demonstrate, practice and extend their skills and knowledge? How will I identify varying levels of competency?  What would characterize & differentiate    excellent performance? competent performance? unacceptable performance?   Will the feedback on their performance be informative and supportive of their learning?  Grades are not very informative or supportive! Will the feedback on their performance be informative to my instruction? Knowing What Students Think 2/26/03 42 Observations Interpretation Cognition  How will I interpret the observations?  Informal, intuitive or qualitative  Rubrics: Explicit characteristics of different levels of work  Norm or Criterion Referenced  Item analysis & Rule analysis  Frequencies & Patterns of correct and wrong answers 43 2/26/03 Knowing What Students Think All good assessment connects these 3 components Observations Interpretation Every assessment: Cognition •Is based in a “theory” of how people learn, what they know and how knowledge and skill progresses over time •Embodies assumptions about which tasks are most likely to elicit demonstrations of what students know and can do. •Is based on assumptions about how best to interpret the outcomes to draw meaningful inferences about what students know and can do. Knowing What Students Think 2/26/03 44 Example from Physics Observations Interpretation (Item Analysis) 45%: it will weigh less because no air pressing down on it. 25%: it will weigh more because no air under scale holding it up. 30%: it will weigh the same because the removal of the up & down force of air pressure balance out A block on a scale weighs 10lb. If the scale and the block are covered and sealed & all the air is removed, how much will the block weigh? Why? Cognition Prior belief - Air pressure affects weight of objects 2/26/03 Knowing What Students Think 45 Integrating & Aligning Instruction & Assessment Bloom’s taxonomy* Knowledge Dimension  Factual -basic elements or components of domain needed to solve problems in it  Conceptual - interrelationships among elements, how they function  Procedural - how to do task, methods of domain, criteria for using skills, techniques, methods, etc  Meta-cognitive - knowledge of cognition, such as how strategies can help, requirements of different tasks, & awareness of one’s own cognition, such as strengths, weaknesses, etc *Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives.Longman: New York, NY. 2/26/03 Knowing What Students Think 46 Cognitive Processes Dimension Remember - recall, recognize, identify  Understand - interpret, exemplify, classify, infer, summarize, explain, compare   Interpret - clarify, paraphrase, translate, represent Apply - execute, implement, use, carry out  Analyze - differentiate, distinguish, organize, attribute, select, parse  Evaluate - check, critique, judge, monitor, test  Create - generate, plan, produce, construct, hypothesize  2/26/03 *Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives.Longman: New Knowing What Students Think York, NY. 47 Cognitive Processes Knowledge Dimension Factual Conceptual Remember Understand Apply Analyze Evaluate Create Procedural Metacognitive 2/26/03 Knowing What Students Think 48 History Example  Learning Objectives  Understand the factors that led to the dominance of the English in the Americas  Remember the timeline of important historical events surrounding the settlement of the English in America  Draw conclusions or explanations using available evidence of historical events and circumstances  What kinds of assessment tasks would enable you and your students to measure progress towards these goals? Knowing What Students Think 2/26/03 49 Assessment Question: Question: What was the date of the battle of the Spanish Armada? Student 1: 1588 [correct] Student 2: Around 1590 [close but incorrect] What learning objective does this question address? •What does this tell us about their level of knowledge of the event? •What does this tell us about the students’ representations and organization of knowledge? •What does this tell us about their ability to use evidence to make inferences, draw conclusions? •How would student Knowing What Students Think performance impact further 2/26/03 50 Assessment Question Revised Give the date of the Spanish Armada and provide some historical evidence that would support your answer. Student 1: 1588 - I memorized it from the textbook. Student 2: Around 1590. The English began to settle Virginia just after 1600, not sure of the exact date. They wouldn’t have dared start overseas exploration if Spain still had control of the seas because of the danger and potential loss of money, ships, etc. It would take awhile to get expeditions organized, so England must have gained naval supremacy around the late 1500s. 2/26/03 Knowing What Students Think 51 What learning objective does this question address? • What does this tell us about their level of knowledge of the event? • What does this tell us about the students’ representations and organization of knowledge? • What does this tell us about their ability to use evidence to make inferences, draw conclusions? • How would student performance impact further instruction? 2/26/03 Knowing What Students Think 52 Interpreting Performance • Responses to the first question would be: • objectively & quantitatively scored • Interpretation: Student 1 rated above Student 2. • Responses to the revised question could be: • objectively & quantitatively scored • Qualitatively scored based on appropriate and accurate use of evidence, reasonableness of inference, remembering relevant events in timeline, etc • Interpretation: Student 2 rated above Student 1. • If this was the type of answer you characterize as “Excellent” what kind of instructional activities would support it? 53 2/26/03 Knowing What Students Think Using rubrics to assess performance   What is a rubric? A rubric is the scoring rules or criteria against which student work will be judged. One purpose of a rubric is to make public the key criteria that students can use in developing, revising, and judging their own work. 2/26/03 Knowing What Students Think 54 Components of a Rubric  Levels of Mastery  Classification of performance.  Excellent, Good, Needs Improvement, Unacceptable,  Exemplary, Proficient, Acceptable, Unacceptable. Listing of the components that comprise quality performance  E.g., written communication, logical reasoning, executing ANOVA  Dimensions of Quality   Organizational Groupings  Skill dimensions subdivided into component skills.  E.g., Written communication maybe assessed on grammar, organization, appropriateness for audience, use of evidence, documenting sources, etc)  Commentaries Information on the critical features& standards for each of the mastery levels.  e.g., translation task: “Excellent” in grammar: “no grammatical or spelling errors, uses most appropriate sentence structure, etc,  “Good”: “few minor grammatical and spelling errors, occasional awkwardness in sentence construction, etc.  2/26/03 Knowing What Students Think 55 Why do Assessment-driven instruction?  It helps us to design tests, homework and practice activities that are better aligned with desired outcomes.  Avoid the pitfall of busywork  It provides instructors with a clearer view of how to design and structure instructional activities to better support student learning toward meeting performance objectives It helps us more clearly explain and justify to ourselves and students the purposes and goals of instructional and assessment activities “If we don’t know where we are going, how will we know how to get there?”  2/26/03 Knowing What Students Think 56