Making Students' Thinking Visible

Document Sample
Making Students' Thinking Visible Powered By Docstoc
					                                                                                    Lesson Study Supplement

                                 Gathering Evidence of Student Learning
                                              Bill Cerbin

To investigate how students learn during a lesson instructors need to
    1. design a lesson in which students’ thinking is open to observation and analysis.
    2. determine what aspects of student work and activity count as evidence of learning.
    3. determine how to collect evidence of student learning systematically.

Making students’ thinking visible during the lesson. The object of study—student learning—should
be open to observation and analysis during the lesson. Student thinking is visible or audible when they
answer questions, participate in discussions, work out problems, write answers, and demonstrate skilled

In most classes it is not too difficult to design a lesson in which students externalize their thinking. But it
is important to observe student thinking throughout the lesson not just at the end of the period. And,
instructors need to make sure that the class activities evoke students’ thinking relevant to the learning
goal(s) of the lesson. Even in highly interactive classes, instructors may want to include tasks or prompts
during the lesson to insure access to key aspects of student learning. Ideally, the task would be a
pedagogically relevant activity. For example, UWL Biology instructors sometimes use problem solving
activities in which students draw a model of a particular process or idea. These are visible representations
of their understanding of the topic. In large classes instructors can use a technique called “think-pair-
share.” The instructor poses a thought-provoking question, and gives students a minute or two to write a
response. Students then compare their responses to classmates sitting next to them. Finally, the instructor
asks several students who have different views to report their responses to the entire class. These
techniques makes students’ thinking visible and help students develop better understanding of the

Evidence of student learning. Lesson study focuses on the interaction between teaching and learning.
Ideally we want to monitor the way students interpret and construe the material, and how the instructional
activities support student learning throughout the lesson. Simply put, we want evidence of what students
“get” and how they “got” it.

   Evidence related to the lesson’s learning goal(s). First and foremost it is important to collect evidence
   of student learning related to the lesson’s goal(s). For example, the UWL Psychology Lesson Study
   Group designed a lesson to develop students’ ability to “think in terms of multiple factors that
   influence behavior.” The group wrote scenarios depicting certain types of social behavior and asked
   students to predict and explain how the protagonists would act. Students wrote individual responses to
   the scenarios before class, discussed their responses during class and wrote responses to a new set of
   scenarios after class. This produced information directly related to how students think in terms of
   multiple variables prior to, during and after the lesson.

   Evidence about how the lesson “works.” It is also important to collect evidence about how the
   instructional activities influence or affect student learning. In the psychology lesson, for example,
   students worked in groups for much of the period. Team members observed the student groups in
   order to learn how student interaction and discussion led to productive ideas, confusion,
   misconceptions or irrelevant activity.
Collecting evidence of student learning systematically. There are times during class when teachers
would like to freeze the moment, peer into the minds of students and think of how best to respond to
them. Sometimes we do freeze the moment by asking the class to write a “minute paper” or poll the class
on their understanding of a concept under discussion. More often, teachers observe on the fly; we listen as
students discuss a topic, circulate to hear what groups are doing, think about the meaning of a student’s
question and how best to respond. It is difficult for a teacher to attend simultaneously to teaching the
lesson and also attend to how various students are thinking about the subject. As one Japanese teacher
commented, “A lesson is like a swiftly flowing river.” For the most part, teachers’ impressions of student
learning are based on fragmented and scattered observations.

In lesson study instructors collect more complete and representative samples of student learning and
thinking. The key to collecting good evidence during a lesson is deciding in advance
    1. who to observe (e.g., the entire class, small groups of students, individual students)
    2. what to observe (student-student interactions, types of questions students ask in class, student
        responses to instructor questions, etc.)
    3. how to observe (e.g., use a rubric to categorize responses, keep a written record of student activity,
        use focal questions to look for certain types of responses)

Develop an observation protocol. The group should develop guidelines for observers. The protocol should
    1. an overview of the lesson so that observers know what to expect during the class period
    2. guidelines for observing the lesson

There is no single best method for observing students in class, but consider a combination of structured
and unstructured observation strategies. A structured observation focuses on certain forms of behavior.
Observers can use focal questions, rubrics, or charts during the lesson to record certain types of responses
and their frequency. Unstructured observations involve observers in keeping a written record of student
activity throughout the lesson (i.e., observers record what they see).

We recommend that groups try to collect as much information as possible the first time the lesson is
taught. If you focus narrowly on one aspect of student learning you may miss unexpected and interesting
features of the activity. Videotape can add to the record of the lesson, but it is not a substitute for multiple
observers in the classroom. Videotape can depict only limited portions of the activity. Moreover, it is
difficult to record good audio in classrooms. In some cases you may have a visual record of student
activity that is inaudible.

An example of an observation protocol from the Psychology Lesson Study group is attached. The group
videotaped their research lesson and had four observers in the room; two members of the lesson study
team and two psychology instructors unfamiliar with the lesson. Their protocol included:
      Information about the lesson—the topic, the rationale for the lesson, goals of the lesson, the lesson
      Focal questions and suggestions about what to look for in the class.
      An evaluation form with questions about how students interacted in small groups during the lesson
      Questions about what worked well and did not work well and suggestions for improving the

                               SAMPLE OBSERVATION PROTOCOL

                                        CONSTRUCT VALIDITY
                                      Introduction to the problem

Construct validity, addresses the question “Does this test measure what it claims to measure?”
Construct validity is theory dependent. Given the theory about some construct (i.e. a hypothetical trait
such as depression or intelligence), we can make predictions about how people with high or low levels of
the construct might behave. For example, given current theory of the biology of depression, we can
predict that depressed individuals who take anti-depressant medications will experience lower levels of
depressed after 2 to 4 weeks of taking the medications. The logic of construct validity follows the theory.
If a test actually measures depression, depressed individuals should receive higher scores on the test
before they take medication than they do after they have been taking the medication for 2 to 4 weeks.

Students have had trouble understanding that logic. Another process for examining construct validity is
to look at group differences. For example, if a test does a good job of measuring depression, a group of
individuals who are diagnosed as depressed should score higher on the test than a group of individuals
who are not diagnosed as depressed. When students are asked this question on an exam, many
students become confused, and focus on demographic group differences while failing to address the
logic of how different groups should score differently on the test. The actual exam question and some
examples of student confusion are listed below.

Test question: One of the processes used to examine construct validity is examining group
differences. Explain the logic behind this process. (2 pts)

Some classic problems:
“The environment in which you are raised in has a huge effect on your performance. Sometime students
that have lower socioeconomic status tend to not do as well in academics because it may be due to
higher stress levels. Ethnicities also have to same effect sometimes. Your ethnicity may have biological
as well as environmental differences. How we are perceived sometime has an effect on how we do in
many aspect of society”

“The logic behind this is that people are very different from one another. Going from culture to culture or
even looking at gender, you can find many things that are different. We need to take this into
consideration when working questions on a test because someone may take a question differently than
another person.”

The following lesson was designed to help students discover and better understand (hopefully!) the logic
of construct validity.
                                         Goals of the Lesson

Short-term goals/objectives:
   1. To develop students’ understanding of construct validity
   2. To develop students’ understanding of the logic behind the evaluation of construct validity
   3. As a result of the lesson students should be able to explain several ways to assess whether a
       measure has acceptable construct validity, and demonstrate knowledge of the logic behind these

Long-term goals:
   1. To develop students’ ability to analyze and evaluate instruments that measure psychological
   2. To develop “informed skepticism” toward popular claims about human characteristics depicted in
       the media, in work settings and everyday life.
   3. To appreciate the differences between popular conceptions of human qualities and those based
       on careful measurement.
                                       CONSTRUCT VALIDITY
                                            Lesson Plan
                               (To be conducted by the lead instructor)

Step 1: 10 minutes

   o   Break students into groups and distribute group worksheets (direct students to introduce selves to
       group members)
   o   Describe basic task:
           students will be creating “mini” tests of depression (as they define it) and proposing
              research studies to determine if their tests actually do a good job in measuring depression
   o   Additional directions
           Take individual notes so that you have the information (you will be handing in the group
              worksheet) – accommodations for the “recorder”
           Be sure everyone reads their definition out loud
           Your group definition does not have to be “the perfect” definition, given time limit
           You will have about 10 minutes to complete this step

Step 2: 10 minutes

   o   Instruct groups to finish definitions and begin writing items
   o   Items should related to their definitions/essential characteristics of a depressed person
   o   Review likert scale anchors
   o   Remind them to spend about 10 minutes to complete step

Step 3: 30 to 40 minutes

   o   Instruct groups to finish writing items and begin to propose research studies to determine if their
       tests actually measures depression
   o   Remind them:
             Unlimited resources – funding, people from a variety of settings (e.g. clinics, university,
               whatever), ages, diagnoses, etc.
             Think about your definitions of depression, what you know about depression as you think
               about what results you might expect
   o   Monitor groups progress; if groups seem to be going well, let them continue
   o   If groups struggling, off track call groups attention
             Pick some examples to review

Step 4: 15 minutes

   o   Call for groups attention and provide instructions for final method
   o   Distribute additional measures
   o   Think about how you might use these tests to provide evidence of the ability of your test to
       measure depression well
            Hint: think about the statistical methods we have covered in class to this point

Step 5: 10 minutes

   o   After all groups have completed the task, if time, discuss some of the definitions and methods
   o   Have group members describe as worksheets are displayed on visualizer
   o   Save 5 minutes at end (if possible) for writing exercise:
            What was the most difficult part of this exercise?
            What was the most important thing you learned from today’s lesson?
            What is still confusing?

                                         CONSTRUCT VALIDITY
                                          Observation Protocol

The purpose of having many instructors observe the class is to gather as much information about the
process of the lesson as possible. Your primary task is to observe how the students conduct the
lesson and make some conclusions about how well the LESSON worked. In other words, please
note behaviors of the students and the benefits/difficulties of the lesson, NOT the behaviors of the
instructor !

You will be observing one group of approximately 5 students.

Given the goal of helping students understand the logic underlying construct validity, please look for
evidence/examples that students are tying their understanding of depression to how people should score
on their test, if, indeed their test does a good job at measuring depression.

Please do take notes on your group’s behavior. In addition to noting any good and poor examples of
their ability to think about the logic of construct validity, please also note such things as
             o How the group developed their definition of depression. Did they integrate their individual
                  definitions? Did they simply string their individual definitions together? Something else?
             o Any evidence that the students seemed interested and/or engaged in the lesson
             o Any derailing of the process
             o Any problems in the group dynamics (dominating members, quiet members, etc.)
             o Any problems understanding the directions
             o Anything else you think is substantial!

Please do not make comments to your group. I.e. do not correct any misconceptions, clarify instructions,

                                      CONSTRUCT VALIDITY
                                  Observer Reactions to the Lesson

Now that you have observed the lesson, please answer the following questions.

                                                    Totally                               Totally
                                                    Disagree                              Agree

   1. All members participated in the process           1      2    3      4    5     6      7

   2. The group was able to stay on track with          1      2    3      4    5     6      7
      the lesson (i.e. did not derail, discussing
      irrelevant information)

   3. The group seemed confused about the               1      2    3      4    5     6      7
      technical processes of the lesson

   4. The group seemed confused about the               1      2    3      4    5     6      7
      concepts the lesson was addressing

   5. The group seemed to understand the                1      2    3      4    5     6      7
      concept of construct validity

   6. The group seemed to understand the logic          1      2    3      4    5     6      7
      of construct validity

   7. Given your observations, what aspects of the lesson need to be changed? How could the lesson
      be improved?

   8. What aspects of the lesson should remain the same? What worked well?


Lingjuan Ma Lingjuan Ma MS
About work for China Compulsory Certification. Some of the documents come from Internet, if you hold the copyright please contact me by