Calibrated Peer ReviewTM
Writing, Collaboration, and
Critical Thinking
http://cpr.molsci.ucla.edu
Arlene A. Russell, UCLA
University of Maryland
November 13, 2003
http://cpr.molsci.ucla.edu
Calibrated Peer Review (CPR)
• was created under the Molecular Science Project,
an NSF-funded systemic reform project
• is an instructional tool that uses writing and peer
review to teach higher-order thinking skills and
peer collaboration
• manages the writing and anonymous review
processes of instructor-selected assignments
• Integrate telecommunications and
technology into instructional processes so
that students learn to
• explore ideas independently
• write about chemistry
• collaborate
• work to a mastery level
• take responsibility for their own learning
Tenets of CPR
• Expository writing promotes understanding
• Clear writing demonstrates clear thinking
• Peer review and evaluation require higher-
order, critical thinking skills
• Writing-to-learn Learning-to-write
– But, writing-to-learn can (and should) include
attention to writing skills
Writing-to-Learn
• Used extensively in humanities and social sciences
• Requires small classes
– Science and engineering have put their resources into labs
• Imposes intensive grading workload
~~
• Seldom used in large science classes, because
– Workload too high, not the job of scientists, large classes, not
objective,…….
Peer Review
• Accepted process for validating scientific
research
• Essential skill required of all scientists
~~
• Seldom taught in science classes
• Not used in large lecture courses
Peer Review
• “Peer Assessment Between Students in
Colleges and Universities” Review of
Educational Research, Fall 1998, 68, pp
249-276
• Little use of computers except to submit
work
Peer Review
• “Student Peer Assessment in Higher Education: A
Meta-Analysis Comparing Peer and Teacher Marks”
N. Falchikov and J. Goldfinch, Review of Educational
Research, Fall 2000, 70, pp 287-322
– Peer assessments resembled teacher assessments most closely
“when global judgements based on well understood criteria
were used.”
– Advanced students not better assessors than beginner
students.
Calibrated Peer Review (CPR)
• Both writing and reviewing improve
students’ understanding of the topic
– involves writing (reflective thinking)
– involves reviewing (critical thinking)
Student’s Interaction with CPR
Students
– carry out a guided study of source material
– write an essay (~300 words) about what they have
studied.
– are trained as reviewers of this topic by studying a set of
“Calibration” essays and evaluating them on specific
questions
– anonymously review three of their peers’ essays and
provide written feedback to them
– apply their evaluation skills to self-review their own essays
Instructor Responsibility
• Selects the topics from the library of
assignments
• Sets grading criteria and due dates for
submission of essays and reviews
• Monitors student progress and performance
(Participates in the assignment as a student)
Assignments in Personal and Public Libraries
C
he
m
100
125
150
175
200
225
250
275
25
50
75
0
En istr
g y
C
om
Bi p
Te olo
ch gy
W
r
Ec iting
on
/M
an
M
at
G h
eo
lo
gy
Et
h
So ics
ci
ol
og
y
Ps
y
Pr M ch
op ed
os ic
al ine
W
Complete
rit
in
g
Sc
iE
d
En
H
ig g L
Discipline
h i
Incomplete
Sc t
ho
Ph ol
ys
ic
s
C
As P
tro R
no
C m
CPR Assignments (July 2003)
om y
p
Sc
i
La
Po w
li
Sc
Bi
oc i
En h
gi em
ne
er
in
N g
ur
At sing
m
os
Incomplete Assignments = 353
Complete Assignments = 889
Sc
O i
th
er
s
So What? Do CPR Assignments
Make a Difference?
• Is content knowledge improved?
• Are evaluation skills developed?
• Do students learn to collaborate?
• What do students think about CPR?
Evaluation of CPR
Economics - CSUN
• Comparison of intact classes
• Ten case-study assignments
• Three sections (same instructor) of
Economics 200; 2 CPR based, 1 traditional
with essays but no evaluation component
• Repeated over two semesters
Average score by quint ile on midt erm
( Economics 2 0 0 , Fall 1 9 9 9 )
80
70
60
Average score on midterm
50
CPR
40
no CPR
30
20
10
0
q1 q2 q3 q4 q5
quint ile
Evaluation of CPR
Biology - CSUF
• One class, one instructor
• Time-series design
• Ten topics, 5 taught through CPR, alternately with
5 taught through lecture and simulation
• Nancy J. Pelaez “Problem-based Writing with
Peer Review Improves Academic Performance in
Physiology” Advan Physiol Educ, Sep 2002; 26:
174 - 184.
100%
.
y = 0 0007x +0. 1068
2
80% .
R = 0 4305
60%
.
y = 0 001x - 0.208
40% R2 = 0 5049
.
CPR
con cep ts
Le cture
Concept s
20% Linea r (CPR
con cep ts)
Linea r (Le cture
Concept s)
0%
500 600 700 800 900 1000
Total points for BIO310
Mean multiple choice test scores were signi ficantly better for topics taught
using CPR compared with those taught using traditional instruction.
72 + 2% versus 65 + 2%, P 6 Sp01 6
Peer Review Feedback
Rating: 6; Explanation: Small gramma tical errors: -
"rules can be bend if necessary" - "the rights ethics" -
"Duty ethics focuses on that each person has a duty..." -
"ma ny similarity exist" - "When such incident arises" - "I
worked night shift " Fragme nts: "While in rights ethics,
each person has the rights to life. " note: Only John Stuart
Mill is not recognized. Also, no introduction at all!
RATING: I gave this writer a 6 because I feel that the
theories were clearly described, despite the vast amount of
gramm atical errors. There was no introduction to this
memo rendum at all. This is not an English class so I can't
mark down that much for these errors; however, I feel it
imperative for engineers to be able to clearly write a
sentence in correct English. How else are they to write
proposals, reviews and test reports, etc. in the corporate
world, where English is the international language of
business. By the way, very good explanation of the
dilemma .
Introductory Organic Chemistry
• Research University
– lecture three times/week
– no discussion
– separate lab course
– most students “premed”
• one CPR assignment: Study of Cycloalkanes
– uses CHIME interactive tutorial to study ring strain
• given as extra credit assignment at end of quarter
• feedback e-mail to instructor required for credit
Introductory Organic Chemistry
Student Buy-In
• 219 of 372 students did the assignment and
submitted comments!
Student Perceptions
Student Comments
Technical Issues
• “The chime plugin was a little problem because I too use
netscape 6.1 so I ended up using Internet Explorer 6.0. It
handled the plugin well and did not give me any problems. The
web site performed decently with a 56K modem connection,
except the chime illustrations took about 30sec to 45sec to load
but acceptable. Now that the calibrations are available the
calibration screens seem very user friendly but I would tell
everyone that it is best to use a screen resolution of at least
800x600 because there is so much to info to display. The
questions take half the screen and the essay takes up the other
half.”
Source Material/Reviewing
• “I thought that the CPR was really interesting. I liked the fact
that they made us look at 3D models before we could answer the
questions because it gave us a good idea of the concept involved
in the question. I think that the process we had to take before
writing the essay was great because it took us step by step to the
answer, allowing us to thoroughly comprehend the strain (angle
and torsional) in cycloalkanes in comparison to their alkane
counterparts and to other larger or smaller cycloalkanes…. I also
like that we will get to grade other students’ essays and later see
if our own essays were of good quality. I believe that it will let us
know if we really understand the part that strain plays in
cycloalkanes.”
Workload, Course Value
• “This program is similar to something that my girlfriend is
doing in her Christian theology class at BIOLA. She writes
a 400-word response to a "poll" question based on the 400-
500 page reading assignment that week, every week. Then
she reviews the work of three others (but not her own),
followed by an in class discussion once a week. Of course,
an in-class discussion is not appropriate for a chemistry
course and the material covered requires more work than a
simple read and response. This is why I feel that the CPR
program would be more beneficial if it were assigned more
than once a quarter. Something more like one assignment
every 2-3 weeks. I really feel that this would improve the
overall information retention of the class.
Writing
• By writing this assignment in a mini-essay form I
was able to come at my own conclusions and
express the material in my words. I believe that
this was very helpful because in order to
understand the material it is extremely useful to be
able to describe and put the material one learned
into his/her own words
Workload, Course Value
• I enjoyed the CPR assignment. I liked having
another format to think about organic chemistry.
If you do this again, I think optical rotation would
be a great subject. I thought the writing
assignment was a good length--not too much to
read six examples of writing
Workload, Value, Reviewing
• I thought the idea of CPR was a very good idea because it
helped expanding students' knowledge and developing
insight into the materials we had studied. However, I didn't
really enjoy this project. I didn't like it because there was
too much research involved and questions were very hard to
answer. This project itself was time consuming as well. It
took me a long time to finish the practice before I moved
onto the actual project. Another time-consuming part was
the peer reviewing. It took me about an average of 10
minutes each to review one paper. It required a lot of
concentration as well
Course Value
• I thought CPR to be good learning tool. The
project made the participants review the course
material. As a result a better understanding of the
material is attained…. It’s like having two lectures
instead of one. Making an environment where
students have to think is a good way to make the
material understandable.
Reviewing, Course Value
• Truthfully, I enjoyed CPR. Writing the essay was a
great way to review the specifics of cyclohxane
strain, but when I reviewed the peer essays I think
I learned most. I was able to see what
strong points my essay had, and more importantly
where I was lacking. I think CPR can be a very
useful tool for students to use if they take it
seriously and take the time to explore the website.
Also, it is another chance for points, which is
always good. By the way, thanks for the extra
credit.
Timing, Workload
• this project was sort of like high school work. it
was only assigned to keep me busy. first of all i
don't have a computer so i had to go out of my
way to use one. we studied cyclohexanes for
the first midterm. it would've been a lot better if it
was material from the current chapters.
Word Count, Technical Issues
• I just wanted to give you my opinion of cpr. I felt
like it was to much of a hassle. It was hard to log
onto and frustrating to deal with. When I logged
on and began writing my text the HTML
tutor wasn't accessible so I didn't know the
editing commands. I know that we were supposed
to be concise, but I felt that 230-390 words
just wasn't enough to truly answer the questions
and write a good essay. Anyways, I didn't think
cpr was really worth it because it was to much of
a hassle.
• .just submitted my CPR essay. All in all it isn't that
bad. It's nice and short so that I ended up editting a
lot of what I wrote so that I reached the maximum
length and that's better than trying to think of crud to
insert here and there to make a nice long essay. After
the hurdle of trying to sit down and writing something
about ochem, the experience itself is relatively brief.
It would have been nice to know who our target
audience is. I know that other peers are reading this,
but for an example, are we supposed to clearly
explain what eclipsing bonds means? Initially I did
what it is, but there isn't enough word allowance for
it so I took it out. The bad thing I don't like is that
the topic is a bit late. I wouldn't mind covering a
subject that is supposed to be covered in the next few
weeks but I think writing a paper about something
Foreign Students
• The calibrated peer review helped me to
review many knowledges about the
conformation of cyclohexane. I learned a lot
from it
Technical Difficulties and
Attitude to CPR
Positive Negative
Comment Comment
Mentioned problems
with CHIME 59% 41%
(&/or UCR and CPR login)
No technical difficulties 82% 18%
mentioned
http://cpr.molsci.ucla.edu
• Integrates telecommunications and
technology into instructional processes so
that students learn to
explore ideas independently
write about chemistry
collaborate
work to a mastery level
take responsibility for their own learning
Institutions with CPR Accounts
400
Jul-03
350
300 Jan-03
250
Number
Sep-02
200
150
100 Aug-01
Jul-00
50
0 Sep-97 Sep-98
Fe b-97 Jun-98 Oct-99 M ar-01 Jul-02 De c-03
Date
Courses Using CPR
1600
Jul-03
1400
Jan-03
1200
1000
Number
Jul-02
800
600
Aug-01
400
200
Jul-00
0 Sep-97 Dec-98
Fe b-97 Jun-98 Oct-99 M ar-01 Jul-02 De c-03
Date
Cumulative Student Accounts
60000
50000 Jul-03
Jan-03
Sep-02
40000
Number
Jul-02
30000
20000
Aug-01
10000
Jul-00
Dec-98
0 Sep-97
Fe b-97 Jun-98 Oct-99 M ar-01 Jul-02 De c-03
Date
Acknowledgements
Funding sources
• National Science Foundation
• Howard Hughes Medical Institute
• University of California, Office of the
President
• University of California, Los Angeles
Acknowledgements
Colleagues
• Orville L. Chapman
• Tim Su
• Krista Motschiedler Brand
• Stephen Schimpf (Michael Fiore)
• authors
• students
Thank you