Assessment Questions Equity Answers

Document Sample
Assessment Questions Equity Answers Powered By Docstoc

UCLA's Center for the Study of Evaluation &
The National Center for Research on Evaluation, Standards, and Student Testing

                                                                                               Winter 1994

                           Assessment Questions: Equity Answers
                           Proceedings of the 1993 CRESST Conference

                                     Robert Rothman, CRESST/UCLA

Z      eroing in on one of the most
       critical issues in the shift to
new forms of assessment, more
                                         time, he said, the report of the
                                         National Council on Education
                                         Standards and Testing and the
                                                                                  Dean Theodore R. Mitchell of
                                                                                the UCLA Graduate School of
                                                                                Education said the current de-
than 200 researchers, policy mak-        Clinton Administration’s Goals         bate represents a “unique histori-
ers, and teachers gathered at the        2000 legislation have put equity       cal moment.” “For the first time,”
UCLA campus September 12-14,             at the top of the agenda in the        he said, “Americans are discuss-
1993 to discuss questions sur-           federal and state governments.         ing both equity and excellence,
rounding equity.                            “States and the national effort     taking into account both inputs
   Meeting at the 1993 CRESST            are focusing heavily on trying to      and outcomes. We are at the
conference, entitled “Assessment         establish ambitious content stan-      threshold of a breakthrough,”
Questions: Equity Answers,” par-         dards,” Linn said. “And assess-        Dean Mitchell said.
ticipants outlined many of the           ment has been central to all the         But CRESST Associate Direc-
concerns associated with the topic       work put forward in curriculum         tor Joan Herman cautioned that
and offered some possible solu-          reform. This has led to demands        researchers do not yet have all the
tions.                                   for high standards of student per-
                                         formance assessed by new assess-
CRESST Criteria of Fairness              ments congruent with the con-                Eight New
  CRESST Co-director Robert              tent standards.”
Linn said in opening remarks that           “If you put the two together,”
equity is at the center of debates       Linn said, “that leads immedi-
over standards and assessments.          ately to concerns about equity:         See pages 13-16 for de-
Fairness is one of the most critical     what it means to give students a        scriptions of eight new
of the criteria developed by             fair shot [at meeting the stan-         CRESST/CSE technical
CRESST to evaluate new assess-           dards], especially if there are high    reports.
ments, he noted. At the same             stakes attached.”

                        Assessment Questions: Equity Answers
                         Proceedings of the 1993 CRESST Conference

answers to enable us to walk          uity, resources must be distrib-          inequities will remain regardless
through that door. “Most don’t        uted sufficiently and opportuni-          of what happens to assessments.
agree on what the central ques-       ties for demonstrating compe-                Lauren B. Resnick, CRESST/
tions are,” she said.                 tency must be diverse. “You may           LRDC, University of Pittsburgh,
                                      expose all persons to the same            argued that assessment can play a
Defining Equity                       standard,” Gordon said, “but if           major role in alleviating inequi-
  While focusing their attention      the manner in which the standard          ties. Like Gordon, Resnick said
on the role of assessment, many       is presented isn’t one that matches       that the real issue is learning, and
agreed that the issue of equity       the characteristics of each per-          she said that changing assessment
involves the education system as      son, one cannot assert that it has        is vital to creating opportunity to
a whole. “If we can adequately        been presented equitably.”                learn for all Americans.
teach all youngsters, we don’t           Gordon also laid down a chal-
need to be as concerned with          lenge to the assessment commu-
equity,” said Edmund W. Gor-          nity and outlined three ways to
                                                                                  “…equity…is the
don, CRESST/City University           make assessment more equitable.
                                                                                  right to achieve at lev-
of New York and Yale University.         First, make better use of the
                                                                                  els sufficient to par-
  “However,” Gordon added,            information provided from assess-
                                                                                  ticipate productively,
“that fact does not let those in-     ment to allocate resources equita-
                                                                                  and in a rewarding
volved with assessments off the       bly. Second, develop new instru-
                                                                                  way, economically
hook. Assessments themselves can      ments and procedures to tap stu-
                                                                                  and civically.”
be made more equitable,” he said,     dents’ affective traits, not just their
“and assessments can help make        cognitive skills. And third, con-           Resnick argued that we are
inputs more equitable.”               duct research and development             heading down a revolutionary
                                      work to build on what is known            path, at the end of which all stu-
                                      about pluralism. Although port-           dents will have a right to achieve.
    To   attain equity…               folios appear promising as a way          “What we mean by equity,” she
    opportunities for dem-            of assessing students’ abilities          said, “is the right to achieve at
    onstrating compe-                 through diverse ways, Gordon              levels sufficient to participate pro-
    tency must be diverse.            warned against latching onto port-        ductively, and in a rewarding way,
                                      folios as the solution to every           economically and civically.”
  But while agreeing on equity as     problem.                                    Reaching such equity, she said,
a goal, researchers offered differ-      Gordon also cautioned that the         will require defining the level of
ing perspectives on how to define     problem of inequity is a problem          achievement all children must at-
the concept and how it might be       of the larger society outside             tain, a process that is now under
achieved. To Gordon, equity is        school. As long as society contin-        way through the development of
not equivalent to equality, but       ues to reward winners and to              national standards. In addition, it
rather, sufficiency. To attain eq-    screen out people “not like us,”          will require holding ourselves re-
                                                                                sponsible for providing the op-

                          Assessment Questions: Equity Answers
                          Proceedings of the 1993 CRESST Conference

portunity for all students to           of instruction for all students,      views, which examine test con-
achieve to the desired level. Al-       particularly low-income, minor-       tent with an eye toward eliminat-
though the right to achieve could       ity students. What is needed are      ing offensive or stereotyping ma-
eventually become a legal stan-         opportunity-to-learn standards        terial; and statistical analyses—
dard that would enable young            that would define a fair share of     are inadequate for use with per-
people to demand the opportuni-         resources for schools. But that in    formance assessments.
ties to achieve, in the meantime        itself may not be enough, Oakes
this right could serve as a moral       added, because it is “astonishing
obligation for society.                 how quickly good data disappear.”
                                                                                Math assessments that
   Lorraine McDonnell, CRESST/          Instead, she said, schools should
                                                                                demand substantial
University of California, Santa         make the data-collection process a
                                                                                linguistic ability may
Barbara, argued that attaining          part of the improvement process.
                                                                                be unfair to those lack-
equity is a political, not a moral,        “My best guess at this point,”
                                                                                ing in that skill.
challenge. She noted that Ameri-        Oakes said, “is that whatever pro-
cans have long supported what           cess we engage in for the regular     Linn proposed additional factors
she called procedural equity, or a      business of collecting informa-       that must be considered to deter-
process that ensures that every-        tion and feeding it back into our     mine if new assessments are fair.
one has access to valued goods.         systems probably better look          These new factors include:
But substantive equity, or equal        pretty much like the complex kind
results, has never enjoyed public       of teaching and learning we’re          • The intent of the measure,
support, in part because it de-         hoping for in schools. Ideally, the       or the extent to which as-
mands redistribution of resources.      process of assessing the quality of       sessments measure ancillary
Achieving that type of equity,          students’ opportunities will be-          skills that might provide an
McDonnell argued, demands ap-           come indistinguishable from the           advantage to a particular
pealing to voters’ self-interest for    very effort to create and improve         group. As an example, Linn
a better society in which young         those opportunities.”                     noted that math assess-
people are better educated. “One                                                  ments that demand a sub-
hopes for altruism, but it’s hard to    Evaluating the Fairness of As-            stantial amount of linguis-
build a political majority that way,”   sessments                                 tic ability could prove a dis-
she said.                                 Looking specifically at the issue       advantage to those with
   Similarly, Jeannie Oakes of the      of fairness and assessment,               math skills who lack read-
UCLA Graduate School of Edu-            CRESST Co-director Robert                 ing and writing ability.
cation also doubted that moral          Linn noted that the traditional
suasion would be sufficient to          methods of evaluating whether           • Comparability, or the ex-
ensure equity. Many schools, she        tests are fair—impact analyses,           tent to which the assess-
said, lack the human and material       which measure differences in              ments allow variability in
resources needed to create the          group performances; sensitivity re-       format or scoring.
conditions to provide high levels

                          Assessment Questions: Equity Answers
                          Proceedings of the 1993 CRESST Conference

    • Choice of task. By allowing     may balk at assessments that de-       ics classes like the open-ended
      students to choose their        mand that they write responses,        ones on the assessment.
      tasks, such as reading a        particularly in subjects other than
      book, we may be providing       English. He pointed out that in
      an advantage to those who       the 1990 National Assessment of
                                                                               P erformance-based
      are already familiar with the   Educational Progress, many stu-
                                                                               assessments use a small
      books they choose.              dents simply skipped over open-
                                                                               number of tasks that
                                      ended items, and there were eth-
                                                                               may favor one group
    • Delivery standards. To as-      nic group differences in the omit
                                                                               over another.
      sure equity, one must look      rates.
      not only at assessments, but       In addition to the variations in       H.D. Hoover of the University
      the provision of the instruc-   student experiences, differences       of Iowa, however, said it is un-
      tional experiences to stu-      in teachers’ experiences can also      clear whether differences in group
      dents.                          affect students’ performance,          performance on performance as-
                                      Burstein said. In reform environ-      sessments reflect differences in
   Leigh Burstein, CRESST/            ments, urban teachers tend to have     curricular experiences or test bias.
UCLA, also said that examining        fewer chances to participate in        He said that, unlike multiple-
students’ opportunity to learn is     developing and scoring new as-         choice tests, which can include a
essential to evaluate the fairness    sessments than do teachers from        wide range of questions, perfor-
of new assessments. “You can’t        suburban and rural areas, since it     mance-based assessments use a
measure achievement without           costs districts to send teachers to    small number of tasks that may
knowing the instructional condi-      such sessions.                         favor one group over another. A
tions under which achievement            Burstein described two studies      certain reading passage and its
occurs,” he said. But he cautioned    designed to examine students’ cur-     corresponding questions may ap-
that studies of opportunities to      ricular experiences. The first looks   peal more to a student who has an
learn have thus far only taken        at classroom “artifacts”—text-         inherent interest in the subject
place in low-stakes environments,     books, logs of daily activities,       matter of that passage.
not in situations where schools       homework assignments, in-class            “For fairness, you need a diver-
were held accountable for provid-     quizzes, and major assignments—        sity of content and contexts,” said
ing such opportunities.               and compares the findings with         Hoover. “Those of us who build
   Burstein pointed out two ways      surveys of teachers that attempt       standardized tests—that’s what we
in which students’ opportunities      to get at students’ learning op-       do. We ask lots of questions, and
could have an effect on their per-    portunities. The second, con-          balance questions.”
formance on alternative forms of      ducted as part of the California
assessments. First is the students’   Learning Assessment System, asks       Data From Large-Scale Assess-
own experiences. Students accus-      students and teachers if they have     ment Programs
tomed to multiple-choice tests        done problems in their mathemat-          Whether because of differences
                                                                             in opportunity to learn or because

                       Assessment Questions: Equity Answers
                        Proceedings of the 1993 CRESST Conference

of the assessments themselves,       erty.” He concluded that improv-
the gaps between advantaged and      ing the level of student perfor-
                                                                             We    have seen teach-
disadvantaged students are not       mance through the use of portfo-
                                                                             ers break an integrated
closing as schools shift to new      lios is a “hard, arduous task” that
                                                                             unit into bits, and
forms of assessment. In fact, re-    will require money and an infra-
                                                                             teach toward mastery
searchers who have studied new       structure to enable teachers to
                                                                             of the bits.
large-scale assessment programs      adapt to changing curricular and
have found that the gaps may be      instructional demands.                   “It is not yet clear,” said Smith,
widening.                               Likewise, CRESST partner           “how ASAP will affect instruc-
   Daniel Koretz, CRESST/            Mary Lee Smith from Arizona           tion, because or in spite of its
RAND, said that some teachers        State University found that the       high-stakes function, although
in Vermont agree that that state’s   first year of implementation of       there are already foreshadowings.
pioneering portfolio assessment      Arizona’s performance-based as-       It is predictable that districts with
program improved education for       sessment program failed to ad-        adequate resources will do what is
traditionally low-performing stu-    dress the disparities between low-    necessary to raise low scores.
dents. However, he noted, such       performing and high-performing        Whether they will take the high
improvements were erratic.           schools. Although teachers had        road—undertaking the time-con-
                                     believed that the program would       suming and expensive professional
                                     be a low-stakes exercise that would   and curriculum development
  The use of portfolio               enable them to improve instruc-       work necessary to teach toward
  assessments requires               tion, in part through developing      ambitious standards and a think-
  teachers to adapt to               and scoring the assessment, in        ing curriculum—or the low
  everchanging curricu-              practice the Arizona Student As-      road—finding the tricks to inflate
  lar and instructional              sessment Program (ASAP),              scores—remains to be seen. At
  demands.                           proved quite different. The scor-     this point, we have already docu-
                                     ing was done by a commercial          mented such activities as teachers
   “We see sign after sign after     publisher, not teachers, and the      focusing pupils’ attention on those
sign that teachers vary enor-        stakes went up when newspapers        parts of the assessment that will
mously in response to perfor-        ranked school districts according     be scored. We have also seen
mance assessment,” Koretz said.      to test scores. The state, more-      instances of what we call dis-inte-
“They vary in how quickly they       over, provided little professional    grating, in which teachers who
adapt to demands, and to what is     development to boost the capaci-      lack a thorough understanding of
expected of them.”                   ties of schools.                      constructivist teaching take what
   “Moreover,” Koretz said, “the        Whether the ASAP program will      was designed to be an integrated
program itself of course has done    eventually narrow the gaps in         unit, break it into bits, and teach
nothing to alleviate the condi-      school performance is difficult to    toward mastery of the bits.”
tions that have plagued low-per-     predict.                                 As with the large-scale pro-
forming students, such as pov-

                        Assessment Questions: Equity Answers
                        Proceedings of the 1993 CRESST Conference

grams, Lorrie A. Shepard,            Harvey now knows her students          them, since policy makers should
CRESST/University of Colorado        better than she ever did and she       ideally weigh the costs and ben-
at Boulder, said that in classroom   has evidence of their progress that    efits of a proposed reform. But
assessments, gains in students’      she can show to parents. She also      quantifying the potential benefits
ability to perform well on perfor-   cited the case of one of her stu-      has proved elusive, noted David
mance tasks come slowly and un-      dents, Jeff.                           Monk of Cornell University, and
evenly. Drawing from her                “I had given him a running          as a result, most of the discussion
CRESST research involving third      record in October,” said Harvey;       of costs has focused solely on ex-
graders in three Colorado schools,   “he read about a page and a half       penditures.
Shepard cited a problem that asked   in 10 minutes. At the end of that         “You can’t talk about cost with-
the students to complete a table     time, I couldn’t bear to watch his     out dealing with the benefit side
to determine how many pitchers       face anymore, it was so painful.       of the equation,” Monk said.
(of 4 cups each) it would take to    So we stopped…. [He read] the          “That’s a problem with perfor-
have enough cups for all the stu-    exact same page in January. And        mance assessment. The simple fact
dents in the class. There were no    he blew right through it. He did       is, we don’t know very much about
differences in performance be-       very well. Again, he was third         what performance assessment pro-
tween Anglo and Latino students,     grade, it was a preprimer, but         duces, or what kind of levels of
but initially very few students in   that was progress. And I showed        resources are required for this to
either group could even attempt      it to him and said, ‘Look what         take place. In the absence of that
the problem. In the second year      you’ve done.’ And he beamed            kind of knowledge, you’re in a bit
of the program, the majority of      and said, ‘I’m getting better,         of a dilemma trying to carry out a
students could complete the table    aren’t I?’ That’s where it’s at, for   cost analysis that’s more than an
and wrote more to explain their      me.”                                   expenditure analysis.”
answers than they did the first                                                Monk said that an analysis he
year, Shepard explained. But                                                conducted for the New Standards
many still had a long way to go.
                                       Ideally, policy makers               Project produced a range of cost
“Anyone who thinks this can be
                                       should weigh the costs               estimates for performance assess-
put in place in a year or two is
                                       and benefits of a pro-               ments, depending on the extent
probably crazy,” she said.
                                       posed reform before                  to which the assessment is added
   But Jennifer Harvey, a teacher
                                       implementation.                      on to existing programs and the
at Cherry Drive Elementary                                                  extent to which every student is
School in Thornton, Colorado,        Costs of Performance Assess-           tested. In the worst case scenario,
who is part of the CRESST/Uni-       ment                                   in which the assessment was an
versity of Colorado at Boulder         The benefits of performance          addition to existing programs and
study, said even small gains are     assessment should also have a bear-    every student in three grades was
valuable. By using a variety of      ing on policy makers’ decisions        tested, the assessment cost $97.4
alternative assessment methods,      about whether to implement             million or $29 per pupil for a
                                                                            large state (Texas), $27.9 million

                         Assessment Questions: Equity Answers
                         Proceedings of the 1993 CRESST Conference

or $28 per pupil for a medium-         whether we want to compare in-       A. Romberg of the University of
sized state (Virginia), and $3.5       dividual students or school dis-     Wisconsin, Madison. Romberg
million or $37 per pupil for a         tricts, how many tasks are needed    said that the NCTM defines eq-
small state (Vermont). In all cases,   to provide a reliable estimate of    uity as providing all students the
a little less than a fourth of the     student abilities, how much train-   opportunity to demonstrate their
expenditures went toward staff         ing is needed for teachers, and      mathematical power, and he noted
development, and the total ex-         what are the competing claims for    that current tests do not match
penditures amounted to between         resources.                           the council’s goals.
0.6 percent and 0.7 percent of            As in other issues of public
each state’s education budget.         policy, Picus said, deciding
   Fritz Mulhauser of the U.S.         whether to invest in performance
                                                                              “ …[Native Ameri-
General Accounting Office              assessment involves tradeoffs:
                                                                              cans] also value pa-
(GAO), which conducted an              “How much of this do you want
                                                                              tience, whereas tests
analysis of the cost of testing for    to do versus how much can you
                                                                              demand rapid re-
Congress, said that the GAO study      afford to do in terms of time and
                                                                              sponses and immedi-
came up with an estimate for a         resources?”
                                                                              ate decisions.”
proposed national test that was
far lower than previous esti-          Equity and Assessment Design            Other researchers suggested
mates—from $42 million for a             In looking ahead to new pro-       that new assessments must take
multiple-choice test and little        grams, researchers also discussed    into account the needs of diverse
added time to $209 million for a       possible ways to ensure equity in    students in order to be equitable.
test including short performance-      assessment.                          Michael Pavel of the UCLA
based questions and 30 minutes                                              Graduate School of Education said
of added testing time. But                                                  that Native American students are
Mulhauser noted that the true
                                         Equity   is one of the             ill-served by traditional tests. Na-
cost depends on the purpose for
                                         key criteria by which              tive Americans value placidity, a
such a test, and he urged Con-
                                         educators can judge                characteristic that may result in
gress to be clear about the pur-
                                         tests.                             their being viewed as slow or back-
pose before determining the pro-                                            ward, he said. Similarly, patience
posed cost.                               One step toward that end is       is valued, whereas tests demand
   Similarly, Lawrence Picus,          being taken by the National Coun-    rapid responses and immediate
CRESST/University of Southern          cil of Teachers of Mathematics       decisions.
California, also raised a number       (NCTM). In that group’s Assess-         He added that assessment
of questions that need to be an-       ment Standards for School Math-      should be better adapted to Na-
swered before determining              ematics, equity is one of the key    tive American people who need
whether the benefits of perfor-        criteria by which educators can      to improve on their academic per-
mance assessment exceed the            judge tests, according to Thomas     formance and make teachers more
costs. Among these questions are:

                        Assessment Questions: Equity Answers
                        Proceedings of the 1993 CRESST Conference

aware of how they can address         programs or else exempt them          Establishing the link between
these academic needs. Therefore,      from tests altogether. “I am on       classroom assessments, such as
to assist Native Americans, assess-   the side of trying to develop a       portfolios, and accountability is
ment results should be used to        middle ground where there is ac-      one way to further the use of
guide student learning, refine cur-   countability for student learning,”   multiple assessments.
riculum, and improve instruction.     she said. “If you exclude them
   Charlene Rivera of George          completely, they are not consid-      Portfolios
Washington University said as-        ered in the policy debate. But           Likewise, portfolios can also
sessment reform must also con-        how to assess them? There is no       provide diverse opportunities for
sider students for whom English       definitive answer. The best prac-     students to demonstrate their skills
is not the native language. In ad-    tices will result from experimen-     and knowledge. A school project
dition to learning content knowl-     tation.”                              in Pasadena, California, that inte-
edge and skills, she pointed out,                                           grates language arts and visual
“English language learners”                                                 and performing arts instruction
(ELLS) are also learning a second
                                        Experiments already                 by asking students to create works
language, something native En-
                                        underway suggest pos-               of art and to reflect on their own
glish speakers do not have to do.
                                        sible solutions for tra-            and other works has shown dra-
   “To date, reform efforts have
                                        ditionally underserved              matic results among a group of
not considered needs of ELLS
                                        groups.                             formerly low-achieving children,
students,” Rivera said. “The guid-                                          according to Pam Aschbacher,
ing assumptions have been: What          Some experiments already un-       CRESST/UCLA.
will work for monolingual stu-        der way suggest possible solu-           But Aschbacher also pointed
dents will also work for ELLS         tions for English language learn-     out that implementing portfolios
students. Once ELLS students          ers and other traditionally           alone may not improve educa-
learn a little English, new im-       underserved groups. One such          tional opportunities for all youths
proved assessment systems will fit    experiment is a primary-grades        and the portfolios themselves can
them too. However, experience         assessment currently being devel-     demonstrate this. In a separate
doesn’t support this assumption.      oped by a consortium of six states    project, Aschbacher examined
While ELLS students can and do        under the auspices of the Council     portfolios to open a window on
learn to high standards, assessing    of Chief State School Officers.       classroom instruction and found
their achievements in the same        Jackie Cheong of the University       that teachers were not always
way as their monolingual peers        of California, Davis, said one of     meeting reform objectives. Un-
will greatly underestimate their      the guiding principles of that ef-    der the program, known as
accomplishments and potential.”       fort is to provide multiple assess-   Humanitas, students were ex-
   Rivera added that schools cur-     ment strategies to tap a range of     pected to be personally invested
rently include English language       what students from diverse back-      in the work that they’re doing in
learners in inappropriate testing     grounds know and are able to do.      class, to make interdisciplinary
                                                                            connections, to engage in com-

                          Assessment Questions: Equity Answers
                          Proceedings of the 1993 CRESST Conference

plex reasoning, and to evaluate         ing two R&D projects on class-         member, provides an opportu-
themselves and show some growth         room assessment, one focused on        nity to do just that.
over time, among other goals.           writing and the other on math-            But Noreen Webb, CRESST/
   The portfolios showed a direct       ematics, Gearhart said, “I have        UCLA, noted that group inter-
relation between the kind of as-        found that elementary teachers         actions are complex and that sim-
signments teachers gave their stu-      do not typically develop in their      ply asking students to cooperate
dents and what students learned.        understandings of student work         may not result in everyone’s ac-
For example, she found that stu-        without a substantive focus on         quiring shared knowledge and
dents’ work showed more higher          subject matter. New assessments        understandings. Factors such as
order thinking skills and interdis-     require teachers to make informed      the composition of the group
ciplinary connections when their        judgments,” she said, “but teach-      and the incentives they have for
class assignments required them         ers cannot judge work that they        working together can influence
to make those connections and to        do not understand. In our              whether group interactions are
use complex thinking. “That’s a         projects, the goal is to create pro-   functional or dysfunctional,
strong message,” said Aschbacher,       totypes of assessment practices        Webb said.
“to these teachers that says: ‘Look,    integrated with reform curricula.         Moreover, she added, relying
when you use assignments that           We can’t simply exhort teachers        on group assessments as a mea-
do not explicitly ask kids to do the    to collect student work and assess     sure of student abilities may ad-
kind of thinking this program calls     it,” said Gearhart. “We need to        versely affect equity. Depending
for, they don’t do it. You get          give them specific models built        on the way groups work together,
what you ask for.’”                     upon specific curriculum.”             she said, group performances may
   Similarly, Catherine Smith of the                                           not be a valid measure of the
Michigan Department of Educa-           Group Assessment                       performance of individual mem-
tion said that a program in that           Similar caution flags went up       bers, and, as a result, group per-
state to use portfolios to gauge        over another potential avenue for      formances may mask individual
students’ workforce readiness also      ensuring equity: the use of group      deficiencies. If that is the case,
raised equity concerns. “Students       work. Wayne Neuberger of the           Webb warned, low performers
in inner-city and rural areas were      Oregon Department of Educa-            may miss out on needed instruc-
less likely than those in suburbs to    tion suggested that allowing stu-      tional help.
be aware that they could use evi-       dents to work in groups in ad-
dence from part-time jobs and other     vance of an assessment could level     Equity and Interpreting As-
afterschool activities to demonstrate   the playing field by enabling stu-     sessment Results
readiness skills,” she said.            dents to compare notes and en-           In addition to the design of
   Maryl Gearhart, CRESST/              sure that they all had the same        new assessments, the way results
UCLA, also reported that imple-         background to prepare for the          are interpreted also has a bearing
menting portfolio assessment            assessment. The New Standards          on questions of equity. David
alone may not have much impact          Project, of which Oregon is a          Bayless of Bayless and Associates
on classroom practice. Describ-

                         Assessment Questions: Equity Answers
                         Proceedings of the 1993 CRESST Conference

said that the traditional way of       males on certain subscales, such        gram to identify exemplary as-
analyzing assessment results is to     as measurement. Muthén is also          sessment programs. One of the
look at subgroups, such as gen-        analyzing the results of the Lon-       criteria for being considered ex-
der, socioeconomic status, and         gitudinal Study of American             emplary should be that the assess-
ethnicity. But those analyses can      Youth to detect the effects of track-   ments have demonstrated mak-
provide little guidance for im-        ing and coursetaking on student         ing a contribution to improving
provement, because they cannot         performance over time.                  the success of poor and minority
be changed. “It’s difficult to           Looking at a separate way of          student populations.
change [a student’s] gender,” he       examining possible sources of bias,
said. Rather than select demo-         Jamal Abedi of CRESST/UCLA              Equity and the Research Agenda
graphic subgroups that are easy to     said that different methods of             Although some answers to the
measure, Bayless argued, analysts      analyzing interrater reliability may    equity questions surrounding new
should examine assessment results      yield different estimates. As a re-     forms of assessment are begin-
according to factors that can be       sult, he suggested using different      ning to emerge, the research
influenced by intervention, such       applicable approaches. “I cannot        agenda remains long. Pauline
as opportunity to learn.               name a single best approach to          Brooks, CRESST/UCLA, out-
                                       establishing interrater reliability,”   lined a host of questions in light
                                       Abedi said. “Depending on the           of the unequal representation of
  Examining single test                form of data, compute as many           gender, socioeconomic status, and
  scores may mask im-                  applicable approaches as you can,       diverse cultures throughout the
  portant differences in               and then draw your conclusions          history of testing and assessment.
  subgroups’ perfor-                   based on those outcomes.”               Included on her list were:
  mance.                                 Michael T. Nettles of the Uni-
                                       versity of Michigan proposed an           • the effects of the teacher’s
   Bengt Muthén, CRESST/               additional step. After citing eth-          cultural expectations on
UCLA, said that examining single       nic group differences in perfor-            his/her judgments of cul-
test scores may mask important         mance on various assessments, he            turally diverse student per-
differences in subgroups’ perfor-      said that we should look at assess-         formances;
mance. In one study, Muthén said       ments that show good results with
he is using multivariate analysis to   traditionally low-performing stu-         • the extent to which the new
gain a broader picture of student      dents.                                      assessments provide vary-
performance on the National As-          Under the auspices of the Ford            ing opportunities for stu-
sessment of Educational Progress.      Foundation, Nettles said that               dents of different cultural/
Preliminary findings suggest that      during the next year he expects to          SES/language backgrounds
while female and male students         develop a second symposium on               to demonstrate their
perform about equally well over-       equity and educational testing and          knowledge;
all, male students outperform fe-      assessment and develop a pro-
                                                                                 • the correlation between

                         Assessment Questions: Equity Answers
                         Proceedings of the 1993 CRESST Conference

    performance on standard-         • Make sure assessment for          communities and constitu-
    ized achievement tests and         “classroom utility” and as-       encies.
    current performance assess-        sessment for public ac-
    ments—do the racial/cul-           countability are in sync, so    • Develop university-school
    tural and SES gaps narrow,         that teachers have an incen-      partnerships to share knowl-
    remain about the same, or          tive to focus on improved         edge about children’s learn-
    widen with the use of per-         instruction.                      ing and provide ongoing
    formance assessments?                                                staff development for teach-
                                     • Keep expectations high for        ers.
  • the characteristics of teach-      all students to eliminate any
    ers’ interactions with dif-        incentive to relegate low-      • Conduct longitudinal and
    ferent cultural groups of          performing students to a          comparative studies to ex-
    students, in both instruc-         second-class education.           amine the impact of new
    tional and assessment set-                                           assessment strategies on
    tings; and                       Practitioners:                      teachers and students.
                                     • Make the purposes and
  • relative levels of support for     learning outcomes of new        Foundations:
    the new assessments among          assessments clear and make      • Make sure new programs
    communities that vary eco-         sure that large-scale and         are practical and feasible
    nomically and culturally.          classroom-level assessments       so that they can be imple-
                                       are integrated.                   mented in schools and
Working Group Results                                                    yield useful findings.
  In addition, working groups        • Encourage flexibility so that
representing various key constitu-     all students have the time      • Make sure programs are
encies outlined action plans for       and opportunities to suc-         credible to parents, teach-
designing equity-sensitive perfor-     ceed on new assessments.          ers, and funding agencies
mance assessments. The groups                                            and that the programs are
recommended the following:           • Provide choices in assess-        actually implemented.
                                       ment alternatives to chil-
  Policy Makers:                       dren that are appropriate to    • Try to focus on the “hard-
  • Clearly articulate the pur-        their ethnicity, their gen-       to-crack” cases.
    poses of new assessments,          der, the possible existence
    so that the public under-          of handicap, the language       Business Community:
    stands that assessments are        that they use, etc. Incorpo-    • Educators, businesses, and
    aimed at meaningful and            rate diverse groups during        the community at large
    effective accountability, not      the design and piloting pro-      need to find a better way
    an attempt to evade account-       cess so that we engage all        to collaborate to produce

                         Assessment Questions: Equity Answers
                         Proceedings of the 1993 CRESST Conference

     diagnostic assessments that       Parents:                             Urbanski. “They’re just not learn-
     enable our students to ad-        • Involve parents in all as-         ing the same things. Some stu-
     vance their own goals in the        pects of the development           dents are learning math and En-
     classroom and in the work-          of new assessments via ad-         glish and foreign languages and
     place.                              vocates or site-based man-         the arts and physics. And some are
                                         agement, helping to ensure         learning a lot about exclusion and
 • Assessments must also be              that they are fair and better      failure and discrimination and lack
   predictive and show stu-              for all children.                  of opportunities and low expecta-
   dents’ readiness for the                                                 tions of them. But you can’t stop
   workplace as well as their          • Collect data on instruction,       students from learning. You can
   ability to transfer skills from       testing participation, growth      only assist in channeling them to
   school to work.                       in student performance, and        certain ‘kinds’ of learnings. That
                                         teacher grading, to accom-         is why the whole issue of stan-
 • Consider whether all assess-          pany test-score data.              dards and assessment is so ger-
   ments must be administered                                               mane. They are absolutely essen-
   in school or by teachers or         • Conduct research on par-           tial and not only essential, but
   whether other approaches              ent involvement, toxic             indeed the necessary starting point
   are possible, depending on            schools, the role of churches,     for all other reforms. School
   the purpose.                          and communication of test-         change need not be a choice be-
                                         ing information.                   tween making things better or
 Media:                                                                     making things fair. We’re capable
 • Highlight the standards and       In Conclusion                          of doing both. We must make
   content of new assessments          In concluding remarks, Adam          things better and be fair in doing
   as a way of showing what all      Urbanski, the president of the         it.”
   students are expected to          Rochester (NY) Teachers Asso-
   learn.                            ciation, pointed out that reform-      Robert Rothman is a visiting
                                     ing schools so that all students       researcher at CRESST/UCLA.
 • Report opportunity to learn       learn at high levels is a long and
   data and classroom envi-          difficult process that must involve
   ronments to provide a bet-        the entire community. But he said
   ter understanding of eq-          that setting standards and devel-
   uity.                             oping new assessments is the es-
                                     sential starting point of such ef-
 • Develop links between re-         forts in order to ensure both ex-
   searchers and the media to        cellence and equity.
   provide context for test-           “I respectfully suggest that all
   score data.                       students are learning already,” said

                             CRESST/CSE TECHNICAL REPORTS

                The following assessment reports are now available by calling (310) 206-1532.
               Or you may fill in the order form on page 20 and mail to CSE/CRESST Annex,
               Graduate School of Education, 405 Hilgard Ave., Los Angeles, CA 90024-4108.

C VERMONT UPDATE                       solving increased, as did the         how fully the goals of the pro-
Can Portfolios Assess Student          amount of time students worked        gram—and of similar reform pro-
Performance and Influence In-          in small groups. Finally, portfo-     grams centered on performance
struction? The 1991-92 Ver-            lios seem to increase teachers’       assessment—can be met.”
mont Experience                        enthusiasm for their subjects and
Daniel Koretz, Brian Stecher,          for teaching.                         C MORE ABOUT VERMONT
Stephen Klein, Daniel McCaffrey,          While there was widespread         Interim Report: The Reliabil-
and Edward Deibert, RAND               support for the reform at the         ity of Vermont Portfolio Scores
CSE Technical Report 371, 1993         school level throughout the           in the 1992-93 School Year
($9.00)                                state—nearly one-half of the          Daniel Koretz, Stephen Klein,
   Vermont’s statewide assessment      schools were voluntarily expand-      Daniel McCaffrey, and Brian
initiative program has garnered        ing the use of portfolios to other    Stecher, RAND
widespread attention nationwide        grade levels—substantial prob-        CSE Technical Report 370, 1993
because of its reliance on portfo-     lems remain. The mathematics          ($3.00)
lios of student work. This 145-        portfolio assessment created new         This interim report provides the
page CRESST/RAND report de-            burdens for principals, teachers      first results of the second year of
scribes results of a multifaceted      and students, including demands       the Vermont assessment program,
evaluation of the program and          on teachers’ time and school re-      focusing on program implemen-
provides information about imple-      sources. Over 80% of fourth-          tation, effects on education, and
mentation of the Vermont assess-       grade teachers and over 60% of        the quality of performance data.
ment, program effects on educa-        eighth-grade teachers reported           “The program was altered in
tional practice, reliability and va-   that they often had difficulty cov-   many ways in 1992-93,” said the
lidity of portfolio scores, and ten-   ering the required curriculum.        CRESST/RAND research team,
sions that exist between assess-       Researchers anticipate that in time   “which resulted in a clear increase
ment and instructional reform.         some of these demands are likely      in the reliability with which math-
   “Findings from the evaluation,”     to decline, although others repre-    ematics portfolios were scored.”
said the research team, “suggest       sent continuing burdens.              However, the researchers added
that the assessment program re-           “The Vermont experience has        that while this progress is encour-
sulted in changes in curriculum        important implications for reforms    aging, scoring reliability in math-
content and instructional style.”      that are underway or under con-       ematics needs to be increased fur-
Additionally, the researchers          sideration in other jurisdictions,”   ther if the program goals are to be
noted that the amount of class-        said the researchers, “but only       achieved. Refining or simplifying
room time devoted to problem           time and careful scrutiny will show   scoring rubrics and placing fur-
                                                                             ther restrictions on types of tasks

                                CRESST/CSE TECHNICAL REPORTS

considered acceptable for inclu-         and statistical moderation, to link   some sort of hybrid system will be
sion in mathematics portfolios are       performance assessments to sets       required…”
among the types of clarifications        of common standards.
that may result in increased reli-          Although the United States will    C Results From the New Stan-
ability.                                 probably not develop a program        dards Project Big Sky Scoring
   “In contrast,” said the research-     of assessments exactly like those     Conference
ers, “the reliability of writing port-   used in England, it is likely that    Lauren Resnick, Daniel Resnick,
folio scores did not improve sub-        the procedures used to compare        and Lizanne DeStefano
stantially and was considerably          the assessments will be similar.      CSE Technical Report 368, 1993
lower than in mathematics.” The             “Currently English secondary       ($3.50)
researchers believe that it is unre-     school exams in various subjects         Partially funded by CRESST,
alistic to expect a substantial rate     are developed and administered        the New Standards Project is an
of improvement in the reliability        by nine examination boards,” said     effort to create a state- and dis-
of the writing portfolio scores          Burton and Linn. “Individual          trict-based assessment and pro-
unless the program itself is sub-        schools are free to choose the        fessional development system that
stantially revised.                      examination board that best fits      will serve as a catalyst for major
                                         their standards. While local con-     educational reform. In 1992, as
C Comparability Across Assess-           trol and high quality of assess-      part of a professional develop-
ments: Lessons From the Use              ments are maintained, the com-        ment strategy tied to assessment,
of Moderation Procedures in              parison of scores across the boards   114 teachers, curriculum supervi-
England                                  is problematic,” added the re-        sors, and assessment directors met
Elizabeth Burton and Robert L.           searchers.                            to score student responses from a
Linn                                        In this report, Burton and Linn    field test of mathematics and En-
CSE Technical Report 369, 1994           discuss the advantages and prob-      glish language arts assessment.
($4.00)                                  lems of moderation by inspection      The results of that meeting, the
  Although there is considerable         and statistical moderation, to-       Big Sky Scoring Conference, were
interest in developing a system of       gether with an explanation of why     used to analyze for comparability
performance-based examinations           neither approach is satisfactory by   across holistic and anaholistic scor-
in the United States, there is a         itself. The authors concluded that    ing methods.
general lack of agreement on how         some combination of the two              “Interscorer reliability esti-
to compare the results of different      approaches may be necessary.          mates,” said the researchers, “for
performance assessments to sets          “Neither a pure moderation by         reading and writing were in the
of common national standards.            inspection nor a strict statistical   moderate range, below levels
This paper addresses the “com-           moderation system is likely to meet   achieved with the use of large-
parison” problem, drawing on two         this [link between assessments and    scale writing assessment or stan-
major approaches used in En-             standards] need,” said Burton and     dardized tasks. Low reliability lim-
gland, moderation by inspection          Linn. “It seems more likely that      its the use of [the] 1992 reading
                                                                               and writing scores for making

                             CRESST/CSE TECHNICAL REPORTS

judgments about student perfor-        lower-middle-class school district,   intervention designed to help
mance or educational programs,”        researchers in this study set forth   teachers develop performance as-
concluded the research team.           to ascertain parents’ opinions        sessments in reading and math-
   However, interscorer reliability    about assessment, including their     ematics. Seeking to evaluate teach-
estimates for math tasks were          opinions about standardized tests     ers’ knowledge, beliefs, and prac-
somewhat higher than for literacy.     versus performance assessments.       tices about assessment and in-
For six out of seven math tasks,       The researchers sought answers        struction, the researchers also
reliability coefficients approached    to several questions including        studied the changes that occurred
or exceeded acceptable levels.         “How do parents in the sample         to teachers during the first semes-
   The findings suggest that the       respond to Gallup Poll questions      ter of the intervention program.
large number and varied nature of      about the desirability of standard-      Findings from the study indi-
participants may have jeopardized      ized national tests and the poten-    cated that the performance as-
the production of valid and reli-      tial uses for standardized test re-   sessment development and imple-
able data. “Scorers reported feel-     sults?” As part of the study, par-    mentation process resulted in
ing overwhelmed and overworked         ents were given an opportunity to     teachers having better under-
after four days of training and        review performance assessment         standings and new insights into
scoring,” said the researchers.        tasks and decide what type of         students’ thinking and learning
   Despite these difficulties, evi-    assessment, standardized or per-      than when teachers relied exclu-
dence was provided that reliable       formance, was most suitable for       sively on more traditional forms
scoring of large-scale performance     classroom use.                        of assessment. However, it was
assessments can be achieved when          The results indicated that when    not clear to what extent teachers
ample time is provided for train-      allowed to look closely at perfor-    changed their instructional pro-
ing, evaluation, feedback, and dis-    mance assessment problems, most       grams to take advantage of their
cussion; clear definitions are given   parents endorsed performance          newly gained insights. Based on
of performance levels and the dis-     assessments for district purposes     their observations so far, research-
tinctions between them; and well-      and especially preferred their use    ers feel confident that as the pro-
chosen exemplars are used.             in classroom contexts.                gram continues, more extensive
                                                                             changes will occur.
C Parent Opinions About Stan-          C Teachers’ Ideas and Prac-
dardized Tests, Teacher’s In-          tices About Assessment and In-        C Dilemmas and Issues in
formation and Performance              struction                             Implementing Classroom-
Assessments                            Hilda Borko, Maurene Flory, and       Based Assessments for Literacy
Lorrie A. Shepard and Carribeth        Kate Cumbo                            Elfrieda H. Hiebert and Kathryn
L. Bliem                               CSE Technical Report 366, 1993        Davinroy
CSE Technical Report 367, 1993         ($4.00)                               CSE Technical Report 365, 1993
($4.00)                                  Participants involved in this       ($3.50)
  Using parents of third-grade         study were part of a year-long           Researchers in this study in-
students in a working-class and

                              CRESST/CSE TECHNICAL REPORTS

vited third-grade teachers from         mathematics, and reports on            authenticity, portfolios support
an urban school district to col-        changes in their instruction and       performance-based assessments
laborate in a classroom-based lit-      assessment as a result of the          that may incorporate shared read-
eracy assessment project. The           project.                               ings of common background
study focused on a series of lit-         During the study, many dilem-        texts, collaborative planning, and
eracy workshops designed to             mas and issues arose that were         opportunities for students to re-
implement a long-standing per-          unique to each of the three schools    vise their work.
spective on curriculum, instruc-        studied, but the most challenging         Based on an in-depth analysis
tion and assessment adapted to          problem was teachers’ focus on         of nine elementary school teach-
classroom-based assessment.             what was important to teach (and       ers actively using writing portfo-
   Some of the early outcomes           therefore assess), and how chil-       lios in their classrooms, the re-
from observations and transcrip-        dren could learn what was              searchers of this study focused on
tions of the workshops indicated        taught—all within the constraints      a technical issue not yet directly
that teachers struggled with a va-      of limited teacher time. As ex-        investigated in R&D studies of
riety of issues including the task of   pected, preliminary results of the     portfolio assessment: “Whose work
embedding assessments like run-         project were mixed, but hopeful.       is it?” If students collaborate with
ning records and written summa-         Researchers believe that future        peers or receive assistance from
ries into their instructional pro-      development and implementation         parents or teachers, the author-
grams. Despite many challenges,         of performance assessments in          ship of classroom work of the
at least one of the schools moved       these classrooms hinge on teach-       student is in question. Focusing
quickly to implement the assess-        ers’ beliefs in these assessments as   here just on the teachers’ contri-
ments and use the information           useful and practical tools.            butions to student work, the au-
the assessments provided.                                                      thors documented patterns of in-
                                        Whose Work Is It? A Question           structional support across writing
C Dilemmas and Issues for               for the Validity of Large-Scale        assignments and students. The
Teachers Developing Perfor-             Portfolio Assessment                   work raises technical issues con-
mance Assessments in Math-              Maryl Gearhart, Joan L. Herman,        cerning the meaningfulness of
ematics                                 Eva L. Baker, and Andrea K.            ‘student’ scores derived from as-
Roberta J. Flexer and Eileen A.         Whittaker                              sessment of student portfolios.
Gerstner                                CSE Technical Report 363, 1993
CSE Technical Report 364, 1993          ($3.00)
($4.00)                                    Portfolio assessment represents
  This paper examines some of           a growing commitment to bridge
the dilemmas and issues that arose      the worlds of public accountabil-
during the first two terms of work      ity and private classroom, and
with teachers participating in the      policy maker and child. Thus,
development of assessments in           within the move toward further

                       More CRESST/CSE TECHNICAL REPORTS

Performance-Based Assessment         The Reliability of Scores From the   Design Characteristics of Science
and What Teachers Need               1992 Vermont Portfolio Assess-       Performance Assessments
  Higuchi                            ment Program                           Glaser, Raghavan, & Baxter
  CSE Technical Report 362, 1993       Koretz, Stecher, & Deibert           CSE Technical Report 349, 1992
  ($4.00)                              CSE Technical Report 355, 1993       ($3.00)
Sampling Variability of Perfor-                                           Accountability and Alternative As-
mance Assessments                    Assessment of Conative Constructs    sessment
  Shavelson, Gao, & Baxter           for Educational Research and           Herman
  CSE Technical Report 361, 1993     Evaluation: A Catalogue                CSE Technical Report 348, 1992
  ($4.00)                              Snow & Jackson                       ($4.00)
                                       CSE Technical Report 354, 1992
Raising the Stakes of Test Admin-      ($8.00)                            Benchmarking Text Understand-
istration: The Impact on Student                                          ing Systems to Human Perfor-
Performance on NAEP                  The Apple Classrooms of              mance: An Exploration
   Kiplinger & Linn                  Tomorrowsm: The UCLA Evalua-           Butler, Baker, Falk, Herl, Jang, &
   CSE Technical Report 360, 1993    tion Studies                           Mutch
   ($4.00)                              Baker, Gearhart, & Herman           CSE Technical Report 347, 1991
                                        CSE Technical Report 353, 1993,     ($5.00)
Issues in Innovative Assessment         ($3.50)
for Classroom Practice: Barriers
and Facilitators                     Collaborative Group Versus Indi-     The Influence of Problem Context
  Aschbacher                         vidual Assessment in Mathemat-       on Mathematics Performance
  CSE Technical Report 359, 1993     ics: Group Processes and Outcomes      Webb & Yasui
  ($4.50)                               Webb                                CSE Technical Report 346, 1992
                                        CSE Technical Report 352, 1993,     ($4.00)
Writing What You Read: Assess-          ($4.00)
ment as a Learning Event                                                  Report on Multilevel and Longitu-
 Wolf & Gearhart                     Educational Assessment: Expanded     dinal Psychometric Models: Latent
 CSE Technical Report 358, 1993      Expectations and Challenges (1992    Variable Models for Analysis of
 ($4.00)                             Thorndike Award Address)             Growth
                                       Linn                                 Muthén & Nelson
Omitted and Not-Reached Items          CSE Technical Report 351, 1992,      CSE Technical Report 345, 1992
in Mathematics in the 1990 Na-         ($3.50)                              ($2.50)
tional Assessment of Educational
Progress                             The Vermont Portfolio Assessment     Measurement of Workforce Readi-
   Koretz, Lewis, Skewes-Cox, &      Program: Interim Report on Imple-    ness Competencies: Design of Pro-
   Burstein                          mentation and Impact, 1991-1992      totype Measures
   CSE Technical Report 357, 1992,   School Year                            O’Neil, Jr., Allred, & Baker
   ($4.00)                             Koretz                               CSE Technical Report 344, 1992
                                       CSE Technical Report 350, 1992       ($4.00)
Latent Variable Modeling of            ($6.00)
Growth with Missing Data &                                                Measurement of Workforce Readi-
Multilevel Data                                                           ness: Review of Theoretical Frame-
 Muthén                                                                   works
 CSE Technical Report 356, 1992                                             O’Neil, Jr., Allred, & Baker
 ($2.50)                                                                    CSE Technical Report 343, 1992
                                                                            ($4.00 )

                           CRESST/CSE TECHNICAL REPORTS

Will National Tests Improve Stu-     A New Mirror for the Classroom:       The Validity and Credibility of the
dent Learning?                       A Technology-Based Tool for           Achievement Levels for the 1990
  Shepard                            Documenting the Impact of Tech-       NAEP in Mathematics
  CSE Technical Report 342, 1991     nology on Instruction                   Linn, Koretz, Baker, & Burstein
  ($3.00)                              Gearhart, Herman, Baker, Novak,       CSE Technical Report 330, 1991
                                       & Whittaker                           ($6.00)
Implications for Diversity in Hu-      CSE Technical Report 336, 1992
man Characteristics for Authentic      ($5.00)                             For a complete list of more than 140
Assessment                                                                 technical reports, monographs, and
  Gordon                             Cross-State Comparability of          resource papers, please call Kim Hurst
  CSE Technical Report 341, 1991     Judgements of Student Writing:        at (310) 206-1532.
  ($2.00)                            Results From the New Standards
The Natural Language Sourcebook        Linn, Kiplinger, Chapman, &
Read, Dyer, Baker, Mutch, Butler,      LeMahieu
Quilici, & Reeves                      CSE Technical Report 335, 1992
  CSE Technical Report 340, 1991       ($5.50)
                                     Effects of Standardized Testing on
Language Assessment Instruments:     Teachers and Learning—Another
LAUSD Language Development           Look
Program for African American Stu-      Herman & Golan                        UCLA’s Center for the Study
dents                                  CSE Technical Report 334, 1991              of Evaluation &
  Butler, Herman, & Yamaguchi          ($5.50)                                 The National Center for
  CSE Technical Report 339, 1991                                               Research on Evaluation,
  ($4.00)                            Conceptual Considerations in           Standards, and Student Testing
                                     Instructionally Sensitive Assess-         Eva L. Baker, Co-director
Discovering What Schools Really      ment                                     Robert L. Linn, Co-director
Teach: Designing Improved Indi-        Burstein                            Joan L. Herman, Associate Director
cators                                 CSE Technical Report 333, 1990            Ronald Dietel, Editor
  McDonnell, Burstein, Ormseth,        ($2.00)                              Katharine Fry, Editorial Assistant
  Catterall, & Moody                                                           Brenda R. Thomas, Layout
  CSE Technical Report 338, 1990     Multilevel Factor Analysis of Class
  ($5.00)                            and Student Achievement Compo-        The work reported in this publica-
                                     nents                                 tion was supported under the Educa-
Writing Portfolios at the Elemen-      Muthén                              tional Research and Development
tary Level: A Study of Methods for     CSE Technical Report 332, 1990      Center Program cooperative agree-
Writing Assessment                     ($3.00)                             ment number R117G10027 and
  Gearhart, Herman, Baker, &                                               CFDA catalog number 84.117G as
  Whittaker                          Complex, Performance-Based As-        administered by the Office of Educa-
  CSE Technical Report 337, 1992     sessment: Expectations and Vali-      tional Research and Improvement,
  ($4.00)                            dation Criteria                       U.S. Department of Education. The
                                       Linn, Baker, & Dunbar               findings and opinions expressed in
                                       CSE Technical Report 331, 1991      this publication do not reflect the
                                       ($3.00)                             position or policies of the Office of
                                                                           Educational Research and Improve-
                                                                           ment or the U.S. Department of

                          MONOGRAPHS           AND    RESOURCE PAPERS

       MONOGRAPHS                                               RESOURCE PAPERS

Assessing Student Achievement:        Writing What You Read: A           Analytic Scales for Assessing
A Profile of Classroom Practices      Guidebook for the Assessment of    Students’ Expository and Narra-
  Dorr-Bremme & Herman                Children’s Narratives              tive Writing Skills
  CSE Monograph 11, 1986               Wolf & Gearhart                      Quellmalz & Burry
  ($11.00)                             CSE Resource Paper 10 ($4.00)        CSE Resource Paper 5 ($3.00)

Evaluation in School Districts:       Improving Large-Scale Assessment   Criteria for Reviewing District
Organizational Perspectives             Aschbacher, Baker, & Herman      Competency Tests
  Bank & Williams (Editors)             CSE Resource Paper 9 ($10.00)      Herman
  CSE Monograph 10, 1981 ($7.50)                                           CSE Resource Paper 4 ($2.00)
                                      Improving Opportunities for
Values, Inquiry and Education         Underachieving Minority Stu-       Issues in Achievement Testing
 Gideonse, Koff, & Schwab (Editors)   dents: A Planning Guide for          Baker
 CSE Monograph 9, 1980 ($11.00)       Community Action                     CSE Resource Paper 3 ($2.50)
                                        Bain & Herman
Toward a Methodology of Natural-        CSE Resource Paper 8 ($11.00)    Evaluation and Documentation:
istic Inquiry in Educational                                             Making Them Work Together
Evaluation                            Designing and Evaluating Lan-        Burry
   Guba                               guage Programs for African-          CSE Resource Paper 2 ($2.50)
   CSE Monograph 8, 1978 ($4.50)      American Dialect Speakers: Some
                                      Guidelines for Educators           An Introduction to Assessment
The Logic of Evaluative Argument        Brooks                           and Design in Bilingual Educa-
 House                                  CSE Resource Paper 7 ($2.00)     tion
 CSE Monograph 7, 1977 ($4.50)                                              Burry
                                      A Practical Approach to Local         CSE Resource Paper 1 ($3.00)
Achievement Test Items—Methods        Test Development
of Study                                Burry, Herman, & Baker
  Harris, Pearlman, & Wilcox            CSE Resource Paper 6 ($3.50)
  CSE Monograph 6, 1977 ($4.50)

                                            FOLD AND SECURE

                                      CSE/CRESST Annex
                                      Graduate School of Education
                                      405 Hilgard Avenue
                                      Los Angeles, California 90024-4108
                                      Mail Code: 139448

                                                       Order Form
                        Attach additional sheet if more room is needed. Form is pre-addressed on reverse.

                       CSE Reports/Monographs/Resource Papers/Videotapes

     Report Number          Title                                        Number of copies     Price per copy       Total Price
     ____________       _______________________________________          ______________       _____________        __________
     ____________       _______________________________________          ______________       _____________        __________
     ____________       _______________________________________          ______________       _____________        __________
     ____________       _______________________________________          ______________       _____________        __________
     ____________       _______________________________________          ______________       _____________        __________
     ____________ c     _______________________________________          ______________       _____________        __________

     POSTAGE & HANDLING                                                       ORDER SUBTOTAL                ____________
     (Special 4th Class Book Rate)
                                                            POSTAGE & HANDLING (scale at left) ____________
     Subtotal of    $0 to   $10     add   $1.50
                                                                       California residents add 8.25% ____________
                   $10 to   $20     add   $2.50
                   $20 to   $50     add   $3.50                                               TOTAL         ____________
                     over   $50     add   10% of Subtotal

                                                                  Orders of less than $10.00 must be prepaid
 Your name & mailing address—please print or type:

                                                                   u    Payment enclosed          u   Please bill me

 _______________________________________________                   u   I would like to receive free copies of the
                                                                       CRESST Line and Evaluation Comment
 _______________________________________________                       publications.

     CSE/CRESST Annex                                                                           NONPROFIT ORG.
     Graduate School of Education                                                                  U.S. POSTAGE
     405 Hilgard Avenue
     Los Angeles, California 90024-4108
     Mail Code: 139448                                                                                 U.C.L.A.