�Assessment Nuts and Bolts� by Ut4uFcs


									     ―Assessment Nuts and Bolts‖


          Barbara D. Wright

          Assessment Coordinator,
    Eastern Connecticut State University
& Former Director, AAHE Assessment Forum

    California Assessment Institute
           Palm Springs, CA
          September 29, 2002

                                               Goals, Qs




                  THE ASSESSMENT LOOP

Assessment —
A systematic process of setting goals for or asking questions about
student learning, gathering evidence, interpreting it, and using it to
improve the effects of college on students’ learning and development
Words, words…


    Evidence, documentation

    Direct evidence
    Indirect evidence




DOMAINS for assessment of student learning –

1. Basic (entry) skills

2. College-level skills

3. General education

4. The major/ vocational program/ certificate program

5. Social/ ethical/ spiritual development

LEVELS of assessment

1. Institution (a.k.a. ―institutional effective‖)

2. Program

3. Multiple-section course

4. Individual student

ACTORS in assessment –

1. Faculty

2. Students

3. Academic support staff

4. Administrators

5. Internship supervisors

6. Employers

7. Graduates

8. Faculty from other institutions

   Thinking                        Meaning

                The recall of information
 Knowledge      (tell, list, describe, relate, locate,
                write, find, state, name)
                To show an understanding of information
Comprehension   (explain, interpret, outline, discuss,
                distinguish, predict, restate, translate,
                compare, describe)
                To use some previously learned
 Application    knowledge, rule or method in a new
                (solve, show, use, illustrate, calculate,
                construct, complete, examine, classify)
                To break information into parts to explore
  Analysis      understandings and relationships
                (analyze, distinguish, examine, compare,
                contrast, investigate, categorize, identify
                explain, separate, advertise)
                To put together ideas in a new way to
  Synthesis     develop a new or unique product
                (create, invent, compose, predict, plan,
                construct, design, imagine, improve,
                propose, devise, formulate)
                To judge the value of materials or ideas on
 Evaluation     the basis of set criteria
                (judge, select, choose, decide, justify,
                debate, verify, argue, recommend, assess,
                discuss, rate, prioritize, determine)
                     Higher order thinking is…

*nonalgorithmic. That is, the path of action is not fully specified
 in advance.

*complex. The total path is not “visible” from any single vantage

* often yields multiple solutions, each with costs and benefits

* requires nuanced judgment and interpretation

* involves application of multiple criteria, which sometimes
 conflict with one another

* often involves uncertainty. Not everything that bears on the task
 is known

* requires self-regulation. Someone else is not “calling the plays”
 at every step

* involves imposing meaning, finding structures in apparent

* is effortful. The elaborations and judgments required entail
 considerable mental work

(adapted from Lauren B. Resnick, Education and Learning to Think, National Academy
Press, 1987)
Surface Learning                               Deep Learning
Unrelated bits of knowledge                    Relationships, connections

Memorization, formulas                         Patterns, principles

Difficulty ―making sense‖                      Active integration

Little meaning, value in course, tasks         Logic, evidence, conclusions

Study without reflection, strategy             Understanding metacognition

Feelings of pressure, worry                    Active interest, satisfaction

   Levels of understanding include…
   *Mentioning: incoherent factoids w/o structure

   *Describing: brief, derived form materials provided

   *Relating: outline w/ explanations but lacking detail, supporting

   *Explaining: relevant evidence in structured, coherent arguments

   *Conceiving: individual conceptions of topics produced through

                                             (adapted from Noel Entwistle)
Assessment for deep learning, understanding…

    Uses open-ended formats

    Focuses on ―essentials‖

    Poses authentic, relevant problems, tasks

    Scores for understanding

                                      (adapted from Noel Entwistle)

And avoids the disconnects…
across-the-curriculum skills               isolated course assignments

higher-order thinking                      focus on teaching, testing of

intellectual curiosity, questioning        acceptance, reproduction of

complexity, nuance                         ―right‖ answers – or total

going beyond the textbook, the formula     fear of overstepping bounds

confidence, courage                        insecurity, risk-aversion,
Some approaches to assessing complex outcomes…


Capstone projects

Secondary readings

Common assignments

Locally designed tests

*Content validity

*Face validity

*Consequential validity
Shifts in Assessment Practice
(from Assessment Standards for School Mathematics, 1995)

From                                    To

Assessing only knowledge of             Full range of students’
specific facts, isolated skills         mathematical power, significant

Memorization, reproduction              Problem solving: investigating,
                                        reasoning representing, applying, &

Comparing student performance           Comparing performance against
with that of other students             established criteria

Indicating right & wrong answers        Providing detailed feedback to
                                        students about their answers

A single way for students to            Multiple methods, multiple
demonstrate mathematical                opportunities, e.g. observation,
knowledge, e.g. multiple-choice or      open-ended tasks, projects, writing
short-answer test                       assignments as well as tests

Over-simplified evidence                Complex, high-quality evidence

“Teacher-proof” assessment              Supporting teachers & their

A secret, exclusive & fixed process     Open, public & participatory process

Reporting only group means              Detailed analyses of group data, e.g.
                                        disaggregation, analysis of variations

Assessment to filter, select students   Assessment to support all student learning

Students as objects of measurement      Students as active participants

Assessment as episodic, conclusive      Assessment as continual, recursive
Excerpt from
Assessment Standards for School Mathematics, 1995, p. 60

…The Mathematics Level I and Level II sections of the SAT consist of
fifty multiple-choice questions to be answered in an hour (1.2 minutes
for each question). Similarly, in 1994, the American High School
Mathematics Examination consisted of thirty questions to be answered
in ninety minutes (3 minutes for each question). Such time-limited,
multiple-choice examinations may assess some important skills but are
limiting in that they fail to include opportunities for students to
demonstrate their capability to use those skills to solve non-routine
problems…The message sent to teachers and students by this type of
examination is that it is the rapid use of well-learned techniques that is
most important (my emphasis – BDW)…

…Examinations given for certification in other countries and in
international competitions provide examples of more complex problems.
Chantal Shafroth’s (1993) study of high school leaving examinations in
several industrialized nations found that they are typically composed of
a few independent problems, each made up of several parts, and that
the time allocated to answer each part was more than ten minutes…The
mathematics examination for liberal arts students in the Netherlands
consists of four problems, each with four parts, to be completed in three
hours…This allows students to demonstrate what they can do on every
problem while probing their depth of understanding…

…(Most) examinations given in international competitions…contain a
small number of complex problems, with solutions judged by a jury that
may rate responses on the clarity of presentation, reasoning used, and
mathematical elegance in the solution process, in addition to accuracy of
solution. For example, the International Mathematical Olympiad
requires that each competitor complete the total of six problems during
a two-day period.
Descriptions are adapted from General Education: Explorations in Evaluation. The Final
Report by Paul L. Dressel and Lewis B Mayhew (Washington, DC: 1954)


I. Ability to recognize and state problems: The student is expected to
   a) recognize and identify the central problem;
   b) indicate, with reasons, whether a given problem is stated specifically enough to begin an
      investigation of it; and
   c) indicate whether non-scientific factors (e.g., value judgments, matters of faith) are
      contained in the problem.

II. Ability to select, analyze, and evaluate information in relation to a problem: The student is
    expected to
    a) recognize when given information is inadequate;
    b) indicate sources of appropriate additional information;
    c) evaluate the authenticity of given sources of information in relation to a given problem;
    d) apply information (s)he possesses or has gathered to solution of a given problem.

III. Ability to recognize, state, and test hypotheses and other tentative explanations: The student
     is expected to
     a) formulate or recognize hypotheses based on given data or situations;
     b) identify the evidence necessary to judge the truth of a given deduction from a hypothesis;
     c) formulate an experiment which will test the truth of a given hypothesis;
     d) recognize when observations or experimental data do or do not support the hypothesis,
         and to what degree; and
     e) recognize assumptions involved in a hypothesis.

IV. Ability to formulate, recognize, and evaluate conclusions: The student is expected to
    a) recognize the generalization(s) in an interpretation or conclusion;
    b) detect the unstated assumptions involved in a conclusion;
    c) recognize when evidence is adequate for drawing a conclusion;
    d) recognize in a line of reasoning whether an observation plays the role of a premise or
       verifies the conclusion;
    e) recognize the use of such forms of reasoning as deduction, induction, citing of authority,
       or analogy;
    f) differentiate between fact and assumption; and
    g) recognize the difference between interpretations based on scientific evidence and those
       which contain opinion.

V. Ability to recognize and formulate attitudes and take action after critical consideration: The
   student is expected to

   a) recognize proper or improper use of such concepts as causality, teleology, simplicity,
      consistency, tentative nature of truth, operationalism; and
   b) assess a situation and recognize appropriate action in harmony with the nature of science
      and society.
and Mayhew)

I. Subjective reaction: The student is expected to
   a) recognize what the painting (poem, etc.) is about;
   b) describe how it affects him/her;
   c) describe how it might affect other people;
   d) speculated on what the feelings of the artist seem to have been;
   e) etc.

II. Analysis: The student is expected to
    a) identify what seem to be the creative and/or technical problems (matters of function,
       purpose, form, space limitations, etc. ) with which the artist is dealing;
    b) describe the general method of expression (handling of space, distortion, use of light,
       interest in textures of objects represented, etc. );
    c) comment on any relationship between results obtained and the medium used, and on
       appropriateness of the choice of medium;
    d) describe the elements of the painting (things that can be considered separately, such as
       line, color, intensity and use of light and dark, shape, texture of paint, etc.);
    e) discuss organization of the painting, including the use of specific devices or plans to hold
       the elements together, or the lack or such a plan; and
    f) indicate characteristics of the painting that lead the student to believe it belongs to some
       particular period school, or artist.

III. Synthesis and expression of overall judgment: The student is expected to
     a) choose judiciously from the observations made above;
     b) produce a judgment of the general significance and meaning of the painting; and
     c) express that judgment in a good, well-rounded essay, oral presentation, or other format.
Goal: Critical Reading Ability (based on SUNY Fredonia)

1. Subgoal: Identify the main point or thesis.
     Can the student identify the main point or thesis of the article?
     Can the student distinguish between the main point and other
       assertions, information?

2. Subgoal: Identify supporting arguments
     Can the student identify supporting arguments?
     Can the student distinguish between supporting arguments and
       main thesis?

3. Subgoal: Identify examples
     Can the student identify examples used to support arguments?
     Can the student distinguish between argument and supporting

4. Subgoal: Recognize bias, assumptions, gaps
     Can the student identify, describe expression of bias?
     Can the student identify, describe assumptions, both expressed and
     Does the student recognize when logical steps, positions,
       information are omitted?

5. Subgoal: Recognize rhetorical means used for persuasive effect
     Can the student identify examples of effective or biased use of
     Can the student identify elements of the author’s style of thinking
       that influence effectiveness?

6. Subgoal: Relate the content to personal values, experience
     Can the student explain how the argument relates to personal
       values, convictions?
     Can the student relate the argument to personal experience?
                     University of Northern Colorado
                         General Scoring Rubric

                   Department of Mathematical Sciences

4   Advanced: Completed on schedule.
              Develops multiple, original problem models.
              Demonstrates with multiple models at high level of clarity.
              Execution exhibits in-depth solution(s).
              Verifies, justifies, and extends the solution(s).
              Demonstrates the ability to vary presentations for any audience.

3   Proficient: Completed on schedule.
                Develops an original problem model.
                Demonstrates with multiple models with a reasonable level of
                Execution exhibits appropriately correct solution(s).
                Verifies and justifies the solution(s).
                Demonstrates the ability to vary the presentation for more than
                      one audience.

2   Essential:   Completed in a reasonable manner.
                 Extends the textbook problem(s) model.
                 Demonstrates a model with a reasonable level of clarity.
                 Execution shows a reasonable effort with slight execution
                 Verifies the solution.
                 Presentation is appropriate for a given audience.

1   In Progress: No attempt, incomplete or completed in an unreasonable time.
                 Uses only the textbook problem(s) model.
                 Model and/or clarity is lacking.
                 Execution shows minimal effort – no solution.
                 Minimal, partial, or no verification.
                 Presentation is inappropriate for a given audience.

0   No Effort
                                                                          The Rating Scale

         Can support opinion, hypothesize, discuss abstract topics, and
                 handle a linguistically unfamiliar situation.

                   Can narrate and describe in past, present
                    and future time/aspect, and handle a
                    complicated situation or transaction.

                           Can create with language
                            ask and answer simple
                             questions on familiar
                              topics, and handle a
                                simple situation
                                 or transaction.

                                 No functional
                                  limited to

                  Inverted Pyramid Showing Major Levels
                          of ACTFL Rating Scale
                              Illistration 2-A

American Council on the Teaching of Foreign Language, Inc. February, 1989
Assessment is not…

     Something (entirely) new

     A fad

     Program review

     Faculty evaluation

     Testing, grading

     A quick fix


     An assault on academic freedom

     A scientific experiment

     Gathering data, writing reports

     Comparisons against other programs

     Quality assurance

     An end in itself
BENEFITS of assessment done right—

1. Better information

2. More and better student learning

3. Stronger programs

4. Intellectual fun, inquiry, for faculty

5. Enhanced collegiality

6. Better campus-wide communication

7. Evidence-based administrative decisions

8. Happier employees of your graduates

9. Better public relations

10. Enhanced fundraising, grant-getting

11. A more successful accreditation

To top