Get documents about "�Assessment Nuts and Bolts�
Shared by: HC111211204333
-
Stats
- views:
- 5
- posted:
- 12/11/2011
- language:
- pages:
- 18
Document Sample
―Assessment Nuts and Bolts‖
with
Barbara D. Wright
Assessment Coordinator,
Eastern Connecticut State University
& Former Director, AAHE Assessment Forum
wrightb@easternct.edu
California Assessment Institute
Palm Springs, CA
September 29, 2002
Institutional
Mission
Educational
Goals, Qs
Use
Gathering
Evidence
Interpretation
THE ASSESSMENT LOOP
Assessment —
A systematic process of setting goals for or asking questions about
student learning, gathering evidence, interpreting it, and using it to
improve the effects of college on students’ learning and development
Words, words…
Inputs
Outputs
Outcomes
Baseline
Measurement
Evidence, documentation
Direct evidence
Indirect evidence
Summative
Formative
Cross-sectional
Longitudinal
Accountability
Improvement
Assessment
Evaluation
DOMAINS for assessment of student learning –
1. Basic (entry) skills
2. College-level skills
3. General education
4. The major/ vocational program/ certificate program
5. Social/ ethical/ spiritual development
LEVELS of assessment
1. Institution (a.k.a. ―institutional effective‖)
2. Program
3. Multiple-section course
4. Individual student
ACTORS in assessment –
1. Faculty
2. Students
3. Academic support staff
4. Administrators
5. Internship supervisors
6. Employers
7. Graduates
8. Faculty from other institutions
BLOOM’S TAXONOMY OF COGNITIVE
PROCESSES
Thinking Meaning
Processes
The recall of information
Knowledge (tell, list, describe, relate, locate,
write, find, state, name)
To show an understanding of information
Comprehension (explain, interpret, outline, discuss,
distinguish, predict, restate, translate,
compare, describe)
To use some previously learned
Application knowledge, rule or method in a new
situation
(solve, show, use, illustrate, calculate,
construct, complete, examine, classify)
To break information into parts to explore
Analysis understandings and relationships
(analyze, distinguish, examine, compare,
contrast, investigate, categorize, identify
explain, separate, advertise)
To put together ideas in a new way to
Synthesis develop a new or unique product
(create, invent, compose, predict, plan,
construct, design, imagine, improve,
propose, devise, formulate)
To judge the value of materials or ideas on
Evaluation the basis of set criteria
(judge, select, choose, decide, justify,
debate, verify, argue, recommend, assess,
discuss, rate, prioritize, determine)
Higher order thinking is…
*nonalgorithmic. That is, the path of action is not fully specified
in advance.
*complex. The total path is not “visible” from any single vantage
point.
* often yields multiple solutions, each with costs and benefits
* requires nuanced judgment and interpretation
* involves application of multiple criteria, which sometimes
conflict with one another
* often involves uncertainty. Not everything that bears on the task
is known
* requires self-regulation. Someone else is not “calling the plays”
at every step
* involves imposing meaning, finding structures in apparent
disorder.
* is effortful. The elaborations and judgments required entail
considerable mental work
(adapted from Lauren B. Resnick, Education and Learning to Think, National Academy
Press, 1987)
Surface Learning Deep Learning
Unrelated bits of knowledge Relationships, connections
Memorization, formulas Patterns, principles
Difficulty ―making sense‖ Active integration
Little meaning, value in course, tasks Logic, evidence, conclusions
Study without reflection, strategy Understanding metacognition
Feelings of pressure, worry Active interest, satisfaction
Levels of understanding include…
*Mentioning: incoherent factoids w/o structure
*Describing: brief, derived form materials provided
*Relating: outline w/ explanations but lacking detail, supporting
arguments
*Explaining: relevant evidence in structured, coherent arguments
*Conceiving: individual conceptions of topics produced through
reflection
(adapted from Noel Entwistle)
Assessment for deep learning, understanding…
Uses open-ended formats
Focuses on ―essentials‖
Poses authentic, relevant problems, tasks
Scores for understanding
(adapted from Noel Entwistle)
And avoids the disconnects…
across-the-curriculum skills isolated course assignments
higher-order thinking focus on teaching, testing of
facts
intellectual curiosity, questioning acceptance, reproduction of
answers
complexity, nuance ―right‖ answers – or total
relativism
going beyond the textbook, the formula fear of overstepping bounds
confidence, courage insecurity, risk-aversion,
anger
Some approaches to assessing complex outcomes…
Portfolios
Performances
Capstone projects
Secondary readings
Common assignments
Locally designed tests
Validity…
*Content validity
*Face validity
*Consequential validity
Shifts in Assessment Practice
(from Assessment Standards for School Mathematics, 1995)
From To
Assessing only knowledge of Full range of students’
specific facts, isolated skills mathematical power, significant
mathematics
Memorization, reproduction Problem solving: investigating,
reasoning representing, applying, &
communicating
Comparing student performance Comparing performance against
with that of other students established criteria
Indicating right & wrong answers Providing detailed feedback to
students about their answers
A single way for students to Multiple methods, multiple
demonstrate mathematical opportunities, e.g. observation,
knowledge, e.g. multiple-choice or open-ended tasks, projects, writing
short-answer test assignments as well as tests
Over-simplified evidence Complex, high-quality evidence
“Teacher-proof” assessment Supporting teachers & their
judgments
A secret, exclusive & fixed process Open, public & participatory process
Reporting only group means Detailed analyses of group data, e.g.
disaggregation, analysis of variations
Assessment to filter, select students Assessment to support all student learning
Students as objects of measurement Students as active participants
Assessment as episodic, conclusive Assessment as continual, recursive
Excerpt from
Assessment Standards for School Mathematics, 1995, p. 60
…The Mathematics Level I and Level II sections of the SAT consist of
fifty multiple-choice questions to be answered in an hour (1.2 minutes
for each question). Similarly, in 1994, the American High School
Mathematics Examination consisted of thirty questions to be answered
in ninety minutes (3 minutes for each question). Such time-limited,
multiple-choice examinations may assess some important skills but are
limiting in that they fail to include opportunities for students to
demonstrate their capability to use those skills to solve non-routine
problems…The message sent to teachers and students by this type of
examination is that it is the rapid use of well-learned techniques that is
most important (my emphasis – BDW)…
…Examinations given for certification in other countries and in
international competitions provide examples of more complex problems.
Chantal Shafroth’s (1993) study of high school leaving examinations in
several industrialized nations found that they are typically composed of
a few independent problems, each made up of several parts, and that
the time allocated to answer each part was more than ten minutes…The
mathematics examination for liberal arts students in the Netherlands
consists of four problems, each with four parts, to be completed in three
hours…This allows students to demonstrate what they can do on every
problem while probing their depth of understanding…
…(Most) examinations given in international competitions…contain a
small number of complex problems, with solutions judged by a jury that
may rate responses on the clarity of presentation, reasoning used, and
mathematical elegance in the solution process, in addition to accuracy of
solution. For example, the International Mathematical Olympiad
requires that each competitor complete the total of six problems during
a two-day period.
Descriptions are adapted from General Education: Explorations in Evaluation. The Final
Report by Paul L. Dressel and Lewis B Mayhew (Washington, DC: 1954)
OBJECTIVES IN SCIENCE
I. Ability to recognize and state problems: The student is expected to
a) recognize and identify the central problem;
b) indicate, with reasons, whether a given problem is stated specifically enough to begin an
investigation of it; and
c) indicate whether non-scientific factors (e.g., value judgments, matters of faith) are
contained in the problem.
II. Ability to select, analyze, and evaluate information in relation to a problem: The student is
expected to
a) recognize when given information is inadequate;
b) indicate sources of appropriate additional information;
c) evaluate the authenticity of given sources of information in relation to a given problem;
and
d) apply information (s)he possesses or has gathered to solution of a given problem.
III. Ability to recognize, state, and test hypotheses and other tentative explanations: The student
is expected to
a) formulate or recognize hypotheses based on given data or situations;
b) identify the evidence necessary to judge the truth of a given deduction from a hypothesis;
c) formulate an experiment which will test the truth of a given hypothesis;
d) recognize when observations or experimental data do or do not support the hypothesis,
and to what degree; and
e) recognize assumptions involved in a hypothesis.
IV. Ability to formulate, recognize, and evaluate conclusions: The student is expected to
a) recognize the generalization(s) in an interpretation or conclusion;
b) detect the unstated assumptions involved in a conclusion;
c) recognize when evidence is adequate for drawing a conclusion;
d) recognize in a line of reasoning whether an observation plays the role of a premise or
verifies the conclusion;
e) recognize the use of such forms of reasoning as deduction, induction, citing of authority,
or analogy;
f) differentiate between fact and assumption; and
g) recognize the difference between interpretations based on scientific evidence and those
which contain opinion.
V. Ability to recognize and formulate attitudes and take action after critical consideration: The
student is expected to
a) recognize proper or improper use of such concepts as causality, teleology, simplicity,
consistency, tentative nature of truth, operationalism; and
b) assess a situation and recognize appropriate action in harmony with the nature of science
and society.
CRITICAL ANALYSIS AND JUDGEMENT IN THE HUMANITIES (adapted from Dressel
and Mayhew)
I. Subjective reaction: The student is expected to
a) recognize what the painting (poem, etc.) is about;
b) describe how it affects him/her;
c) describe how it might affect other people;
d) speculated on what the feelings of the artist seem to have been;
e) etc.
II. Analysis: The student is expected to
a) identify what seem to be the creative and/or technical problems (matters of function,
purpose, form, space limitations, etc. ) with which the artist is dealing;
b) describe the general method of expression (handling of space, distortion, use of light,
interest in textures of objects represented, etc. );
c) comment on any relationship between results obtained and the medium used, and on
appropriateness of the choice of medium;
d) describe the elements of the painting (things that can be considered separately, such as
line, color, intensity and use of light and dark, shape, texture of paint, etc.);
e) discuss organization of the painting, including the use of specific devices or plans to hold
the elements together, or the lack or such a plan; and
f) indicate characteristics of the painting that lead the student to believe it belongs to some
particular period school, or artist.
III. Synthesis and expression of overall judgment: The student is expected to
a) choose judiciously from the observations made above;
b) produce a judgment of the general significance and meaning of the painting; and
c) express that judgment in a good, well-rounded essay, oral presentation, or other format.
Goal: Critical Reading Ability (based on SUNY Fredonia)
1. Subgoal: Identify the main point or thesis.
Can the student identify the main point or thesis of the article?
Can the student distinguish between the main point and other
assertions, information?
2. Subgoal: Identify supporting arguments
Can the student identify supporting arguments?
Can the student distinguish between supporting arguments and
main thesis?
3. Subgoal: Identify examples
Can the student identify examples used to support arguments?
Can the student distinguish between argument and supporting
example?
4. Subgoal: Recognize bias, assumptions, gaps
Can the student identify, describe expression of bias?
Can the student identify, describe assumptions, both expressed and
unspoken?
Does the student recognize when logical steps, positions,
information are omitted?
5. Subgoal: Recognize rhetorical means used for persuasive effect
Can the student identify examples of effective or biased use of
language?
Can the student identify elements of the author’s style of thinking
that influence effectiveness?
6. Subgoal: Relate the content to personal values, experience
Can the student explain how the argument relates to personal
values, convictions?
Can the student relate the argument to personal experience?
University of Northern Colorado
General Scoring Rubric
Department of Mathematical Sciences
4 Advanced: Completed on schedule.
Develops multiple, original problem models.
Demonstrates with multiple models at high level of clarity.
Execution exhibits in-depth solution(s).
Verifies, justifies, and extends the solution(s).
Demonstrates the ability to vary presentations for any audience.
3 Proficient: Completed on schedule.
Develops an original problem model.
Demonstrates with multiple models with a reasonable level of
clarity.
Execution exhibits appropriately correct solution(s).
Verifies and justifies the solution(s).
Demonstrates the ability to vary the presentation for more than
one audience.
2 Essential: Completed in a reasonable manner.
Extends the textbook problem(s) model.
Demonstrates a model with a reasonable level of clarity.
Execution shows a reasonable effort with slight execution
errors.
Verifies the solution.
Presentation is appropriate for a given audience.
1 In Progress: No attempt, incomplete or completed in an unreasonable time.
Uses only the textbook problem(s) model.
Model and/or clarity is lacking.
Execution shows minimal effort – no solution.
Minimal, partial, or no verification.
Presentation is inappropriate for a given audience.
0 No Effort
The Rating Scale
SUPERIOR
Can support opinion, hypothesize, discuss abstract topics, and
handle a linguistically unfamiliar situation.
ADVANCED
Can narrate and describe in past, present
and future time/aspect, and handle a
complicated situation or transaction.
INTERMEDIATE
Can create with language
ask and answer simple
questions on familiar
topics, and handle a
simple situation
or transaction.
NOVICE
No functional
ability;
speech
limited to
memorized
material.
Inverted Pyramid Showing Major Levels
of ACTFL Rating Scale
Illistration 2-A
American Council on the Teaching of Foreign Language, Inc. February, 1989
Assessment is not…
Something (entirely) new
A fad
Program review
Faculty evaluation
Testing, grading
A quick fix
One-size-fits-all
An assault on academic freedom
A scientific experiment
Gathering data, writing reports
Comparisons against other programs
Quality assurance
An end in itself
BENEFITS of assessment done right—
1. Better information
2. More and better student learning
3. Stronger programs
4. Intellectual fun, inquiry, for faculty
5. Enhanced collegiality
6. Better campus-wide communication
7. Evidence-based administrative decisions
8. Happier employees of your graduates
9. Better public relations
10. Enhanced fundraising, grant-getting
11. A more successful accreditation
