Points for Knowledge Survey Design

							Points for Knowledge Survey
            Design
      Concept of a Knowledge Survey




1. = I have insufficient knowledge to answer this question.
2. = I have partial knowledge or know where to quickly (20
minutes or less) obtain a complete answer to this question.
3. = I can fully answer this question with my present
knowledge.
   A Framework for Planning
•Responsibilities
 –Goals
   •Outcomes
     –How would I confirm student mastery?
     (questions, challenges, applications,
     skills)
        »Organize items in order of course
        presentation
  Responsibility Example --to meet
    a published goal statement
"Goal 5: To understand how the physical
sciences explain the natural world. Courses in
the physical sciences which fulfill this
requirement (1) examine the processes by
which scientific knowledge is gained, (2)
introduce the basic concepts and terminology
of one or more of the physical sciences, and
(3) explore how scientific knowledge
influences human society." (Idaho State
University Undergraduate Catalog, 2006)
        Important Definitions
• GOAL: A general descriptive statement statement of
  intent, reason, or vision for a particular educational
  experience. Example - the 12 General Education
  Goals in the ISU Catalog.

• LEARNING OUTCOMES: Measurable skills or
  knowledge that are a result of instruction. These are
  action statements—verbs—that specify what
  students should be able to do at the end of a planned
  educational experience. These are the most
  important assessment criteria in accreditation
  reviews. Example--Drafted outcomes for the 12 goals
  now in notes.
         Example of outcomes:
            Students will be able to:
1. describe the scientific method and provide an example of its
   application;
2. pick a single theory from the science represented by this
   course and explain its historical development;
3. provide two examples of testable hypotheses;
4. provide two specific examples that illustrate why it is important
   to the everyday life of an educated person to be able to
   understand science;
5. describe two current examples of the relationship between
   physical science and public policy…. (4 more)
          Evaluation versus
            Assessment
• Evaluation - addresses measures applied to make
  decisions about individuals such as tests, graded
  papers & projects--grades of students and
  summative ratings of individual professors.
• Assessment - addresses measures of students’
  success in meeting specific learning goals
  through performance on outcomes. Uses
  aggregated data to describe performance of units
  --courses, classes, programs, institutions--as a
  whole.
         Assessment Instruments Contrasted with
                   Evaluation Tools
•   Assessments examine learning          •   Evaluations examine individuals’
    outcomes in aggregate by unit             performance
    (class, specific major, specific      •   Based mainly on instructors’ goals &
    institutional degree)                     outcomes
•   Based mainly on a unit’s chosen       •   Traditionally based on a few faculty-
    goals & outcomes                          made tools of undocumented
•   Where possible, based on multiple         reliability
    tools of known reliability            •   Examples of evaluative input for
•   Examples: pre-post tests of               grades: short-answer tests & quizzes,
    established reliability, knowledge        in-class essays, graded homework,
    surveys, standardized exams,              participation, attendance --less often
    concept inventories, records of           based on open-ended challenges,
    success on registration exams or          group projects, rubrics, service
    successful applications for               learning, multifaceted competency
    certification                             levels
•   Work when they inform instructional   •   Work when they are aligned with clear
    planning and improve instructional        information about goals and
    quality                                   outcomes and are enacted with truth
                                              and fairness. For students, we must
                                              test on what we teach.
      Assessment Instruments Are
             Beneficial
•   When they promote students’ learning
•   When they assist instructional planning
•   When they aid curricular design
•   When they have high benefit:cost
•   When information provided is unique
    with only some overlap with (is not
    redundant with) other measures
    An Ideal Assessment Instrument

• (1) furnishes unique data that provides some overlap with
  but does not duplicate that of other tools
• (2) provides reliable, quantifiable data about student
  understanding
• (3) provides data useful to students’ cognitive and
  meta-cognitive growth
• (4) helps faculty improve course design and
  instruction
• (5) helps units improve curricula.
                Knowledge Surveys
•   Are very reliable assessment      •   Are not test substitutes
    tools                             •   Are not primarily evaluative
•   Obtain valuable, unique           •   Are not replacements for
    information, some of which            summative and formative
    overlaps with that of tests,          evaluations
    student ratings                   •   Are optimal for assessment when
•   Reveal levels of course               used with other tools to inform
    challenge                             overall interpretations
•   Information is obtained under     •   Are not simply passive
    different conditions from tests       measures. Their effect to
•   Are amenable to numerical             enhance teaching and learning
    analyses used for tests and           depends on the degree to which
    student ratings                       they are skillfully employed.
•   Are useful to developing self
    assessment skills in students
   KS items are not interpreted in
isolation from one another. Specific
   items map to into more global
              concepts.
 10. Outline your understanding of geologic time and discuss
    how this course opened your mind to the notion of four-
    dimensional science.
 The following KS items mapped to the above Goal 5 outcome
    were: 14, 15, 16, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
    36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
    78, 101, 115, 120, 121, 141, 147, 148, 151, 154, 155, 159,
    185, 210, 211, 222, 238, 263, 268.