Docstoc

Sample Presentation Title

Document Sample
Sample Presentation Title Powered By Docstoc
					Making Our Way from
Anecdote to Evidence

          Presentation for
 Georgia College & State University

      Caroline R. Noyes, Ph.D.
Assistant Director, Office of Assessment
    BACKGROUND AND
     INTRODUCTION




2
    Four Purposes of Evaluation
    (Mark, Henry, & Julnes ,2000)




    1. Assessment of merit and worth: the development of warranted
       judgments, at the individual and societal level, of the value of a
       policy or program.
    2. Program and organizational development: the effort to use
       information to directly modify and enhance program operations.
    3. Oversight and compliance: the assessment of the extent to
       which a program follows the directives of statutes, regulations,
       rules, mandated standards or any other formal expectations.
    4. Knowledge development: the discovery or testing of general
       theories, propositions, and hypotheses in the contexts of policies
       and programs.



3
    Program
    Mission



    Program
     Goals


     Student
    Learning
    Outcomes

4
    Program Mission
    History:
       The department has as its mission through the BA in History to provide
       students with a basic knowledge of selected areas of history; to teach students
       to think critically in analyzing historical issues, to write clearly and cogently on
       historical topics, and to conduct historical research; and to provide students
       with a basic understanding of history as a discipline.

    Theatre:
       We seek to provide the best education possible in theatre and dance for our
       students, preparing them for careers in the professional and educational
       arenas both in and out of the performing arts. We cultivate imagination,
       passion, discipline, and collaboration in the creation of arts and artists. We
       emphasize interdisciplinary connections and a global perspective as essential
       components of a liberal arts education.



5
    Sample Program Goals
    a.   Graduates will be able to communicate their knowledge effectively,
         both orally and in writing.

    b. Graduates will demonstrate knowledge of the major authors, periods,
       movements, and issues in drama and theater.

    c. Graduates will understand and apply basic research methods in
       psychology, including research design, data analysis, and
       interpretation.




6
                   Properties of Effective
                Student Learning Outcomes
       Have a clear purpose
       Use action words
       Describe meaningful learning
       Are easily understandable
       Represent high level learning
       Result in observable behaviors/products




7
    Formulating Student Learning Outcomes

          Method
            Inventory of program content


              Rank most important items
              Categorize by outcome type
              Draft outcome statement
             Revise using criteria for quality outcomes
          Use action verbs
          Should be observable (either directly or indirectly
           inferred)



8
    Bloom’s Taxonomy of the Cognitive Domain

                        Evaluation


                        Synthesis


                         Analysis



                       Application



                      Comprehension


                       Knowledge



9
     Action Verbs Accompanying Bloom’s Taxonomy

     Knowledge   Comprehension   Application   Analysis        Synthesis Evaluation
     Describe    Compare         Apply         Analyze         Arrange     Argue
     Find        Describe        Classify      Appraise        Assemble    Assess
     List        Distinguish     Complete      Categorize      Collect     Choose
     Locate      Discuss         Construct     Compare         Combine     Conclude
     Name        Explain         Demonstrate   Contrast        Comply      Decide
     Recall      Express         Dramatize     Debate          Compose     Evaluate
     Record      Identify        Employ        Diagram         Construct   Interpret
     Relate      Outline         Illustrate    Differentiate   Create      Judge
     Repeat      Predict         Interpret     Distinguish     Design      Justify
     State       Recognize       Operate       Examine         Devise      Measure
     Tell        Restate         Practice      Experiment      Formulate   Rate
     Underline   Tell            Schedule      Inspect         Imagine     Recommend
     Write       Translate       Sketch        Inventory       Manage      Revise
                                 Solve         Investigate     Organize    Support
                                 Use           Question        Plan        Value
                                               Separate        Predict     Verify
10
     Bloom’s Taxonomy of the Affective Domain


                       Characterization


                       Organization


                         Valuing


                       Responding


                        Receiving



11
     Action Verbs Accompanying Bloom’s Taxonomy
     http://www.abet.org/_TrainingCD/data/references/Assessment%20Links/Goals_revised_Blooms.pdf




     Receiving             Responding                Valuing                 Organization          Characterization
     Asks                  Answers                   Completes               Adheres               Acts
     Chooses               Assists                   Describes               Alters                Discriminates
     Describes             Complies                  Differentiates          Arranges              Displays
     Follows               Conforms                  Explains                Defends               Influences
     Gives                 Discusses                 Follows                 Explains              Listens
     Identifies            Helps                     Initiates               Generalizes           Modifies
     Names                 Performs                  Joins                   Identifies            Performs
     Selects               Presents                  Justifies               Integrates            Practices
     Replies               Selects                   Reads                   Modifies              Proposes
     Uses                  Tells                     Reports                 Organizes             Qualifies
                                                     Selects                 Prepares              Questions
                                                     Shares                  Relates               Serves
                                                     Studies                 Synthesizes           Solves
                                                     Works                                         Uses
                                                                                                   Verifies

12
     Student Learning Outcomes:
         Oral Communication
      Graduates will be able to communicate their knowledge effectively,
      both orally and in writing.

                        Students will:


      Knowledge         support arguments with relevant and adequate
                        evidence



      Skills            answer questions directly and accurately


      Dispositions      convey enthusiasm for their topic




13
     Student Learning Outcomes:
         Written Communication
      Graduates will be able to communicate their knowledge
      effectively, both orally and in writing.
                      Students will:

       Knowledge      respond to essay questions on exams with clear, well-
                      organized presentations focused on the topic and
                      relatively free of errors



       Skills         recognize well-written texts and advise peers on how to
                      improve their written work


       Dispositions   develop a personal voice and sense of style in writing




14
     Student Learning Outcomes:
         Research Methods (Psychology)
      Graduates will understand and apply basic research methods in
      psychology, including research design, data analysis, and
      interpretation
                      Students will:
      Knowledge       articulate strengths and limitations of various research
                      designs

      Skills          formulate testable hypotheses, based on operational
                      definitions of variables

      Dispositions    recognize that theoretical and socio-cultural contexts
                      as well as personal biases may shape research
                      questions, design, data collection, analysis, and
                      interpretation



15
     WORKING ON LEARNING
     OUTCOMES IN GROUPS




16
     GC&SU Core Outcomes
     • Communicate clearly in written English, demonstrating
       comprehension, analysis, and critical interrogation of a variety of
       texts. (A: Essential Skills; English)
     • Use description, analysis, and synthesis of data, ideas, or
       information appropriate to the purpose. (A: Essential Skills:
       English)
     • Express and interpret mathematical information, concepts, and
       thoughts in verbal, numeric, graphical, and symbolic form, and to
       shift among these different modes when solving problems. (A:
       Essential Skills; Math)
     • Use appropriate technology in the location, evaluation, analysis,
       synthesis, and application of information in problem-solving
       situations (A: Essential Skills; hybrid of Math and English)
17
     FINDING THE EVIDENCE




18
     Things to Consider:
        What’s already in place?
          Exams, assignments, or projects?
          Common exams, assignments, or projects?
          Writing samples (common question)
          Capstone experience products

        Look for logical assessment points in the curriculum
          At program entrance
          Program courses required for the major
          At program conclusion

        The evidence you collect depends on the questions you
         want to answer
19
     Kinds of Evidence, Part 1

     Direct Evidence                      Indirect Evidence
     Measure student learning outcomes    Measure opinions or thoughts
     directly                             about students’ (or alumni)
                                          own knowledge, skills, attitudes,
                                          learning experiences, etc. Also
        Embedded questions/tasks
                                          includes external evaluations.
        Pre-post testing
                                             Alumni surveys
        Evaluation of projects/work
                                             Exit surveys
        Standardized tests (e.g., MFT)
                                             Employer surveys
        Portfolios
                                             Focus Groups
        Presentations

20
     Kinds of Evidence, Part 2

     Quantitative Evidence                 Qualitative Evidence
        Ratings of student                   Observations of student
         skills/performance                    behavior w/ systematic notes
        Scores and pass rates on tests       Summaries of discussion
         (licensure/certification)             threads
        Scores on locally-designed           Student reflections
         tests (final exams, qualifiers,      Portfolios evaluation
         comprehensives) when                 Written work, performances,
         accompanied by test                   or presentations scored with a
         “blueprints”                          rubric
        Classroom response systems           Analysis of comments on
         (clickers)                            surveys
21
     Student Learning Outcome:
     answer questions directly and accurately
     Course                       Point of          Assessment      Results
                                  Assessment        Method

     Introduction to Psychology   Oral              Presentation
                                  presentation on   rubric
                                  40 STCP           (professor)
                                  chapter
     Research Methods             Oral             Presentation
                                  presentation of rubric
                                  research project (professor and
                                                   peers)
     Psychology Seminar           Classroom         Presentation
                                  guest lecture on rubric
                                  historical figure (professor)
                                  in psychology



22
     Oral Presentation Rubric               (http://www1.uprh.edu/cruzmigu/OralRubric.pdf)


                 4                3                   2                        1
     Subject     Student          Student is at       Student is               Student does
     Knowledge   demonstrates     ease with           uncomfortable            not have grasp
                 full knowledge   expected            with                     of information;
                 by answering     answers to all      information              student cannot
                 all class        questions,          and is able to           answer
                 questions with   without             answer only              questions
                 explanations     elaboration         rudimentary              about subject.
                 and                                  questions
                 elaboration




23
     Student Learning Outcome:
     articulate strengths and limitations of various research designs
     Course                       Point of      Assessment        Results
                                  Assessment    Method

     Introduction to Psychology   1st Test      Multiple Choice
                                                or Matching
                                                Question
     Developmental Psychology     1st Test      Application
                                                short answer

                                  Final Exam    Essay question
     Research Methods             a) 1st Test   Application
                                                question

                                  b) Research   Rationale for
                                  Proposal      research
                                                method
                                                selection

24
      IDENTIFYING SOURCES
     OF EVIDENCE IN GROUPS




25
     SHARING YOUR FINDINGS




26
     At Last: Reporting your Findings
        Analyze the data…identify the results

        Integrate results from various assessment methods

        What conclusions can you draw from that evidence?

        What recommendations arise from that data?

        What actions should/will you take based on the
         recommendations?
27
     Student Learning Outcome:
     answer questions directly and accurately
     Course                               Point of               Assessment           Results
                                          Assessment             Method
     Introduction to Psychology           Oral presentation      Presentation         Spring 2010:
                                          on 40 STCP             rubric (professor)   n=55, avg. score
                                          chapter                                     was 1.9

     Research Methods                     Oral presentation      Presentation         Fall 2009: n=25,
                                          of research            rubric (professor    avg. score was
                                          project                and peers)           2.6 (avg. peer
                                                                                      rating was 3.1)
     Psychology Seminar                   Classroom guest        Presentation         Spring 2010:
                                          lecture on             rubric (professor)   n=25, avg. score
                                          historical figure in                        was 2.9
                                          psychology
     In general, as students progress though the major, there is improvement in students’ ability to
     answer questions at the end of their oral presentation, but students may benefit from more
     practice in elective classes. Increase use of peer ratings to increase student critical reflection
     on this task.


28
     Closing the Loop, Option 1


                                  Outcome



                                  Practice



29
     Closing the Loop, Option 2


                                  Practice



                                  Outcome



30
     Conclusions
     The “systematic collection, review, and use of information
     about educational programs undertaken for the purpose of
     improving learning” allows us to address three
     fundamental questions:

     1. What have our students learned?
     2. How well have our students learned that information?
     3. How successful have we been at what we are trying to
        accomplish?


31
     Contact Information
     Caroline R. Noyes, Ph.D
     Assistant Director, Office of Assessment
     Georgia Institute of Technology
     (404) 894-0510
     cnoyes@gatech.edu




32

				
DOCUMENT INFO