Docstoc

Journal Article Research on Assessing Students Learn

Document Sample
Journal Article Research on Assessing Students Learn Powered By Docstoc
					   Evaluating and
Assessing Library
       Instruction
Examples & Best Practices
Presenters

   Martin Garnar – Regis University
   Rhonda Gonzales & Eleni Adrian –
    CSU-Pueblo
   Lorrie Evans – Auraria Library
   Patrick McCarthy - CSU
Evaluating and Assessing
Library Instruction
Regis University
Martin Garnar
ACRL Guidelines for
Evaluation and Assessment
   program evaluation plan
   criteria for evaluation
   learning outcomes and assessment methods
   coordination of assessment with teaching
    faculty
   data gathering and analysis
Evaluating and Assessing
Library Instruction
CSU-Pueblo
Eleni Adrian & Rhonda Gonzales
CSU-Pueblo Library
Instruction Evaluation

Overview

   Librarians at CSU-Pueblo administered a brief
    evaluation form after each instruction session.
   Collected these evaluations from 1999-2004.
   Entered results into an Access Database.
   Data Clean-up
   Statistical Analysis
   Conclusions
CSU-Pueblo Library
Instruction Evaluation


Evaluation Form

Background questions
Satisfaction questions
Ranking question
CSU-Pueblo Library
Instruction Evaluation


Evaluation Forms

Collected Fall 1999 through Summer 2004.
181 classes
2441 evaluation forms
CSU-Pueblo Library
Instruction Evaluation

Access Database

Entered several semesters’ of data
Revealed errors in our database design
New database structure
New input forms
CSU-Pueblo Library
Instruction Evaluation

Access Database
CSU-Pueblo Library
Instruction Evaluation

Access Database
CSU-Pueblo Library
Instruction Evaluation


Data Fixes: New Database Design

Relationship between the classes and the
   evaluations tables.

No answers for background questions.
CSU-Pueblo Library
Instruction Evaluation

Data Fixes: Form Changes

Two questions added, two questions removed.
Re-entered all old evaluation form data to
   match the final version.
Double-checked that the form changes were
   accurately re-entered.
CSU-Pueblo Library
Instruction Evaluation


Data Fixes: Data Integrity

Duplicate Class records
Duplicate and blank Evaluations records
Typos in Status (year in school)
General data entry errors
CSU-Pueblo Library
Instruction Evaluation
Results – Background Questions

                        Status of Respondents
Year in
                                                                       C ategory
School                                 faculty
                                                                       faculty
                                                                       freshman
                                       0.7%
                  sophomore                                            graduate
                      20.1%                                            junior
                                                                       senior
                                                                       sophomore

                                                            freshman
                                                            39.5%



              senior
              16.0%




                                                 graduate
                              junior             3.0%
                              20.6%
CSU-Pueblo Library
Instruction Evaluation
Results – Background Questions

                                Attended Session Before
Attended            1600
Previous                                     1436
                    1400
Session
                    1200

                                                          973
                    1000
            Count




                    800

                    600

                    400

                    200
                           32
                      0
                           NA                No           Yes
                                       Attended Before
CSU-Pueblo Library
Instruction Evaluation
Results – Background Questions

Specific                        Student Had a Specific Assignment
                     2000
Assignment                                                          1800




                     1500
             Count




                     1000


                                                            579

                     500


                                           61
                            1
                       0
                                          NA                 No     Yes
                                        Had a Specific Assignment
CSU-Pueblo Library
Instruction Evaluation
Results – Background Questions

Professor                           Professor Participated in Session
                       2000                                             1948
Participated

                       1500
               Count




                       1000




                       500
                                                      349

                              144


                         0
                              NA                      No                Yes
                                               Professor Present
CSU-Pueblo Library
Instruction Evaluation
Results – Satisfaction Questions

                      This presentation helped me learn how to start my research.
Start
                                                                                1194
                   1200
Research
                   1000                                                952



                   800
           Count




                   600


                   400
                                                              243
                   200

                           7         14           14   17
                     0
                           *         1            2    3       4        5        6
                                                       A
           1=strongly disagree 6=strongly agree
CSU-Pueblo Library
Instruction Evaluation
Results – Satisfaction Questions

Clear                                        The presentation was clear.
                                                                                 1374
Presentation           1400

                       1200

                       1000
                                                                           832
                       800
               Count




                       600

                       400

                                                                     193
                       200
                                         6            7      26
                              3
                         0
                              *          1            2      3        4    5      6
                                                             B
               1=strongly disagree 6=strongly agree
CSU-Pueblo Library
Instruction Evaluation
Results – Satisfaction Questions

                     The hands-on experience will help me to complete my assignment.
Hands-on                                                                        1173
                     1200
Experience
                     1000
                                                                       836
                     800
             Count




                     600


                     400
                                                               282

                     200
                                                    50   64
                             17        19
                       0
                             2         1            3    *     4        5        6
                                                         C
             1=strongly disagree 6=strongly agree
CSU-Pueblo Library
Instruction Evaluation
Results – Satisfaction Questions
                         The librarian treated each student with respect.
Respect           1800
                                                                            1680

                  1600

                  1400

                  1200

                  1000
          Count




                  800

                  600

                  400                                     338      361


                  200
                                                 51
                          4            7
                    0
                          3           1          4        *        5         6
                                                      D
          1=strongly disagree 6=strongly agree
CSU-Pueblo Library
Instruction Evaluation
Results – Ranking Question
                       The most important thing that I learned was:
Most
Important                                     5
                                         15.1%
Thing                                                           1
                                                                36.1%
Learned
                                  4
                             15.8%




                                     3
                                 8.9%


                                                  2
                                                  24.1%

            Category         1=how to locate a journal article or book
            1                2=how to access the library's databases
            2                3=how to use Interlibrary Loan to get my information
            3                4=how to decide which databases I should use
            4
                             5=how to search the Internet more efficiently
            5
CSU-Pueblo Library
Instruction Evaluation
Comparisons
             Significance of Having an Assignment on Students' Feeling that
                     Presentation Helped Them Start Their Research

       100.00%

        50.00%

          0.00%
                      1          2            3           4              5           6

   No Assignment     0.52%     1.22%         1.39%      11.98%        43.75%       41.15%
   Had Assignment    0.61%     0.39%         0.50%      9.41%         37.79%       51.31%
   All Respondents   0.59%     0.59%         0.72%      10.03%        39.23%       48.84%


                             No Assignment    Had Assignment     All Respondents
CSU-Pueblo Library
Instruction Evaluation
Comparisons
            Significance of Having an Assignment on Students' Feeling
                       that Hand's-On Practice Was Helpful

        60.00%

        40.00%
        20.00%

          0.00%
                      Strongly     Moderately    Slightly   Slightly Agree   Moderately   Strongly Agree

    No Assignment      1.42%         1.06%       2.48%         15.93%         35.22%         43.89%
    Had Assignment     0.57%         0.63%       2.00%         10.61%         34.91%         51.28%
    All Respondents    0.78%         0.73%       2.11%         11.91%         34.99%         49.48%


                                 No Assignment    Had Assignment         All Respondents
CSU-Pueblo Library
Instruction Evaluation
Conclusions

   Distribution of responses skewed towards high
    end.
   Student evaluations of faculty teaching across
    campus are not normally skewed towards the
    positive to this extent.
   These results show that a large majority of
    students felt that our instruction sessions were
    helpful!
CSU-Pueblo Library
Instruction Evaluation
Conclusions - Meeting Our Goals?

Since most of our sessions are introductory in nature,
    one of our primary goals is to show students how
    to start their research using our library.
     88.17% of students that took the survey indicated that
      they moderately or strongly agreed that the sessions
      helped them learn how to start their research.
     36.12% of students who took the survey responded that
      learning how to locate a journal article or book was
      the most important thing they learned.
CSU-Pueblo Library
Instruction Evaluation
Conclusions - Factors Affecting Results
       What factors may have affected student responses?
            Chi-square tests indicate that the following factors proved to be statistically
             significant:
               Having a specific assignment in conjunction with this presentation was a
                significant factor on the number of students that strongly agreed with A & C
               Attending a prior session was a slightly significant factor in determining how
                many students strongly disagreed with D
            Survey Design:
               Even number of response choices forced students to choose positive or
                negative
               Unclear wording – “My Professor participated in this session.”
               Lack of N/A forced students to choose a response even if it did not apply
            Survey Administration
               Administered at the end of each session, when students were in a hurry to
                leave.
               Tendency of librarians may be to skip the survey if the session hasn’t gone
                well.
CSU-Pueblo Library
Instruction Evaluation
Indications for Best Practices

Role of Evaluation:
   combine learning assessment with program
   evaluation

Survey Design:
   concise, clearly worded, avoid mild statements,
   allow neutral and N/A responses, define
   hypotheses prior to creating the form
Evaluating and Assessing
Library Instruction
Auraria Library
Lorrie Evans
Evaluation of instruction at
Auraria Library
Survey student learning in single session library
 classes (random selection).

Survey students in full semester credit bearing
 classes (assignments, final project)

Survey the faculty toward the end of the
 semester – after papers and assignment are
 turned in.
Challenges in the single session
library class

                       Do the students
                        understand the most
                        important concepts?

                       No other chance for a
                        follow up or review.

                           When students leave
                            the room we will not
                            see them again
       Look in the mirror – are we
Why?   teaching what we think we are
       teaching.

       Collaborate with campus faculty on
       a pro-active level.

       Make library instruction a more
       integrated part of the academic
       culture.

       Accreditation – assessment
       mandates driven externally.
 The tool – short online survey


In class – three to five
   questions mapped to
   learning goals.              Broadcast results, review
                                answers & main concepts.
Customized for each class
                                       Students share
                                       concerns.

 •Students see their answer along with classmate responses.
 •Librarian can respond to student input.
 •Review main points.
Sample results
   English 1020
       Student comments view


   Psychology freshman seminar data

   Graduate Political Science - single session post test.

   Graduate English Literature - semester course

                                                   Jump past
                                                     results
Comments




       return
Psych


Freshman
Seminar

sample
items




 return
Graduate English Literature survey:
    sample items




                                      return
Political
Science


post test



   return
Right tool for the job question
Faculty survey -- end of the
semester


   Program effectiveness as reflected in quality
    of student projects and assignments.

   Satisfaction with content, scheduling and
    teaching.
Faculty
Evaluation

survey
results




 Zoomerang
    link
Faculty
evaluations

sample
items
How can we improve?
    Feedback loop
   In class survey – instant feedback for both students and
    librarian. Librarian can go over confusing concepts.

   Results discussed at departmental meetings – used as
    source of information to make adjustments in teaching
    (more depth with less quantity).

   Results made available to instructors and students.

   End of the year report submitted to the campus
    Outcomes Assessment Advisory Committee. The
    committee provides a review of the report, recommends
    improvements.
Evaluating and Assessing
Library Instruction
CSU – Fort Collins
Patrick McCarthy
Evaluating Learning Outcomes
at the Reference Desk - CSU
   Why?
       Determine if learning takes place as a result of
        interactions at the reference desk
       Identify those skills most suitable to reference
        desk instruction
       Establish reasonable reference desk learning
        outcome benchmarks
       Recommend behaviors and methods that lead to
        student learning
Categories of Reference
Assessment
   Accuracy response rate
   User satisfaction
   Course/instructor opinion survey
   Learning outcomes
Methodology?
   Quantitative approach
       Powerful, not practical


   Qualitative approach
       Powerful, versatile
Procedures
1.   Patron completes preliminary survey at
     completion of reference interaction
2.   Researcher analyzes preliminary surveys
     and selects relevant patrons for follow-up
     assessment
3.   Selected patrons are interviewed on
     specifics of reference transaction
4.   Interview recordings are transcribed and
     coded for assessment
Analysis
   Transfer assessment
   Accuracy assessment
   Patron must provide evidence
   Authentic assessment
Group Discussion –
Guiding Questions
1. How have you determined the learning needs of
   your community?
2. What's your best method for evaluating library
   instruction?
3. What's your best method for assessing progress on
   learning outcomes?
4. Do you have examples of successful collaboration
   with classroom faculty for evaluation and/or
   assessment?
5. How have you used data from your instruction
   program evaluations and/or assessments?
 Assessment of Instruction – Examples &
 Best Practices
 Presenter Contact Information
Martin Garnar, Reference Librarian          Rhonda Gonzales, Interim Dean
Dayton Memorial Library, Regis University   Eleni Adrian, Instruction Coordinator
3333 Regis Blvd., Mail Stop D-20            University Library
Denver, CO 80221-1099                       Colorado State University – Pueblo
303.964.5459                                2200 Bonforte Blvd.
303.964.5497 (fax)                          Pueblo, CO 81001
mgarnar@regis.edu                           719.549.2361
                                            rhonda.gonzales@colostate-pueblo.edu
                                            eleni.adrian@colostate-pueblo.edu

Lorrie Evans,                               Patrick McCarthy
Library Instruction Services                Coordinator of Instruction
Auraria Library                             Morgan Library
1100 Lawrence Street                        Colorado State Univ.
Denver, CO 80204                            501 University Avenue
303.556.6776                                Fort Collins, CO 80523
Lorrie.Evans@cudenver.edu                   720.491.1881
                                            pmccarthy@manta.colostate.edu

				
DOCUMENT INFO
Description: Journal Article Research on Assessing Students Learn document sample