Docstoc

Assessment in Higher Education Institutions

Document Sample
Assessment in Higher Education Institutions Powered By Docstoc
					Assessment in Higher Education
Institutions
                    Presentation
                         to
  National Conference for the Development of Study
 Plans, Teaching & Learning, and Scientific Research,
                  June 15-17, 2010, Amman, Jordan.
                                                          By
                            Karma Fahoum El Hassan, PhD.
                               American University of Beirut.
Outline
   Changes in Higher Education Scene
   Scholarship of Assessment:
       Definition
       Levels of Assessment
       Purposes of Assessment
       Characteristics of Effective Assessment
   Institutional Assessment
   Assessment at Program Level
   Assessment at Course Level
   Conclusion
Changes in Higher Education Scene
   Rapidly changing societal demands of higher education
    intensified with calls for
       accountability,
       planning, and
       demonstrating effectiveness to internal and external stakeholders
   HE institutions had to meet
       increased demands for their services,
       increasingly complex reporting requirements,
       shrinking resources, and
       the need to plan more effectively in a highly competitive
        educational market
Changes in Higher Education Scene

   Societal conditions and industry reshaping forces
    interacted to promote change through higher
    education
       Knowledge-based economy
       Client-centered administrative structures
       Performance-based budgets
       goal attainment, outputs & outcomes accountability
        models
       revolution in undergraduate instruction reflection impact
        of technology
Changes in Higher Education Scene
   These changes and pressures have
     made it necessary to understand the social, political, and
       economic currents shaping the society at large and the external
       constituencies they serve in particular to be successful
     increased the value of information to institutions of higher
       education.
   Redefinition of the nature and role of knowledge required in an
    industry with rapidly changing participants and dynamics
   Assessment thus has become
     an unavoidable condition of doing business because of external
       stakeholder and market forces.
     a necessary condition for undertaking meaningful undergraduate
       reform, as examining systematic evidence of student learning is
       needed.
Scholarship of assessment

   Systematic work that involves
     carrying out assessment,

     determining what methods work best over time, and

     adjusting practice accordingly, then

     reassessing to see if the desired ends were achieved
      and
     sharing findings.
Assessment: Definition
   The systematic and ongoing method of gathering, analyzing and
    using information from various sources about ( an institution, a
    program, and/or extent of attainment of course outcomes) to
    help in decision making and to better understand and improve
    student learning.
   It involves
     making expectations explicit and public
     setting appropriate criteria
     systematically gathering, analyzing, and interpreting evidence to
        determine how well performance matches those expectations
     using the resulting information to document, explain, make
        inferences, and improve performance.
(Thomas A. Angelo, AAHE Bulletin, November 1995, p.7).
Assessment Levels
    Institutional Effectiveness
1.   Institutional assessment, assessment of campus-wide
     characteristics and issues.
    Teaching/Instructional Effectiveness
1.   Program assessment, assessment of academic and
     support programs
2.   Course assessment, assessment of a specific course,
3.   Classroom assessment, assessment of individual
     students at the course level typically by the class
     instructor.
Purposes of Assessment
    Continuous Improvement
     Formative: It is
         ongoing assessment intended to improve performance and outcomes in the
          context of a course or program, or overall institutional improvement.
         Guides or monitors processes while they are still in progress and
         provides feedback
    Accountability
     Summative:
         It occurs at the end of a unit, course, or program.
         The purpose is to
             determine whether or not overall goals have been achieved, and to
             provide information for an individual student or statistics about a course or
              program for internal or external accountability purposes
         Assess outputs and outcomes
    Other
Formative Assessment


   Student Feedback
      •Immediate
                                                   Modify instruction
                                                       •Next steps
        •specific
                                                    •Student activities



                      Monitor student progress
                         •Informal observations
                              •questioning
                        •Student self-assessment
                            •Peer assessment
Characteristics of Effective Assessment
   Systematic involves planning with clear purposes related to goals
   Simple and realistic
   Assessment ongoing not episodic
   Learning is multi dimensional requiring multiple measures
   Assess processes and outcomes
   Undertaken in a supportive environment, requires resources
   Involving stakeholders, and incorporates continuous
    communication with constituents
   Ensures use of assessment data to improve programs and
    services, Commitment to intended use should be a driving force
    in evaluation
   Incorporates ongoing evaluation and improvement of
    assessment itself
Institutional Assessment
   Assessment of extent institution
       a clearly written and actively implemented statement of
        philosophy/mission
       Success in planning, resource allocation and institutional
        renewal
       Efficient use of institutional resources
       Integrity of leadership and governance
       Administrative and organizational services
       Support of institutional resources and processes to
        student learning
Program Assessment: What is Being
Assessed?
Academic Program Assessment Plan
   A well-defined systematic approach to conduct outcomes
    assessment in the context of the college‟s mission.
   The Plan guides the efforts and it should specify
       Content, assessments taking place based on department‟s needs
       Focus, informal document to be internally distributed or formal for
        an external audience.
       When will assessments take place, timeline or schedule for
        implementation and continuance
       Who will be involved in the various steps of the evaluation process,
        distinct division of labor, evidence of faculty and student
        involvement.
       The anticipated consequences of that process.
Model

   The Deming Cycle
   The plan-do-check-act cycle provides a basis for developing
    assessment plans that match needs of the program.
Inventory Existing & Needed Assessment
Tools
   Examine current practices and available information in the context of
    outcomes assessment before considering new initiatives. (30-35% of data
    usually available)
   Possible sources of information include
     Transcript analysis
     Patterns of course-taking
     Trends in student performance
     Tracking exam or course grades over time
     Trends in student enrollment
     Input descriptors for enrolled students (student data such as SAT, ACT,
       high school percentile rank, honors, GPA, or GRE scores)
     Output descriptors for graduates (student data such as GPA, students'
       perceptions of quality of program, employment patterns, or continuing
       education).
Inventory Existing & Needed Assessment
Tools
   Having identified assessment processes that are
    currently in place, can pinpoint questions that are
    not being answered by currently available data
    sources.
   Some guidelines for selecting assessment methods
       Use multiple methods to assess each learning outcome.
       Include both direct and indirect measures.
       Include qualitative as well as quantitative measures.
       Include passive as well as active methods of assessment.
    Examples of Assessment Methods
Direct Assessment Methods              Indirect Assessment Methods
   Course-related assessment           Institutional and Program Surveys
     Capstone Course                      Alumni Surveys
       Assignments/projects
                                           Employer Surveys
     Case Studies
                                           Graduating Seniors and
     Classroom Assessment
                                             Graduates Surveys
     Content Analysis
                                           Student Satisfaction Surveys
     Course-embedded Questions and
       Assignments                      Other
     Portfolios                           Focus groups
     Essays                               Interviews( faculty members,
     Theses, research and projects,         graduating students, alumni)
       publications
     Awards/grants received
   Standardized assessments
   Other
     Transcript analysis
     Placement record of graduates
Matching Assessment Methods and Targets for
Each Learning Outcome
Guidelines for identifying assessment methods for each learning
    outcome
1.   Align assessment method with outcome to be assessed.
2.   Evaluate assessment method based on criteria of
     importance to the department.
3.   Select at least two measures for each learning outcome,
     one of which is direct.
4.   For each outcome and means of assessment, criteria for
     success should be established which includes benchmarks
     that the program sets for the outcome.
Align Assessment Method with Outcome
to be Assessed
Example of an assessment method that does not match the
    learning outcome:

   Outcome: Students completing the Hypothetical Engineering
    program will demonstrate competence in engineering principles
    comparable to graduates of other similar national programs.

   Assessment method: In a locally-developed test, 95% students
    will achieve a score of 90.

Note: When comparing graduates of a program to other graduates nationally,
  using locally developed test as the assessment method is not recommended.
Assessment Matrix: Linking Objectives to Data
Gathering Tools
Key
I = Indirect Methods                        Data Gathering Tools
D = Direct Methods

Objectives                   Enrollment     Senior   Capstone      Focus Groups
                             Trends (OIR)   Survey   Assignment    with students
Applying scientific                                      D               I
methods

Work as professional in                        I         D
field

Satisfaction with advising                     I                         I


Proficiency in written                                   D
communication skills
  Evaluate Assessment Method Based on Criteria of
  Importance to Department: Selection Criteria Matrix.
Key
√ = Adequate tool
+ = Valuable tool
- = Not an effective tool

Criteria of value to                                       Measures
department
                            Standardize   Performance   Portfolio     Surveys    Classroom
                                 d                                              Assignments
                               Tests
Curriculum Match                 -            +            +            √           +
Low data gathering               -             -           √            +           +
costs
Reasonable planning             +              -            -           +           +
time
Reasonable analysis             +              -            -           √           √
time/cost
Value to student                √             +            +             -          √
learning
Motivation                       -            +            +             -          +
Select at least two measures for each learning outcome,
one of which is direct: Assessment Mapping

                          Course        Course                        Capstone
Learning outcomes                                     Course 3456
                           1234          2345                          Course

Apply specific theory
                                      Embedded
                          Pretest                       Project        Posttest
                                      Questions

Acquire necessary
skills and knowledge     Knowledge                     Practical
                                     Skills Pretest                 Team Project
                          Pretest                     Assessment

Proficiency in written
communication skills
                                                                    Comprehensive
                           Paper
                                                                       Paper
 For each outcome and means of assessment,
 criteria for success should be established
1.   Criteria are most often stated in terms of percentages,
     percentiles, averages, or other quantitative measures.
2.    For each learning outcome describe where you would
     like to be within a specified time period (e.g. 10%
     improvement in student performance within two years).
3.   Should be set both at „overall‟ and „component‟ levels as
     benchmarks for program performance.
4.    Component part proves useful for improvement
     purposes.
Example of Means of Assessment and Criteria
for Success – Undergraduate English
   Program Outcome 1:
        Students completing the baccalaureate program in English will
    compare very favorably in their knowledge of literature with those
    students completing similar programs nationally.

   Means of Assessment and Criteria for Success
       The average score of the graduates of the BA program in English
        on the MFAT Literature in English will be at or near the 50th
        percentile compared to national results and no subscale score will
        be below the 30th percentile.
       On the graduation questionnaire, 90% of program graduates will
        “agree” or “strongly agree” with the statement “In the field of
        literature, I feel as well prepared as the majority of individuals
        nationwide who have completed a similar program during the past
        year”.
Example of Means of Assessment and Criteria
for Success – Undergraduate English
   Program Outcome 2:
    Graduates will be able to critique a brief draft essay, pointing out the
    grammatical, spelling, and punctuation errors.


   Means of Assessment and Criteria for Success
       As part of a capstone course during the final semester, students
        will critique a short draft essay; identifying grammatical, spelling
        and punctuation errors. 80% of the program‟s graduates will
        identify 90% of the errors in the draft essay. Additionally, none of
        the 14 rubrics utilized to evaluate the student‟s critique will appear
        to be consistently unmet.
       Students completing the baccalaureate program will be capable of
        writing an acceptable journal article.
Oral Communication Evaluation Sheet
Material Organization                    Criteria for Success
        Points
   Subject                 1-10
   Logical organization    1-10           Overall Criteria
   Content                 1-10
   Supporting material     1-10
                                      Average Rating 85 or Higher
Delivery and Presentation

   Voice and enunciation   1-10         Component Criteria
   Language                1-10
   Gestures                1-10     On No Item will Average Score
   Eye contact             1-10
                                        be less than seven (7)
Overall Effectiveness
   Audience appeal          1-10
   Speaker attitude        1-10

       Total                10-100
Sample Departmental Assessment Timeline
Assessment Reporting

        The first step in developing an assessment report is to
         answer the following questions:
    1.     Who is the audience for this report?
    2.     What do they want to know?
        At its most basic, report should have information to
         answer five basic questions:
    1.     What did you do?
    2.     Why did you do it?
    3.     What did you find?
    4.     How will you use it?
    5.     What is your evaluation of the assessment plan itself?
    Sample Report Description
On an annual basis, programs are required to identify and assess at least
  one intended outcome. Although assessment of only one outcome per
  year will be required, all outcomes must be assessed prior to the
  program‟s scheduled program review.

   The annual assessment report to contain the following components
    1.   Intended outcome statement identified for assessment
    2.   Summary of assessment methods/measures to include brief explanation of
         methodology, persons responsible for data collection and analysis, and a
         summary of assessment results.
    3.   A description of the process used to disseminate assessment results to key
         stakeholders and summary of changes or improvements proposed based on
         assessment results.
    4.   A brief description of the plan and timeline for implementing the proposed
         change (s) and any resources needed to implement the proposed change (s), using
         existing resources already in place, reallocating existing resources from other
         areas, or allocation of new funds.
    Western Carolina University Program Assessment Handbook
Assessment at Program Level: Concluding
Remarks
   Assessment of program learning outcomes has
    proven to be useful for
       Assuring quality of academic programs
       Accountability,
       Institutional effectiveness
           Student learning outcome assessment
           Making improvement based on findings
           Taking findings into consideration when allocating resources
Assessment at Course Level: Steps

1.   Identify goals and learning outcomes,
     what students should be able to do
2.   Gather evidence, how well are students
     meeting these goals
3.   Use information for improvement
Assessment: Cycle of Repeated Basic Actions
Guiding Principles for Conducting
Assessments
   Communicate explicit, attainable, and challenging learning
    targets (what)
   Selected assessment technique must match learning target
   Comprehensive assessment requires a variety of procedures,
    at different times from a variety of sources.
   Proper use of assessment procedures requires an awareness
    of their limitations
   Involve students in assessment, promote reflective self
    assessment
   Obtain a representative sample of tasks and use appropriate
    number of items.
Example: Economics & Political Thought
Course
Goal #1:
   Students will use economic theory and modeling to explain government
    policies and their effects.
Outcomes reflecting this goal:
   1. Students will choose one topic relevant to current economic events and
    explain its relevance in terms of economic principle and theory.
    Assignments that demonstrate accomplishment of this objective:
    a.   Five-to-seven-page midterm paper
    b.   Five minute in-class presentation based on midterm paper

  2. Students will develop and run a statistical model analyzing the current rate
   of inflation in relation to the Consumer Price Index (CPI).
   Assignments that demonstrate accomplishment of this objective:
   a. Data analysis preparation report describing data, steps in developing the
       model and methodology
   b. Statistical analysis results and one-page write-up
adapted from Walvoord & Anderson, Effective Grading (1998).
Assessment Matrix
Goals/Objectives   Method   Results   Interpretation   Action to take
Example: Learning Goals for Biology Majors

   Describe and apply basic biological information and
    concepts

   Conduct original biological research and report results
    orally and in writing to scientific audiences.

   Apply ethical principles of the discipline in regard to
    human and animal subjects, environmental protection,
    use of resources, and collaboration with colleagues
Measures                              Goal 1 Goal 2 Goal 3 Use of information
Standardized test given to              X                   Data are reported to the department
all seniors, & final exams                                  annually by the standardized exam
are administered in three                                   committee and the instructors of the
basic biology courses                                       three basic courses. Department
                                                            supports and encourages instructors,
                                                            takes appropriate department-level
                                                            actions, and reports meeting outcomes
                                                            to dean or body who has resources to
                                                            address problem
In senior capstone course,              X      X       X    Annually, senior capstone teachers share
students complete an original                               students’ scores with the department.
scientific experiment, write it                             The department takes action, as needed.
up in scientific report format,
and make an oral report in the
class. The teacher uses a set of
explicit criteria to evaluate their
work.
Alumni survey asks how well                    X       X    Data reviewed annually by department
alums thought they learned to                               for action.
conduct and communicate
scientific research
Sample of regional employers            X      X       X    Data reviewed annually by department for
gathered two years ago to                                   action.
reflect how well our majors are
doing and give advice to
department.
Guiding Principles for Conducting
Assessments
   Communicate explicit, attainable, and challenging learning
    targets (what)
   Selected assessment technique must match learning target
   Comprehensive assessment requires a variety of procedures,
    at different times from a variety of sources.
   Proper use of assessment procedures requires an awareness
    of their limitations
   Involve students in assessment, promote reflective self
    assessment
   Obtain a representative sample of tasks and use appropriate
    number of items.
Weights to Different Assessments

Weights to be given to different assessments
  determined by:
 Importance of learning outcomes they assess

 Length and effort required in assessment

 Extent of coverage of content and objectives

 Reliability and validity of instruments used.
Conclusion
   Assessment is currently a “movement” with a tradition
    of participatory and practice-oriented scholarship, but it
    needs to be transformed into a “culture” where
    assessment practices become embodied on a day-to-day
    basis.
   Transformation is needed at two levels:
       Teaching & learning and requires shifting assessment
        paradigm from an evaluative stance that checks up on results
        to an emphasis on assuming active and collective
        responsibility for fostering student attainment.
       Administration and requires evolving a largely top-down
        management oriented use of information in planning and
        decision making toward a culture that more fully embodies the
        principles of a learning organization.
Universities As Learning Organizations

They
   Are systematic in defining their missions as institutions
    and units within an institution;
   Mindful of the needs and expectations of stakeholders
    as they set goals and organize to deliver services;
   collect systematic evidence concerning outcomes and
    processes;
    use this evidence to improve work, and
   share their successes so that others can build on them.
        ‫شكرا‬
        ‫ً‬

‫د. كرمةًالفاهومًالحسن‬
‫‪kelhasan@aub.edu.lb‬‬

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:7
posted:8/9/2011
language:English
pages:50