Docstoc

plenary

Document Sample
plenary Powered By Docstoc
					Designing Teaching/Learning
Activities and Assessments to
Align with Intended Learning
           Outcomes


            John Biggs
  Principal Consultant OBTL Project
                  Workshops

This talk will be followed by workshops over the next
  few weeks.

It is important that, when attending later workshops,
   you bring at least one set of course ILOs with you. The
   workshops will be hands-on and it is important you
   work with genuine course material.

Your Faculty Consultant will also be working with you
 in designing TLAs and assessment tasks.
                        Overview
   Macro level steps in OBTL:
   1. Derive the programme aims and intended learning
       outcomes in terms of what it is intended students should
       learn. Match courses to PILOs

 Micro level steps using constructive alignment:
 2. State the intended learning outcomes of each course using
      learning verbs that can unequivocally be recognised in
      students’ performances and that state a level of
      performance.
 3. Engage students in learning activities that require them to
      use those verbs.
 4. Decide how well the outcomes have been learned by using
      assessment tasks that also embody those learning verbs.
 5. Determine students’ final grades on the basis of how well
     they have attained those outcomes.
From Programme ILOS to Graded
          Outcomes
Graduate attributes
                                   TLAs



  PILOs          CILOs               Graded Outcome


                      capstone ?     Assessment tasks
Programme specific
    Macro Level: Programme ILOs
   Derive around 5 or 6 generic statements for the PILOs,
    each to be addressed by the Course ILOs. It helps to
    map the CILOs to the PILOs using a common verb or
    learning activity required to achieve the CILO.

   From various sources (graduate attributes,
    professional requirements) classify PILOs by
    knowledge, intellectual skills, practical skills, etc. This
    produces as many as 30 PILOs. A grid or map helps to
    judge that all PILOS are addressed by courses
    (see presentation by Dr. Andy Chan)

Once the courses are established (by either method) the
 verbs in the CILOs are used to align the TLAS and
 Assessment tasks.
            Micro Level: Course ILOS
      “Acquire” to what level of understanding?
   CILOs with verbs like ―understand‖, ―acquire knowledge‖
    or ―discuss‖ do not go far enough. What would a student
    have to do to show that he/she understands at the level you
    want? To simply describe the concept, or explain how it
    relates to other concepts, or be able to apply it in a practical
    setting?
   It is therefore necessary to specify the level of understanding
    so that you and the students can recognize when that level
    has been reached.
   Teachers will know what they mean, but the level of
    understanding needs to be operationalised in a way that can
    be confirmed by assessment. What do students have to do to
    meet the ILO?
   I find the SOLO taxonomy useful for making these
    distinctions.
        Deciding a level of understanding using
                 a hierarchy of verbs
                                                                         Theorize
                                                                         Generate
                                                                         Hypothesize
                                                                         Reflect
                                                    Compare/contrast
                                                    Explain causes
                                  Enumerate
                                  Describe
                                                    Analyze                 ....
                                                    Relate
                                  List              Apply
                                  Combine
                Identify          Do algorithms
                Do simple
                 procedures



 Misses point

Prestructural   Unistructural     Multistructural      Relational      Extended Abstract

                   Quantitative Phase                        Qualitative Phase
                        Constructive Alignment
                                   Graded Course ILOs


                                   A
Teaching / Learning
                         The very best understanding that could be   Assessment Tasks
Activities
                         reasonably expected: verbs such as
                         hypothesise, apply to ―far‖ domains,        Format such that
Designed to
                         generate, relate to principle, etc.         the target verbs are
elicit desired
verbs                                         B                      elicited and
                         Highly satisfactory understanding: verbs    deployed
May be:                  such as explain, solve, understand main     in context.
                         ideas, analyze, compare, etc.
 Teacher-controlled                           C
                         Learning that is clearly passable: verbs    Criteria clearly allow
 Peer-controlled        such as elaborate, can explain major         judgement as to how
                        topics, salvageable higher level attempts.   well the student
 Self-controlled                                                     has met each
                                              D                      course ILOs.
as best suits context   Learning that barely warrants a Pass: low
                        level verbs, also inadequate but
                        salvageable higher level attempts.
 Now to Align Teaching/Learning
Activities and Assessments to ILOs
When the Programme ILOs and Course ILOs have been
 designed, we turn to Course ILO. We need to:

   help students achieve the ILO by activating the verb(s)
    embedded in that ILO with suitable Teaching/Learning
    Activities. There are better alternatives than lectures
    and tutorials, even in large classes.

   obtain evidence from assessment tasks that enable us
    to judge how well an individual student has achieved
    each ILO.

   combine all this evidence to assign a final letter-grade,
    based on how well the ILOs have been met.
   What will encourage students to
         achieve the ILOs?

Designing a situation that requires them to enact the same
  VERB as in the ILO.
The Teaching/Learning Activities (TLAs) (the teaching
  methods you use) embody these verbs.

If the ILO says ―compare X and Y methods of …‖ the
   TLA should require the students to do the comparing,
   not the teacher to tell them about the similarities and
   differences between X and Y


Golden Rule: Don’t you do it, let them do it.
 What kinds of knowledge do teaching methods
                     elicit?
Teacher-controlled
Lecture, set text            Reception of selected content
Think-aloud                  Demonstrate conceptual skills
Questioning                  Clarifying, seeking error
Project                      Research skills, creativity
Concept mapping              Structuring, overview
Tutorial                     Elaboration, clarification
Laboratory                   Procedures, application
Excursion                    Experiential knowledge, interest
Seminar                      Clarification, presentation skills
Peer-controlled
Various                      Elaboration, problem solving, metacognition
Learning partners            Resolve differences, application
Peer teaching                Depends whether teacher or taught
Spontaneous collaboration    Breadth, insight
Self-controlled
Generic study skills         Basic self-management
Content study skills         Information handling
Reflective learning skills   Independence and self-monitoring
Four common (TLAs) and associated teaching
         and learning activities
  TLA             Teacher               Student

LECTURE  talk, explain, clarify listen, take notes, accept, query?
                                 discuss?
TUTORIAL set/answer questions pre-read, prepare questions,
         provide feedback        learn from peers, critique,
                                         analyse
PROJECT set brief, ongoing      apply, create, self-monitor
         feedback               communicate, show teamwork
PBL      set problems            set learning goals, design, apply,
         accessing desired      integrate, solve problems
         content, skills

                     What ILOs do you want?
ILO Verb             Possible TLA
Acquire content      set reading, lecture, field trip
Explain              tutorial, written essay
Integrate            project, assignment
Apply                project, case study
Solve problem        PBL, case study
Design, create       project, creative writing
Hypothesise          experiment, project
Reflect              reflective diary

The point is not how you are going to teach but
 how you want your students to learn
            Relevant TLA Workshops

   Designing teaching and learning activities to align with
    ILOs.
   Facilitating active learning in large classes.
   Facilitating active learning in small classes.
   Providing constructive feedback
   Facilitating achievement of ILOs using case studies
   Facilitating achievement of ILOs in placement settings
   Achieving and assessing ILOs with online learning
   Others as need arises – but main work will be done in
    Faculty with Consultant
 How well do students achieve
          the ILOs?
Assessment tasks provide evidence on how
 well the ILOs have been achieved.

We need then to select assessment tasks
 that are aligned to the ILOs on the basis
 of the verbs involved.
Assessment Task                   Likely Kind of Learning Assessed
Extended prose, essay-type
 Essay exam                       Rote, question spotting, speed structuring
 Open book                        As for exam but less memory, coverage
 Assignment, take home            Read widely, inter-relate, organise, apply copy

Objective test
 Multiple choice                  Recognition, strategy, comprehension, coverage
 Ordered outcome                  Hierarchies of understanding

Performance assessment
 Practicum                        Skills needed in real life
 Seminar, presentation            Communication skills
 Critical incidents               Reflection, application, sense of relevance
 Project                          Application, research skills
 Reflective journal               Reflection, application, sense of relevance
 Case study, problem              Application, professional skills
 Portfolio                        Reflection, creativity, unintended outcomes
Rapid assessments (large class)
 Concept maps                     Coverage, relationships
 Three minute essay               Level of understanding, sense of relevance
 Gobbets                          Appreciating significant detail, why
 Short answer                     Recall units of information, coverage
 Letter to a friend               Holistic understanding, application, reflection
 Cloze                            Comprehension of main ideas
ILO Verb            Possible Assessment Task
Acquire content     written assignment, exam,
                       MCQ
Explain             assignment, exam, oral
Integrate           project, assignment
Apply               project, case study
Solve problem       set problems, case study
Design, create      project, creative writing
Hypothesise         experiment, project
Reflect             reflective diary

The tasks are to provide evidence for how well
 the ILOs have been met.
Relevant Assessment Workshops
   Theory of assessing ILOs
   Designing assessment methods to align with ILOs
   Assessing ILOs in large classes
   Assessing ILOs in placement settings
   Assessing ILOs using portfolio assessment
   Classroom assessment techniques for formative
    assessment
   Assessing group work
   Others as need arises – but main work will be done
    in Faculty with Consultant
            Grading in OBTL
In traditional assessment, both norm- and
  criterion-referenced, grading is based directly
  on performance on the assessment tasks:
   In NRA – comparisons between students' marks (e.g.
     grading on the curve)
   In CRA – levels of performance on the assessment
     criteria determine the student’s grade.
   In OBTL, a student’s grade is based on how well
     the ILOs have been achieved, not directly on
     performance on the assessment tasks.
   OBTL is criterion-referenced in that the ILOs
     supply the criteria for the assessment tasks. The
     difference between OBTL and traditional CRA
     is in how students’ grades are determined.
    Arriving at student’s final grade
 In OBTL, the results on particular assessment
  tasks are not combined to form a grade.
  Grading criteria are based on the ILOs, not the
  assessment tasks.
 Conversion to final grade from performance on
  the ILOs can be analytic or holistic
 If analytic, draw up a proforma, with a set of
  rubrics, telling you (and the student) why you
  graded the ILO as you did on the evidence.

   WORKSHOP: Developing grading criteria
An Analytic Method of deriving Final Grade based
     on CityU’s Existing Grading Scheme
                       Marginal         Adequate            Good                         Excellent          Student’s
                             D         C-     C     C+     B-      B         B+         A-      A     A+    Rating
             Grade-point      1.0     1.7     2.0   2.3     2.7     3.0      3.3        3.7     4.0   4.3
     ILO
1.   Able to reflect x3     barely      Aware of self        Aware of self,            Aware plus apply
     no theory                          no theory            can improve               original theory      ……..



2.   ILO 2        x2       - -- - - - - - - - Similar categories for ILO 1        .. . . .. .               ……..



2.   IloO 3 …….. x 1                        Etc …..                                                          ……..



6.   ILO 6. …        x1                                                                                      ………


7.                Final grade = ∑ individual ILO grade-point/ ∑weightings                                    =====


         Final grade converted into letter grade from the City U’s own table.
           A Holistic Example of Deriving Final Grade

                           Curriculum and Instruction
Grading will be based on your attaining the following ILOs

1. Demonstrate that you correctly understand and can apply the principles of good teaching
    and assessment to chosen contexts.
2. Demonstrate a knowledge of selected aspects of curriculum design and management
   and how they relate to the educational system in Hong Kong.
3. Show how the content and experiences in this course may enhance your effectiveness as
    an EP.
4. Show evidence of reflective decision-making.

Final grades will depend on how well you can demonstrate that you have met all the ILOs:
A   (70+): awarded if you have clearly met all the ILOs, displaying deep knowledge of the
    base content, original and creative thinking, perhaps going beyond established
    practice.
B   (60-69): awarded when all ILOs have been met very well and effectively.
C   (50-59): awarded when the ILOs have been addressed satisfactorily, or where the
    evidence is strong in some ILOs, weaker but acceptable in others.
F   Less than C, work plagiarised, not submitted.
         Assessment: Curriculum and Instruction
Show evidence that you have learned according to the criteria in
    the ILOs. Keep a reflective journal to record useful insights as
    you progress through the course. Use as a data base. The
    evidence will be presented in the following forms:
   A paper, drawing on principles of curriculum and good
    teaching, explaining how you would like to see the Hong Kong
    educational system implement any major educational reforms.
    You should have ILO 2 in mind.
   A report specifically addressing ILOs 3 and 4, a review of
    those aspects of the course that you think will probably
    enhance your work as an EP. This can refer both to your way
    of thinking about your role, as much as to actual skills. Your
    reflective journal will be an important source for this.
   Your own rationale of your group presentation, taking into
    account the evaluation made at the time of presentation. You
    should have ILO 1 in mind.
   A self-evaluation showing how you have addressed each of the
    objectives.
Place these in a portfolio, which will be graded as above. Take a
    total of 5,000 words as a guideline for the complete portfolio.
           Assessing and Grading the C&I Portfolio
   Name…..     (deleted)
   Paper Comments on teacher ed very insightful. Cure: reflective practice
    (you say this later, but isn’t quite clear at this point). Backwash from
    traditional assessment also insightful. More on ILO 1 than 2, but OK
   Report Chinese History activity sounds very familiar both in process
    and outcome. Your diagnosis is sharp and correct. Likewise homework
    policy. Your analysis reminds me that every activity the teacher endorses
    needs to show alignment. Discussion on assessment is frank ILOs 3 and
    4 addressed very well indeed
   Rationale of your group presentation Very good indeed on ILO 1,
    with some reflective application to induction of new teachers.
   Self-evaluation re ILOs Exemplary – very helpful to me as assessor.
   Final Grade. Covered all objectives, penetrating criticisms and high
    reflection – do they amount to creativity, originality? I think
    so            A- (73)
    Grading on outcomes rather than
        the assessment tasks…

   keeps teacher and students focused on what is to be learned
    rather than what is to be tested
   ―backwash‖ is therefore learning-oriented, not test- oriented
   provides much more useful summary of each student’s
    achievements—employers would love it!
   precluded the use of ―marking‖ and quantitative procedures
    such as averaging that can allow inadequate results in some
    tasks to be ignored.
   as grading is based on professional judgement, not averaging,
    grading easier and more flexible than traditional practice
   readily allows affective ILOs, where appropriate, to be assessed
    on same terms as cognitive
                  Summary
 Rarely need  more than 5 or 6 ILOs per course
 ILOs should not contain multiple verbs. The
  lower order ones are usually redundant anyway.
 The final grade is not derived directly from any
  one assessment task but on how well a student
  has met the ILOs.
 Course development is an iterative procedure. As
  you progress you will want to refine previous
  decisions about ILOs.
         EVALUATION OF OBTL
              PROJECT
   Action research – keep monitoring and building in
    improvements as research suggests.
   Advantages and Problems with OBTL
   A. The Student’s perspective
           learning outcomes
           learning activities, assessment preparation strategies
           perceptions
   B. The Teacher’s perspective
           difficulties, advantages, suggested improvements…
   C. Department’s perspective
           impact on teaching, learning
           concerns, future plans re implentation

				
DOCUMENT INFO