Learning Center
Plans & pricing Sign in
Sign Out



									                      ASSESSING GRADUATE ATTRIBUTES
                      IN THE BUSINESS LAW CURRICULUM

                                         PATTY KAMVOUNIAS
                                         DARRALL THOMPSON
In recent years, Australian universities have been requiring academics to include graduate
attributes in curriculum documentation. This ‘top-down’ approach can lead to ‘tick-box’
mapping exercises where learning goals are matched with attribute categories and
assessment processes can remain untouched, inevitably leading students to focus on marks
or grades. In these circumstances, assessment rarely provides feedback to students about
the progressive development of the very attributes universities claim to instil in their
    This article follows a research-based approach to a law teacher’s journey through
various attempts to implement a graduate attributes policy in a business law unit of study
offered to non-law degree students. The integration of graduate attributes with assessment
tasks and assessment criteria coded to attribute categories was facilitated through a process
involving software designed for this purpose.
    The strategies used and the lessons learned in this research are relevant for academics,
academic developers and academic leadership generally.

                                             I. BACKGROUND
                      A. Terminology and Definitions – Important Issues
For the past decade, Australian universities have been required to include graduate
attributes, generic skills, graduate capabilities, competencies and various other terms in the
quality assurance plans they submit to the Commonwealth government.1 These terms are
confusing and contested. For example, the word ‘generic’ implies independence from a
field of study; ‘skills’ is too narrow to embrace attitudes and values; and ‘competencies’
has been used as a tick list against specific skills.
    The term ‘graduate attributes’ seems now to be the most common and has been defined
as ‘the skills, personal attributes and values which should be acquired by all graduates
regardless of their discipline or field of study’.2 Australian Technology Network (ATN)
universities define graduate attributes as:
      … the qualities, skills and understandings a university community agrees its students
      would desirably develop during their time at the institution and, consequently, shape the
      contribution they are able to make to their profession and as a citizen.3
However, there are problems with both these definitions. The first, in using the phrase
‘regardless of their discipline or field of study’ may imply that graduate attributes are best
developed through separate units of study. However, this ‘bolted-on’ approach is not
supported by educational research.4 The second, in using the phrase ‘would desirably
develop during their time at the institution’ seems to let universities off the hook in regard
to any accountability with respect to their involvement in the development of graduate
attributes. The view that students will gain attributes by some kind of osmosis is clearly

      Faculty of Economics and Business, University of Sydney.
      Director of Teaching and Learning, Faculty of Design, Architecture and Building, University of Technology, Sydney.
  1   Department of Education, Science and Training, Striving For Quality: Learning, Teaching and Scholarship (2002) 22.
  2   Higher Education Council Australia, Achieving Quality (1992) 20.
  3   John Bowden, Gail Hart, Bruce King, Keith Trigwell and Owen Watts, ‘Graduate Attributes and Generic
      Capabilities’, Generic Capabilities of ATN University Graduates (2000)
      <> at 1 December 2008.
  4   Simon C Barrie, ‘A Research-based Approach to Generic Graduate Attributes Policy’ (2004) 23(3) Higher Education
      Research and Development 265.


unacceptable as governments, accrediting bodies, professions and society all bring pressure
to bear on a higher education sector5 in which:
      [m]ore than ever before universities are being relied upon as a vehicle for the advancement
      of both the national economy and wider society. They do this through the creation of new
      knowledge and by preparing graduates with appropriate skills and attributes. It makes
      sense, then, for them to maintain a focus on keeping graduate capabilities in line with the
      needs of the economy and society.6
The term ‘graduate capabilities’ mentioned in this quote could be used instead of ‘graduate
attributes’ in an educational context. However, with both terms, care must be taken in the
sense that capability (to do) tends to imply ability in multiple contexts. In an educational
setting, assessment can really only identify ‘ability to do’ as evident in work presented in
any given assessment task. It is an attribute of a student’s work rather than a judgment
about them. In this important distinction between the student and their work, the term
‘attribute’ is a little less problematic than ‘capability’.
    The term ‘graduate attributes’ used in this article is intended to include a very broad
range of personal and professional qualities and skills, together with the ability to
understand and apply discipline-based knowledge.
    Having considered the subtlety of terminology and definitions, the question then
emerges: how do we achieve consensus about these terms in a university community?

             B. Achieving Consensus about Graduate Attribute Frameworks
In an ideal world, a university community should agree on what constitutes the attributes
of its graduates. However, the reality is that such understandings often remain implicit 7
and, even when made explicit, individual academics have quite different views of what
graduate attributes are and how they can (or cannot) be integrated into the curriculum.8
These different understandings can cause deep divisions in a higher educational climate in
which quality assurance predominates and assurance of learning is required by accrediting
bodies. This climate contrasts with the traditional approach in which university teachers
devised their own intended learning outcomes and determined how they would be
communicated to their students.9
    For a university community to achieve consensus, it is helpful to consider the different
institutional levels at which graduate attributes can be conceptualised: university, faculty,
school, department, program of study (or degree) and specific units (or subjects) within
which students will be expected to develop these.
    At the University of Sydney, for example, three ‘overarching graduate attributes’ have
been identified, namely scholarship (students’ attitude or stance towards knowledge),
global citizenship (students’ attitude or stance towards the world), and lifelong learning
(students’ attitude or stance towards themselves).10 To be workable at lower levels, the
following more specific set of attributes have been articulated: (1) research and inquiry; (2)
information literacy; (3) personal and intellectual autonomy; (4) ethical, social and

  5 See, eg, Sue Cranmer, ‘Enhancing Graduate Employability: Best Intentions and Mixed Outcomes’ (2006) 31(2)
     Studies in Higher Education 169; Barbara de la Harpe, Alex Radloff and John Wyber, ‘Quality and Generic
     (Professional) Skills’ (2000) 6(3) Quality in Higher Education 231; Cassandra Star and Sara Hammer, ‘Teaching
     Generic Skills: Eroding the Higher Purpose of Universities or an Opportunity for Renewal?’ (2008) 34(2) Oxford
     Review of Education 237.
  6 Department of Education, Science and Training, Employability Skills for the Future (2002) 25.
  7 Bowden et al, above n 3.
  8 Barrie, above n 4.
  9 See, eg, Debra Bath, Calvin Smith, Sarah Stein and Richard Swann, ‘Beyond Mapping and Embedding Graduate
     Attributes: Bringing Together Quality Assurance and Action Learning to Create a Validated and Living Curriculum’
     (2004) 23(3) Higher Education Research and Development 313; Jennifer Sumsion and Joy Goodfellow, ‘Identifying
     Generic Skills Through Curriculum Mapping: A Critical Evaluation’ (2004) 23(3) Higher Education Research and
     Development 329.
  10 Barrie, above n 4, 269.


professional understanding; and (5) communication.11 To reflect disciplinary differences,
faculties have been encouraged to translate these graduate attributes into a set of more
specific learning goals and outcomes relevant to the degree programs offered. For
example, the Faculty of Economics and Business has the following as a learning outcome
for research and inquiry: ‘apply economic, political, legal, commercial and business
theories and concepts to problems and practice.’ The translation of graduate attributes into
the language of the discipline reflects an approach in which graduate attributes are seen as
embedded rather than distinct from disciplinary knowledge.12
   However, even with strong top-down directives and support from academic
development units, boxes can too easily be ticked without change occurring in assessment
and feedback to students.

                          C. The Context and Methodology of this Study
Developing graduate attributes has become a key focal point in professional disciplines,
including law.13 Consequently, legal academics are now grappling with the task of
embedding graduate attributes in their courses,14 and learning and teaching and curriculum
development in law schools has become the subject of a number of national initiatives.15
    Learning and teaching in law is not limited to law schools teaching law degree
students.16 It also includes teaching law to non-law degree students. This article discusses
the inclusion of graduate attributes in Trade Practices and Consumer Law (‘CLAW2205’),
an elective offered by the Faculty of Economics and Business (the ‘Faculty’) at the
University of Sydney to commerce and economics degree students.
    The methodology used in this article is that of case-study methodology17 but is written
up as the ‘journey’ of one law teacher attempting to implement not only her own teaching
and learning values, but also the university’s requirements and directives in this area. Her
journey began with her concerns regarding the way the university graduate attributes
policy was initially implemented within the Faculty. These concerns were articulated both
within and beyond the university.18 In collaboration with a Faculty colleague, a model for
influencing teaching and learning culture in universities was developed.19 The journey
might have ended there. However, the law teacher (the first author of this article) then
heard about a successful graduate attributes project at the Faculty of Design, Architecture
and Building at the University of Technology, Sydney, using an online criteria-based
assessment system known as ReView.20 ReView is described in Section A of Part III of

  11 See University of Sydney Institute for Teaching and Learning, Graduate Attributes Project
     <> at 1 December 2008.
  12 Barrie, above n 4.
  13 See, eg, Sally Kift, ‘Lawyering Skills: Finding Their Place in Legal Education’ (1997) 8 Legal Education Review 43;
     Elisabeth Peden and Joellen Riley, ‘Law Graduates’ Skills — A Pilot Study into Employers’ Perspectives’ (2005) 15
     Legal Education Review 87.
  14 See: Sharon Christensen and Sally Kift, ‘Graduate Attributes and Legal Skills: Integration or Disintegration?’ (2000)
     11 Legal Education Review 207; Sharon Christensen and Natalie Cuffe, ‘Embedding Graduate Attributes in Law —
     Why, How and Is It Working?’(Paper presented at the 2nd International Lifelong and Learning Conference,
     Rockhampton, 2002)
     <> at 1 December 2008.
  15 See, eg, Richard Johnstone and Sumitra Vignaendra, Learning Outcomes and Curriculum Development in Law: A
     Report Commissioned by the Australian Universities Teaching Committee (AUTC) (2003)
     < > at 1 December 2008; and Australian
     Learning and Teaching Council (ALTC), Discipline Based Initiative, Learning and Teaching in the Discipline of
     Law: Achieving and Sustaining Excellence in a Changed and Changing Environment (2006)
     y_2007.pdf> at 1 December 2008.
  16 As clearly indicated by the existence of a large and active Law for Non-Law Students (LNLS) Interest Group in the
     Australasian Law Teachers’ Association (ALTA). See the ALTA website <> at 1
     December 2008.
  17 Norman K Denzin and Yvonna S Lincoln (eds), Strategies of Qualitative Inquiry (2nd ed, 2003).
  18 See Arlene Harvey and Patty Kamvounias, ‘Bridging the Implementation Gap: A Teacher-as-Learner Approach to
     Teaching and Learning Policy’ (2008) 27(1) Higher Education Research and Development 31.
  19 Ibid.
  20 Developed at the University of Technology, Sydney. Information available through Re:View: Online Criteria-based
     Assessment <> at 1 December 2008.


this article. The law teacher and the designer of ReView (the second author of this article)
met and the law teacher’s graduate attributes journey continued with a new collaboration.
    The process of implementing ReView in one business law subject is described in
Section B of Part III.21 Discussions between the authors in the context of using ReView,
led to the constructive alignment22 of assessment tasks with explicit assessment criteria
coded to attribute categories. Student feedback, both quantitative and qualitative, was
available in the form of responses to the standard Unit of Study Evaluation (USE) forms. 23
This was analysed to determine student views on the implementation of graduate-attribute-
coded criteria feedback through ReView and to identify directions for future research.
    The formative journey of one law teacher’s attempts to respond to graduate attribute
policy initiatives in practical ways is described in the next section.

                  A. First Attempt to Implement Graduate Attributes Policy
Although general statements about graduate attributes appear on university websites and in
documents such as Faculty handbooks and promotional materials, it is in specific unit or
subject outlines that the relevant graduate attributes and learning goals are articulated and
communicated to students and others. In early 2003, the Faculty identified 25 learning
goals that related to the university’s five categories of graduate attributes and mandated the
use of a unit outline template that, inter alia, required teachers to select up to six of these
learning goals and link these to the assessments in their units.24 This approach was
problematic for units such as CLAW2205 because the Faculty list did not include all the
first author’s goals for her unit. Another problem was that although the learning goals were
linked to assessment tasks, they were not linked to the assessment criteria.25 The first
author’s uneasiness about the lack of alignment26 between the learning goals, graduate
attributes and assessment criteria and dissatisfaction with the implementation of the
graduate attributes policy were raised with the Faculty and have been fully discussed in a
previously published article.27

             B. The Current Faculty Approach to Graduate Attributes Policy
In October 2003, the Office of Learning and Teaching in Economics and Business
(OLTEB) was established to provide learning and teaching support for both students and
academics in the Faculty.28 Having demonstrated a commitment to quality learning and
teaching, and a willingness to act as a conduit between OLTEB and disciplinary
colleagues, the first author was appointed the first Learning and Teaching Associate for the

  21 Following this and other pilot studies, ReView is being used as the basis for an Australian Learning and Teaching
     Council (formerly Carrick Institute for Teaching and Learning) Priority Project entitled Facilitating Staff and Student
     Engagement With Graduate Attribute Development, Assessment and Standards in Business Faculties Australian
     Learning and Teaching Council,
     y_2007.pdf> at 1 December 2008.
  22 For a discussion of a ‘constructive alignment’ approach to teaching practice, see generally John Biggs, Teaching for
     Quality Learning at University (2nd ed, 2003).
  23 See the University of Sydney Institute for Teaching and Learning, About the Unit of Study Evaluation System (USE)
     <> at 1 December 2008.
  24 For further details, see Harvey and Kamvounias, above n 18, 35-37.
  25 For a discussion of the link between assessment criteria and learning in law, see: Kelley Burton and Natalie Cuffe,
     ‘The Design and Implementation of Criterion-Referenced Assessment in a First Year Undergraduate Core Law
     Unit’(2005) 15 Legal Education Review 159; Richard Johnstone, Jenny Patterson and Kim Rubenstein, ‘Improving
     Criteria and Feedback in Student Assessment in Law’ (1996) 7 Legal Education Review 267; Nicolette Rogers,
     ‘Improving the Quality of Learning in Law Schools by Improving Student Assessment’ (1993) 4 Legal Education
     Review 113.
  26 Biggs, above n 22.
  27 Harvey and Kamvounias, above n 18.
  28 See University of Sydney, Office of Learning and Teaching in Economics and Business
     <> at 1 December 2008.


Discipline of Business Law.29 The second author was invited by the Faculty’s Associate
Dean (Learning and Teaching) to give a presentation to the Faculty’s Learning and
Teaching associates to introduce ReView to academics in economics and business-related
disciplines. The first author saw the potential for a new ‘deep’ approach to implementing
graduate attributes policy using ReView.30 She therefore enthusiastically agreed to
participate in the semester one, 2007, pilot of ReView in the Faculty along with teachers
from the disciplines of Government and Political Economy.31 In the extended 2008
ReView project, other disciplines within the Faculty and other business faculties at other
universities were included.32 The business law unit that was to use ReView in 2007 and
2008 was CLAW2205, a senior elective with enrolments of about 60 students in each

                      C. Conversations, Reflections and Implementation
About three months before the start of semester one, 2007, and the use of ReView by the
first author, the authors engaged in conversations and reflections about constructive
alignment and the importance of writing explicit assessment criteria for business law units.
     During a series of meetings arranged through the OLTEB, the second author introduced
the first author to four basic concepts that underpinned the design of the ReView system
and its process of implementation. The first concept relates to the importance of
assessment criteria. The anecdotal evidence from university teachers is that students pay
little attention to learning goals simply listed in unit of study outlines. However, when
learning goals and assessment criteria are linked, the assessment criteria have a crucial role
in any attempt to embed graduate attributes within the curriculum. In the view of the
second author, assessment criteria become an important ‘fulcrum of engagement’33 for
both teachers and students. Teachers should therefore be encouraged to develop clear and
explicit wording for assessment criteria and students should be encouraged to self-assess
against these criteria.34 The second concept relates to the reality that university academics
often spend a great deal of time considering the teaching aspects of their work. It is
therefore essential that in the development and refinement of explicit criteria linked to
relevant attribute categories, teachers’ aims and views be valued and their experiences
respected. The third concept relates to the fact that all assessment activities contribute to
the development of attributes, even exams. Therefore all types of assessments can be
marked using criteria and all assessment criteria can be identified or designated as
contributing to the development of a range of attributes. Lastly, whilst it is generally
agreed that ‘assessment is the most powerful influence on student learning in formal
courses’,35 this idea can be seriously misinterpreted.36 It should not necessarily lead to
more tests and exams that focus students’ attention on marks or grades. Instead, it should
encourage university teachers to value the development of attributes enough to reference
them in assessment criteria. In doing this it is more likely students will see attributes as an
important aspect of their learning and may have the effect of reducing their focus on marks
and grades.

  29 See University of Sydney, Faculty of Economics, L&T Associates: Learning and Teaching Associates Network
     <> at 1 December 2008.
  30 Harvey and Kamvounias, above n 18, 38-40.
  31 For a discussion of the outcomes of the pilot study, see: Darrall Thompson, Lesley Treleaven, Patty Kamvounias,
     Betsi Beem and Elizabeth Hill, ‘Integrating Graduate Attributes with Assessment Criteria in Business Education
     Using an Online Assessment System’ (2008) 5(1) Journal of University Teaching and Learning Practice 34.
  32 ALTC Priority Project, Facilitating Staff and Student Engagement, above n 21.
  33 For a discussion of the importance of developing explicit assessment criteria using ReView in previous studies see:
     Darrall Thompson, ‘Integrating Graduate Attributes with Student Self Assessment’, Robert Zehner and Carl Reidsema
     (eds), Proceedings of ConnectED International Conference on Design Education, University of New South Wales,
     Sydney (2007) 1.
  34 This is easily enabled in ReView.
  35 David Boud, Ruth Cohen and Jane Sampson (eds), Peer Learning in Higher Education (2001).
  36 John C McLachlan, ‘The Relationship Between Assessment and Learning’ (2006) 40(8) Medical Education 716–717


    The lively conversations around these concepts elicited many examples from the
classroom. Given broad agreement between the authors about these ideas, it was decided
that ReView would be used in CLAW2205 using a ‘bottom-up’ approach to graduate
attribute development, beginning with the assessment criteria for all assessment tasks.
    As a starting point, the following questions were asked of the first author in the context
of the intended learning outcomes of CLAW2205 and the broad graduate attribute
categories that had been documented by the university:
      What skills do you want students to develop, what knowledge do you want them to
      construct and what qualities do you want them to acquire as a result of their engagement
      with this particular assessment task you have designed?
Not surprisingly, writing assessment criteria that explicitly describe both the intended
range and the level of students’ performance proved to be a complex task. It was therefore
helpful for the second author to provide examples of language used in describing aspects
of students’ work, such as clarity, thoroughness, accuracy, depth, appropriateness,
professionalism and ethical approaches. The suggestion that parts of the task, and the
knowledge and concepts, could be referenced in criteria settled concerns that subject
matter or content would be lost if graduate attribute development became the focus of
    In order to clarify the relationship between aspects of this ‘learning design’, the first
author developed a chart for CLAW2205 that aligned intended learning outcomes with
graduate attributes, teaching and learning activities, assessment tasks and assessment
criteria .37 The articulation and refinement of assessment criteria was soon found to be an
ongoing process. This can be seen by the subtle yet important differences in the statement
of criteria in the initial chart and in the chart appearing in the 2008 unit outline set out
below (Table A). For example, there is an indication in the 2008 chart that some criteria
are for self-assessment purposes only, but acknowledged as an important part of
assessment and enabled by the ReView online system.
    Having identified and refined the assessment criteria for all CLAW2205 assessments
and linked these to intended learning outcomes and graduate attributes, the first author then
proceeded to implement ReView. The next section describes ReView and the
implementation process.

  37 The initial chart is available on the Faculty’s website as an example of ‘best practice’. University of Sydney, Office of
     Learning and Teaching
     <> at 1 December 2008.

         Table A – Chart showing constructive alignment of the subject with explicit criteria for each task coded to attribute categories (headings also apply to the
         second page of the chart).

      CLAW2205               University of         CLAW2205         CLAW2205
      Intended               Sydney                Student Learning Assessment Tasks
      Learning               Graduate              Activities       & Assessment
      Outcomes               Attributes                             Criteria
      Upon successful                                               Participation and Group research              Oral presentation   Quiz                   Individual
      completion of this                                            engagement        paper                       15%                 15%                    Assignment
      unit of study,                                                15%               15%                                                                    30%
      students should be
      able to:
      1. Identify and        Research and          Read text and other * Clear               * Quality of         *Thoroughness of    * Accurate             * Clear
      analyse legal issues   Inquiry (R&I):        materials.          identification of     analysis and focus   research and        identification and     identification of
      with respect to        Graduates of the                          legal issues in the   in response to the   preparation as      application of         legal issue(s) in the
      restrictive trade      Faculty of            Make own notes      tutorial discussion   research topic.      evident in the      relevant law to test   news item.
      practices and          Economics and         and summaries       questions and         * Quality of         presentation.       questions.             * Thoughtful
      consumer               Business will be      before lectures and thoughtful analysis   synthesis of legal                                              analysis and
      protection law         able to create new    tutorials.          and application of    materials and                                                   application of the
      arising from given     knowledge and                             the relevant law.     information                                                     relevant law to the
      fact situations and    understanding         Attend lectures and                       relevant to the                                                 news item.
      real-world             through the process   tutorials.                                research topic.                                                 * Consistent
      contexts.              of research and                                                 * Consistent                                                    support of written
      2. Resolve             inquiry.              Participate in                            support of written                                              statements with
      problems by                                  tutorial                                  statements with                                                 appropriate legal
      applying the                                 discussions.                              appropriate legal                                               authorities.
      relevant law,                                                                          authorities.
      evaluating the
      possible solutions
      and developing
      arguments to
                                                                                                                                                                                     ASSESSING GRADUATE ATTRIBUTES IN THE BUSINESS LAW CURRICULUM

      3. Demonstrate          Information              Access materials                               * Ability to source                         * Efficient sourcing   * Ability to source
      legal research skills   Literacy (IL):           provided online and                            a range of legal                            of information from    current news item
      by locating and         Graduates of the         in library for this unit.                      materials relevant                          own structured         that deals with the
      selecting legal         Faculty of                                                              to the research                             notes collated from    law on restrictive
      materials using         Economics and            Undertake own                                  topic.                                      lectures,              trade practices
      libraries, the web      Business will be able    research.                                      * Consistent use of                         tutorials and          studied in this unit.
      and other sources of    to use information                                                      appropriate method                          readings               * Consistent use of
                              effectively in a range   Evaluate the                                   of legal citation and                       (self-assessment       appropriate method
      legal information.
                              of contexts.             usefulness of                                  referencing.                                only).                 of legal citation and
      4. Manage, analyse,
                                                       information found.                                                                                                referencing.
      evaluate and use
      legal materials and                              Draft/edit/finalise
      information.                                     written work for
      5. Communicate          Communication            Participate in tutorial * Clear and concise    * Clear and concise     * Clear and                                * Clear and concise
      about the law,          (C):                     discussions.            oral articulation of   articulation of ideas   coherent oral                              articulation of ideas
      orally and in           Graduates of the                                 ideas and responses    about the research      articulation of                            about the news item
      writing, to a           Faculty of               Contribute to online    during tutorial        topic in writing,       ideas about the                            in writing, using
      professional            Economics and            discussion forums.      discussions, using     using appropriate       news item using                            appropriate legal
      standard.               Business will                                    appropriate legal      legal language.         appropriate legal                          language.
                              recognise and value      Plan and rehearse oral language.                                       language.
                              communication as a       presentation.           * Effective            * Careful                                                          * Careful
                              tool for negotiating                             interaction with       organisation of         * Quality of                               organisation of
                                                       Draft/edit/finalise                            ideas so that           engagement with                            ideas so that
                              and creating new         individual              peers and tutor in
                              understanding,                                   relation to            argument develops       peers.                                     argument develops
                                                       assignment.                                    logically through                                                  logically through
                              interacting with                                 alternative ideas
                                                                                                                                                                                                 JOURNAL OF THE AUSTRALASIAN LAW TEACHERS ASSOCIATION

                              others, and furthering                           and legal arguments    the paper.              * Appropriate use                          the paper.
                                                       Contribute to the                                                      of written and/or
                              their own learning.      drafting/editing/       during tutorial                                                                           * Correctness of
                                                                                                      * Correctness of        visual aids.                               grammar, spelling,
                                                       finalising of the group discussions.           grammar, spelling,
                                                       research paper.                                                                                                   etc.
      6. Plan and achieve   Personal and          Prepare for lectures * Consistency of       * Management of      * Appropriate use     * Efficient use of    * Management of
      goals and meet        Intellectual          and tutorials.       participation in       own workload to      of the time allowed   time to answer test   own workload to
      new challenges and    Autonomy                                   tutorial discussions   meet submission      for the oral          questions under       meet submission
      deadlines.            (P&IA):               Prepare for test and over the course of     deadline             presentation.         exam conditions       deadline (self–
                            Graduates of the      oral presentation.   the semester.          (self -assessment                          (self–assessment      assessment only).
                            Faculty of            Submit all                                  only).                                     only).
                            Economics and                            * Thoughtfulness
                                                  assessments by due of approach to self-
                            Business will be      date.
                            able to work                             assessment in all
                            independently and                        CLAW2205
                            sustainably in a                         assessments.
                            way that is
                            informed by
                            openness, curiosity
                            and a desire to
                            meet new
      7. Work with          Ethical, Social       Attend lectures and   * Respectful          * Respectful and     * Professional        * Adherence to        * Adherence to
      people from           and Professional      tutorials.            interaction with      professional         approach to oral      principles of         principles of
      diverse               Understanding         Participate in        peers and tutor       interaction with     presentation          academic honesty      academic honesty
      backgrounds with      (ES&PU):              tutorial              during tutorial       group members        arrangements          (self-assessment      (self-assessment
      inclusiveness,        Graduates of the      discussions.          discussion.           during group         (including            only).                only).
      openmindedness        Faculty of                                                        activities           submitting news
      and integrity, and    Economics and         Work                  * Professional        (peer assessment).   item on time with
      manage the            Business will hold    cooperatively with    approach to                                appropriate
                                                                                              * Quality and
      dynamics of           personal values       group members in      tutorial                                   referencing and
                                                                                              extent of
      working within a      and beliefs           and out of class.     attendance.                                adhering to
                                                                                              contribution to
      team.                 consistent with                                                   group activities     presentation
      8. Appreciate the     their role as         Complete/review                                                  schedule).
                                                  Faculty of E&B                              (peer assessment).
      ethical and           responsible
                                                  online academic                             * Adherence to
      professional          members of local,
                                                  honesty module.                             principles of
      dimensions of their   national,
                                                                                              academic honesty
      conduct within and    international and
      beyond the            professional
      classroom.            communities.
                                                                                                                                                                                   ASSESSING GRADUATE ATTRIBUTES IN THE BUSINESS LAW CURRICULUM

                                   A. A Brief Description of ReView
ReView is essentially a web-based automated marking sheet for the criteria-based
assessment of student work. University or Faculty-level graduate attribute categories 38 are
entered on ReView and teachers then enter their assessment criteria, taking care to match
each criterion for each assessment task with the relevant attribute. Colour and symbol
coding for each graduate attribute category (as shown in Figure 1) makes this alignment
process clear to students and users.

   Figure 1 — Screenshot of the criteria coding section where the dropdown menu
   allows the selection of attribute groups to code all criteria with a colour code and

Teachers then use the vertical bars on the ‘data-sliders’ shown to the right in Figure 2 to
assess each criterion relating to each assessment task (indicated by the vertical black lines
in Figure 2). After teachers’ marks are saved, the students’ self-assessments appear against
all criteria (indicated by the light blue triangles in Figure 2).

   Figure 2 — Academics’ marking screen: Students are selected on the list to the left
   and then the colour-coded criteria are referred to whilst sliding the vertical bars to
   generate percentage marks for criteria. Once marks are saved the students’ self-
   assessments appear on the top edge of each data slider.

When teachers finish marking and decide to publish their assessments, students see a much
simpler screen (Figure 3) that does not show the actual marks, but rather broad grey sliders

  38 University of Sydney Institute for Teaching and Learning, above n 11.

to indicate their performance against the criteria in terms of grades only.39 Students also
see their own self-assessment indicated by the light blue triangles at the top of each data
slider as here in Figure 3.

   Figure 3 — Student feedback and self-assessment screen: Students can select each
   of their assignments on the list to the left and then the colour-coded criteria are
   referred to whilst sliding the triangles on the data sliders to self-assess. The grey
   bars on the sliders indicate the tutor’s grading compared to their own.

When criteria for all assessment tasks are entered on ReView, a pie chart and bar chart are
generated showing the attributes developed and assessed in the particular unit of study.
Figure 4 indicates this for CLAW2205.

                                                                                Figure 4 — View of criteria
                                                                                weighting for the complete Unit of
                                                                                Study against attribute categories:
                                                                                Personal      and      Intellectual
                                                                                Autonomy (green), Research and
                                                                                Inquiry    (white),   Information
                                                                                Literacy (red), Communication
                                                                                (yellow) and Ethical, Social and
                                                                                Professional        Understanding

                               B. Discussion of Implementation Issues
The word ‘implementation’ implies a simple and mechanistic application of predetermined
goals. This may be true in some contexts, but in the case of educational environments the
inertia against change can be enormous, and in this pilot project the students demonstrated
a fairly conservative reaction. For example, the authors agreed that students’ self-

  39 On the ‘data-sliders’ shown to the right in Figures 2 and 3, F indicates a fail grade (0-49 marks); P indicates a pass
     grade (50-64 marks); C indicates a credit grade (65-74 marks); D indicates a distinction grade (75- 84 marks) and HD
     indicates a high distinction grade (85-100 marks).

assessment against criteria was important for their learning as reflective practitioners. 40
Whilst the authors were keen to use ReView to promote self-assessment to the students for
their own educational benefit (as shown in Figures 2 and 3), it was clear from student
feedback that not all students were convinced of the value of self-assessment. Although
data on the number of students who self-assessed via ReView is yet to be analysed, a small
number of students had negative responses as follows:
      ReView was a waste of time;
      Over emphasis of ReView and self assessment;
      I wasn’t aware that participation in ReView would affect our participation mark. Would
      have been good to have been informed of this.41
Another example of student conservatism arose with respect to the way feedback was
given by ReView. The idea that percentage marks would not be displayed was challenging
for all concerned. However, the authors agreed that a shift towards viewing criteria-based
feedback without a percentage mark could be beneficial to students understanding in
CLAW2205. The fact that CLAW2205 students were only facing these changes in one of
their subjects made it difficult to convince them about benefits to their learning. The
following student responses were typical:
      The ReView system was useful but I still prefer marks against a set of criteria;
      I prefer a ‘numerical mark’ on Blackboard for my assessment instead of a ‘letter’;
      ReView was a bit misleading and deceptive.
The technical issues in implementation were also problematic. As this was a pilot scheme,
no link with central university systems had been established. Students and staff therefore
had to have different login usernames and passwords for this system. When students
initially enrolled in or withdrew from CLAW2205, there was no automatic update of the
class list on the assessment screen. When marks were required to be exported to the
incumbent Blackboard system Gradebook, ideally a macro excel spreadsheet was needed
to handle the upload. These issues, together with all the other usual issues that accompany
the use of technology, meant that the first author often asked herself why she agreed to
participate in this pilot. These technical issues were also of concern to students, some of
whom commented as follows:
      I found the feedback for assessments adequate but found the ReView system too
      complicated and fiddly to be effective;
      ReView needs to be tweaked a bit;
      Blackboard was a useful tool. I would like to have seen the ‘view grades’ sections used
      and updated throughout the course in addition to ReView.

  40 On the importance of self-reflection as a learning tool see, eg, David Boud, Enhancing Learning Through Self-
     Assessment (1995); David Boud, ‘Sustainable Assessment: Rethinking Assessment for the Learning Society’ (2000)
     22(2) Studies in Continuing Education 151; and David J Nicol and Debra Macfarlane-Dick, ‘Formative Assessment
     and Self-Regulated Learning: a Model and Seven Principles of Good Feedback Practice’ (2006) 31(2) Studies in
     Higher Education 199. On self-reflection and legal education, see, eg, Kathy Mack, Gerry Mullins, Jan Sidford and
     David Bamford, ‘Developing Student Self-Reflection Skills Through Interviewing and Negotiation Exercises in Legal
     Education’ (2002) 13 Legal Education Review 221; and Judith McNamara, Rachel Field and Natalie A Cuffe,
     ‘Designing Reflective Assessment for Effective Learning of Legal Research Skills in First Year’ (Paper presented at
     the First Year in Higher Education Conference, Hobart, Tasmania, July 2008).
  41 Students were informed of this in the unit outline. See Table A and the assessment criteria for ‘Participation and
     Engagement’ that includes ‘thoughtfulness of approach to self-assessment in all CLAW2205 assessments.’

                                 IV. RESEARCH OUTCOMES
   ReView has now been used in two semesters for CLAW2205 (2007 and 2008). Students
   have not been surveyed specifically on their responses to ReView but the standard end of
   semester Unit of Study Evaluation (USE) surveys have elicited some useful data. Student
   feedback was available in both quantitative and qualitative form each time ReView was
   used and compared to that provided in the year preceding the introduction of ReView
   (2006). The number of students enrolled in CLAW2205 in each of the three years was
   about 60 and it should be noted that the USE scores in CLAW2205 were already
   historically high for many of the questions asked.
       Table B indicates the percentage of students in 2006, 2007 and 2008 that agreed or
   strongly agreed with USE questions relevant to assessment processes.
      Table B – Chart showing the percentages of students to agree or strongly agree
      with standard Unit of Study Evaluation questions. (Scores in the 80-90% range
      are considered to be high ratings).

Faculty of Economics and Business                                 2006      2007        2008
Unit of Study Evaluation (USE)
CLAW2205 Trade Practices and                                      Pre-      ReView      ReView
Consumer Law                                                      ReView    Pilot       as part of
Q1. The learning outcomes and expected          strongly/agree:   97.5%     90.9%       90.6%
standards of this unit of study were clear to
Q3. This unit of study helped me develop        strongly/agree:   84.6%     81.8%       88.9%
valuable graduate attributes
(eg, research inquiry skills, communication
skills, personal intellectual autonomy,
ethical, social and professional
understandings, information literacy, etc).
Q8. Feedback on assessment assisted my          strongly/agree:   56.4%     73.8%       80%
learning in this unit of study.
Q10. Online learning (eg, with                  strongly/agree:   70.3%     79%         90.9%
Blackboard) supported my learning in this
unit of study.
Q12. Overall I was satisfied with the           strongly/agree:   97.4%     92.9%       94.4%
quality of this unit of study.

   It should be noted that the data was provided by three different groups of students, so care
   needs to be exercised when making comparison between these scores. CLAW2205 was
   also not exactly the same each year with regard to the configuration of assessment tasks
   and the instructions given to students about them. However, it is worth noting that the three
   questions (3, 8 and 10) relating specifically to attributes, feedback on assessment and
   online learning all show some improvement from 2006 to 2008.

   The USE survey questions do not refer expressly to ReView, so student written
comments about ReView on the survey forms were entirely unsolicited.42 As CLAW2205
was the only unit of study in which the students would have used ReView, it is interesting
that they mentioned it at all. In the 2007 feedback, there were 10 comments regarding the
use of ReView; in 2008 there were 21 comments, indicating perhaps that students were
more aware of ReView in its second iteration. In 2007, the comments were almost evenly
divided between positive (4) and negative (6), whereas in 2008, the comments were
overwhelmingly positive (15).
   When asked to comment on whether learning outcomes and standards were clear to
them (Q1), student responses that referred to ReView were as follows:
      Extremely so. The learning outcomes were emphasised thoroughly before all assessments
      and the ReView system emphasised them also;
      The use of ReView clearly demonstrated learning outcomes/graduate attributes and these
      were reinforced by [lecturer];
      ReView told me, even though @ first I didn’t want to use it.
It is interesting to note that all comments on this point were positive and it would appear
that the assessment criteria were identified by students as being descriptive of the learning
    Also all positive were student comments about whether CLAW2205 helped them
develop graduate attributes (Q3): ‘Student ReView was a good example’; ‘ReView made
me more aware’; and ‘This was effectively shown through the “ReView” online program.’
    Student views on whether feedback on assessments assisted their learning (Q8) were
mixed. In addition to the negative comments about the availability of grades only and the
problems with the software referred to in Part B of Section III, the following were typical
of the positive responses:
      Yes ReView helped me improve on weaker areas.
      ReView was great!! (smiley face).
      Good computer feedback system.
      ReView and good detailed comments on assessments.
      ReView had a nice rating system that covered multiple factors.
As discussed above, there were some technical problems with the interaction of
Blackboard and ReView. Nonetheless, when asked whether online learning supported
learning in CLAW2205 (Q10), a number of students specifically referred to ReView in the
following terms indicating that they see engagement in self-assessment and assessment for
each graduate attribute as part of their learning: ‘Really liked online ReView’; ‘Use of
ReView was good’; ‘ReView was very helpful.’

As a result of the process that both authors experienced in this case study or ‘journey’,
there are a number of important reflections to take us forward in further research.
   Students had mixed reactions as to whether feedback on assessment using ReView
assisted their learning. Their concerns about the availability of grades only on ReView
have prompted further refinement and implementation. For example, in future semesters,
marks will be published a few days after the publication of criteria-based feedback on
   The basic idea underlying ReView, namely that assessment criteria are the key to
embedding graduate attributes within the curriculum, is clearly an important ‘fulcrum of

  42 It is interesting to compare similar data from other units of study involved in the 2007 pilot study. See Thompson et
     al, above n 31.

engagement’ for both teachers and students. Assessment criteria therefore need to be
relevant, explicit and effectively communicated. It is also clear that the development of
criteria should be viewed as an ongoing work in progress, as this has been a significant part
of the journey for the first author. A database of criteria for different types of assessments
would be a useful resource for teachers when formulating criteria specific to an individual
assessment task.
    University teachers also need time and opportunities for meaningful discussions to
allow them to become clear about the concepts involved in a shift from content delivery to
the development of students’ attributes. In a high-pressure university environment where
research is given precedence, academic development and other support for teachers is vital
in facilitating the integration of graduate attributes into curricula. Whilst this paper refers
to a ‘bottom-up’ approach based at assessment task level, it is clear that this would be
futile without whole-institution multi-level leadership regarding graduate attribute
    Implementing new technology to assist learning and teaching takes time, and both
students and teachers need support and clear explanations of why the technology will be
useful. ReView assisted with the embedding of graduate attributes in CLAW2205 but
could similar results have been achieved without it? Certainly, the first author’s journey
could have ended with the development of the chart that aligned intended learning
outcomes with graduate attributes, teaching and learning activities, assessment tasks and
assessment criteria (Table A). But then, how would students know about their progressive
development of graduate attributes and how would teachers be able to evidence and assure
their students’ learning? ReView enables this easily and directly. ReView certainly acted
as a catalyst to conversations, reflections and implementation stages in this study. It also
facilitated student engagement with the curriculum through online self-assessment and
delivery of feedback on student assessment.
    In regard to further study, two areas emerge. The first relates to benchmarking,
standards and mutual understanding of grade descriptors.43 ReView could potentially be
used to engage lecturing staff, tutors and students with these issues, and particularly the
standards required at different levels or years of study. The second concerns student self-
assessment and how students can be encouraged, or perhaps even rewarded, for their
engagement with this feature as an important attribute for lifelong learning and reflective

  43 See, eg, Berry O’Donovan, Margaret Price and Chris Rust, ‘Developing Student Understanding of Assessment
     Standards: A Nested Hierarchy of Approaches’ (2008) 13(2) Teaching in Higher Education 205.
  44 CLAW2205 students’ experiences with self-assessment using ReView will be discussed and analysed in a future


To top