Carrick Institute Online Grant Application by xiuliliaofz


									                            COMPETITIVE GRANTS SCHEME 2011
                            APPLICATION FORM COVER SHEET

Section 1: Project Leader Contact Details

Title:          Dr
First Name:     Elisabeth
Last Name:      Harrison
Phone Ext.      6716
Fax:            9199
Mobile:         0419 496 275
Department:     Linguistics
Faculty or      Human Sciences

Alternative Contact Details
Name:           Eva Nemeth
Phone:          6877
Department:     Linguistics
Faculty:        Human Sciences

Section 2: Department Details

Lead            Linguistics
Ongoing         Name                               Professor Linda Cupples
and             Department                         Linguistics
by Head of
Department:     Signature


Section 3: Faculty Details

Lead Faculty:   Human Sciences

Ongoing         Name                               Professor Janet Greely
and             Faculty                            Human Sciences
by Executive
Dean:           Signature


Section 4: Application Details

Title:                 Development of an assessment rubric for clinical viva exams

Project Alignment:     Please circle whichever is appropriate:

                           2. Assessment and promotion of student learning

Outcomes:              We will produce an assessment rubric for examiners to use in student viva exams in
                       the Master of Speech and Language Pathology.

Method:                In clinical degree programs like the Master of Speech & Language Pathology (MSLP),
                       students must attain high standards of competence in practice settings such as
                       hospitals, community health centres, and schools. While most of the assessment of
                       their clinical learning therefore takes place off-campus, program staff must also
                       assess students’ progress to ensure that the University’s assessment standards are
                       maintained. In the MSLP this assessment occurs in viva exams. While there is strong
                       agreement that this is a valuable assessment tool, we are aware that it can be
                       improved. In this project we will develop and trial the use of an assessment rubric that
                       will (1) provide students with the levels of achievement required for each assessment
                       criterion and (2) better align viva exam feedback with other clinical assessment
                       measures used in the MSLP. The development of the rubric will begin with data
                       gathered from past viva reports, and interviews with examiners and students. The
                       project timeframe allows for an initial draft to be trialled with Year 1 students, and a
                       revised version to be trialled with final year students.

What processes         Recurrent evaluation of the rubric is embedded in the project. It will be trialled in
are built into the     SPH805 at the end of 2011, and twice in 2012 in SPH806. Feedback will be sought
Project to enable      from examiners and students after they use the rubric in viva exams, and their
it to be evaluated?    comments will be used to improve each version. Indirect feedback will also be
Involvement in the     available via LEUs.
 LTC Evaluation
Project will be
Does this Project      Yes. HREC approval will be required before using de-identified student viva reports in
require Human          the first stage of the project. Students will be invited to allow their de-identified reports
Ethics approval?       to be used for the project.
Key Words:             Assessment rubric, clinical competence, feedback

Submitting your application

Applications are due on Friday 27 May 2011*. Submit an electronic version of your application to:

          Ms Barb McLean
          Office of the Provost
          E11A 208

 *This is the University closing date. Different Department and Faculty closing dates might apply. Please liaise with
your Department and Faculty well ahead this closing date to ensure an endorsement and signatures on the application
are obtained before the University closing date.

       Development of an assessment rubric for clinical viva exams

                              Dr Elisabeth Harrison & Eva Nemeth
                                    Department of Linguistics
                                   Faculty of Human Sciences

Project Outcomes and Rationale
The project addresses the priority area of assessment and promotion of student learning,
specifically –
      assessing postgraduate coursework students
      assessing students in practice settings.
Assessment of students in professional Speech Pathology degrees centres on evaluating their
progress towards attainment of profession-specific entry level standards. These standards are
defined in the Competency Based Occupational Standards (CBOS) of Speech Pathology
Australia (SPA), the professional association. The CBOS defines the minimum skill levels,
knowledge base and attitudes required for entry-level practice in the profession.
Australian speech pathology degree programs and SPA have developed and validated a
competency measurement tool called the COMPASS (Competency Measurement in Speech
Pathology). For each of the 11 CBOS competencies, students are rated during each clinical
placement on a scale ranging from ‘Novice’ at the lower end of the scale, to ‘Intermediate’ in
mid- scale, and ‘Entry Level’ at the upper end. As they progress through a series of clinical
placements, these ratings provide students with clear indications of their progress towards
entry-level standard. With the COMPASS used on-site during clinical practicum placements, its
use by fieldwork supervisors ensures authentic and real-life assessments of students’ learning
in practice settings.
In the Master of speech and Language Pathology program, students also undertake viva
exams. These additional assessments are conducted on campus. Viva exams allow program
staff directly to assess students’ ability to relate their theoretically-based knowledge, derived
from the program’s academic components, with their practical experiences and skills developed
in clinical placements.
This project will develop a rubric for viva exams that aligns assessment criteria with the same
gradations of achievement that are used in the COMPASS, i.e., novice, intermediate, and

We have an well-established set of marking criteria for viva exams. Repeated use over the past
10 years leads us to believe that the criteria are thorough and provide a comprehensive
framework to guide students as they prepare for these exams. What is now needed is a rubric
with a set of valid and clear descriptors of students’ levels of achievement on these Critieria.
This rubric will provide students with specific and consistent feedback and show them how they
can improve on each criterion.
Our general approach is to establish the range of possible levels of achievement on each
criterion and to use this information to derive descriptors. To achieve this, de-identified student
viva reports from past years will be analysed so that common themes can be identified. This
initial data will provide experienced examiners’ most and least frequent areas for comment, and
multiple examples of authentic examiner comments.

We will then interview viva examiners, who are academic and clinical staff who teach on the
MSLP. Experienced (more than 5 years) and less experienced examiners will be included in
this stage of the project. The purpose of these interviews will be to validate the thematic
analysis and to gain examiners’ responses to issues that are expected to arise from the text
analysis. For example, we expect to find evidence of inconsistent expectations of student
performance, and subjective and/or ambiguous comments in the reports.
An initial draft of the rubric will be trialled with Year 1 students in the viva exams at the end of
2011. The draft rubric will not be trialled with Year 2 students at that time, because that exam
will be their final viva, and therefore a high-stakes assessment.
Evaluation of the trial version of the rubric will include efficiency and effectiveness of the rubric
for examiners, and effectiveness for students. These reports will inform development of a
further version. Once the rubric has been thoroughly trialled with Year 2 students in 2012 (end
of Sem 1 and Sem 2) the final version will be implemented in SPH805 and SPH806, and made
available to other speech pathology degree programs.
This project aligns with Macquarie@50 and the Academic Plan 2010-2014. The purpose of the
clinical practicum units (SPH805,SPH806) in which the viva exams are embedded, is to ensure
that students develop the specific speech pathology knowledge and skills that enable them to
practice as competent professionals. In this project we are improving the way that we
communicate with students about their progress, giving them clear and consistent feedback
that is fully aligned with learning objectives. By differentiating the levels of achievement
expected in viva exams students will have clear information about their achieved levels and,
crucially, those where further development is required. Making the rubric available to other
university programs raises the profile of the Macquarie program, and highlights its focus on high
quality clinical education for speech pathology students.

Value/Need for the Project
Within postgraduate clinical degree programs, rigorous assessment of students’ clinical
progress is required. While most clinical progress assessment occurs in off-campus sites, the
assessment standards of the university must also be maintained by program staff. Viva exams
are an excellent additional method of assessing clinical learning but this form of assessment is
time consuming (approximately 90 minutes per student for exam and report writing), and the
resulting examiner fatigue can create inconsistent and unhelpful feedback for students.
The viva exam rubric will reduce the time the time required for the ‘exam+report’ to 45-60
minutes per student. In addition, the rubric will increase feedback consistency by developing a
‘marking key’ for each criterion in the rubric. Examiners are the main beneficiaries of the
increased efficiency (although students also receive reports more quickly) and students are the
main beneficiaries of the stronger alignment between criteria and levels of achievement. Both
parties benefit from unambiguous guidelines as to what is expected of students.
The development of the viva exam rubric will benefit other Macquarie clinical degree programs
in clinical audiology, clinical psychology, clinical neuropsychology, physiotherapy, and
All Australian speech pathology degree programs require students to achieve entry level
standards of competence on the CBOS, as measured on the COMPASS. Indeed the
development of both CBOS and COMPASS was achieved with the full participation and
collaboration of university programs with the profession, and the professional association
(SPA). This collegial culture between university programs has resulted in several cross-
program benchmarking projects in speech pathology. The Macquarie viva exam format was
based on the viva exam format used in the speech pathology program at the University of
Newcastle. Dr Sally Hewat, Clinical Program Director at Newcastle, has expressed interest in
using the viva exam rubric there. The collaboration between the Macquarie and Newcastle

programs raises the profile of both university’s speech pathology programs within the
profession generally.
This project could also be used as a means of introducing assessment rubrics more widely into
the MSLP. Development of online versions of assessment rubrics is a further natural
progression and one that could easily be achieved once the initial achievement levels are

Project Management and Budget
This project will use established methods of rubric development, and apply them to the
Macquarie MSLP’s viva exams. A crucial element in the success of this project will be the
recruitment of a project officer who is knowledgeable about speech pathology assessment
standards and, in particular, viva exams. Dr Lynne Mortensen is a former lecturer and clinical
coordinator in two speech pathology degree programs, and has indicated her availability. Dr
Mortensen has the skills and experience in qualitative research methods that are required for
the textual analysis and interviews.
Eva Nemeth, Co-Director of Clinical Education, is also an experienced qualitative researcher
and will assist with data analysis.

Budget outline

         Budget item                    July - Dec 2011                 Jan – Dec 2012

Digital voice recorder and lapel               $260
mic for interviews

Casual rate for LM to analyse      14 hrs @ $54.76 (casual rate,
student reports, identify themes    includes oncosts) = $766.64

Casual rate for LM to interview    28 hrs @ $54.76 (casual rate,   35 hrs @ $54.76 (casual rate,
examiners, students after rubric   includes oncosts) = $1533.28    includes oncosts) = $1916.60
trials, analyse interviews

Prepare draft rubric               14 hrs @ $54.76 (casual rate,   14 hrs @ $54.76 (casual rate,
                                   includes oncosts) = $766.64      includes oncosts) = $766.64

Coordinate trial of draft rubric   7 hrs @ $54.76 (casual rate,    14 hrs @ $54.76 (casual rate,
                                   includes oncosts) = $383.32      includes oncosts) = $766.64

Final report                                                       14 hrs @ $54.76 (casual rate,
                                                                    includes oncosts) = $766.64

Year totals                                  $3,710                          $4,217

Total $$7,927

Time line

LM analyses de-identified student viva reports, identifies potential         July 2011
issues & authentic examiner comments

LM interviews more & less-experienced viva examiners, clarifies            Aug-Sept 2011
potential issues

LM, EH, EN develop draft rubric for Year 1 student viva exams                Oct 2011

EN coordinates initial trial of rubric with Yr 1 students (novice level)     Nov 2011

LM interviews examiners, Yr 1 students re. rubric trial                    Nov-Dec 2011

LM & EN redraft rubric and trial with Yr 2 students (intermediate          Mar-June 2012

LM prepares final draft rubric                                             Aug-Sept 2011

EN coordinates final trial with Yr 2 students (entry level)                  Nov 2012

LM writes final project report                                               Dec 2012

Project management and evaluation
The project will be managed by Dr Elisabeth Harrison and Eva Nemeth. They will work closely
with Dr Lynne Mortensen, the Project Officer, who will analyse student reports, and examiner
and student interviews. Harrison and Nemeth will be responsible to ensure that the project runs
to plan and remains within budget.
Through the project methodology described there is intrinsic and thorough testing of the rubric
throughout its development, and as implementation rolls out. Evaluation will include examiners
and students in order to ensure that the final product is useful for both.
Student evaluation will also be available through Learner Evaluation of Unit (LEU) feedback at
the end of 2011 (SPH805) and 2012 (SPH806).


To top