Docstoc

LTSN Mini-Project Objective Structured Test for Engineers _OSTE_

Document Sample
LTSN Mini-Project Objective Structured Test for Engineers _OSTE_ Powered By Docstoc
					 The OSTE: Objective
 Structured Technical
   Examination for
 Engineering students

Nandini Alinier, Guillaume Alinier
University of Hertfordshire, U.K.
           ICEE 2005
Overview

   Background & origins
   Application in an Engineering context
   Stages of the OSTE
        Development of stations
        Preparation for sessions
        Pilot
        First series
        Second series
   Analysis

N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Background
   Objective Structured Clinical
    Examination (OSCE) (Harden 1975)
   Originally developed in the mid-
    seventies in Dundee, Scotland
        Used in healthcare education
        Method to objectively assess the skills
         of healthcare trainees
   Used in a project funded by the
    British Heart Foundation at UH
    during 2001-03 (Alinier et al 2004)
   Perceived positively by students &
    lecturers (Alinier 2003)

N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
OSCE
   Comprises 15 to 20 stations
   Station: short task performed in 5 mins
       Practical exercise
            Requires an assessor
       Theoretical exercise
            Pen & paper
   Gap period between stations (1 min)
       Students rotate individually through the
        stations
       Allows time for assessors to complete
        marking & reset any equipment
   Timing of session by an electronic clock
N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
OSCE (continued)

   Students given explicit written
    instructions
   Objective marking sheet for
    each station




N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Example of a station from an OSCE




      N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
             Applied to Engineering
   Project undertaken in the School of Electronic,
    Communication & Electrical Engineering at the
    University of Hertfordshire, UK
   Funded by the Higher Education Academy
    Engineering Subject Centre (Learning &
    Teaching Support Network (LTSN) Engineering)
   Objective Structured Technical Examination
    (OSTE)


              N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Study about the OSTE

   Used for formative assessment
   Piloted with level 2 BEng students
   Conducted with level 1 BEng
    students (1st + 2nd semesters)
   Each student did the OSTE twice
   Circulated feedback questionnaires
    for students + assessors


N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
        Stages of the OSTE
Stage
                                    Stage
 No.
 1            Development of OSTE stations

 2            Preparation for OSTE sessions

 3                           Pilot of OSTE

 4           First series of OSTE (in Dec 04)

 5        Second series of OSTE (in Mar 05)

        N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Stage 1: Development of OSTE
stations
   Gathering a variety of ideas about
    essential electrical engineering skills:
        Seminar within School
        Call for ideas via Email
   Identified appropriate range of
    exercises doable in 5 mins
   16 stations were short-listed
   Developed instruction sheets &
    marking criteria for each station

N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Stage 1: Development of OSTE
stations (continued)


   Types of stations:
      Theoretical: pen & paper
      Practical: assessor required

      Computer-based: assessor
       required




N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Stage 2: Preparation for
OSTE sessions
   Development of PC-based clock
    with an integrated voice prompt
   Briefing of students
   Training of staff to act as
    assessors
   Designing of student & assessor
    feedback questionnaires

N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Stage 3: Pilot of OSTE

   Group of 15 volunteer 2nd year
    BEng students
   Resulted in changes in:
      Tasks to be carried out
      Marking sheet

      Instruction sheet




N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Example of a computer-based station




       N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Stage 4: First series of OSTE

   1st year BEng students
   During 1st semester (Dec 04)
   Formative OSTE without feedback
   To determine the level of students’
    performance at the start of the
    course without preparing them too
    much for their 2nd OSTE
   Feedback collected from students +
    assessors

N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Stage 5: Second series of OSTE

   Same group of 1st year students
   During 2nd semester (Mar 05)
   Run as “mixed-mode” OSTE:
                                                     (Alinier 2003)

      Collection of data about students’
       performance
      Feedback to students by assessor

   Feedback collected from students
    + assessors

N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
    Results: students’ perspective
   Out of 46 student feedback questionnaires
       The OSTE was beneficial: 93.5%
       The OSTE helped them developed confidence in using
        equipment: 73.9%
       52.2%: rated the OSTE as very good or good
    (5-point Likert Scale with 1=very good, mean=2.4 with 95% C.I. 1.4–3.4)

       The OSTE should be used as a summative
        assessment tool: 28.3%
       Number of times OSTE should be repeated per year:
        2.9 (95% C.I. 1.4 - 4.4)


                  N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
    Results: assessors’ perspective
   Out of 33 assessor feedback questionnaires
       The OSTE should be incorporated into the 1st year
        BEng curriculum: 93.1%
       The OSTE helped students develop their confidence in
        using equipment they were exposed to: 75.8%
       90.6%: rated the OSTE as very good or good
    (5-point Likert Scale with 1=very good, mean=1.5 with 95% C.I. 0.8 – 2.2)

       The OSTE should be used as a summative
        assessment tool: 75.9%
       Number of times OSTE should be repeated per year:
        2.2 (95% C.I. 1.4 - 3.0)

                  N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Further analysis

   Identify key weaknesses among
    students
   Any correlation between
    performance in OSTE and
    performance as per in-class
    assessment



N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
                                          OSTE results
        1     2    3       4     5     6     7       8    9   10    11    12   13  14    15    16
            Gain+
Station R.   Semi Read Elec. RF solder Set Solde C.           C.   Tina Signal K- Logic Op. LED
       value Log Oscil. Circ. spect. joints Oscil. ring type value
                  Oscil.       spect.       Oscil.                      Gen. Map gates Amp. circuit
            Graph        Diag.
                         Diag.                                                         circuit

OSTE
  1   74.3    13.1   56.0   50.2   20.0   50.6   41.7   42.1   13.0   55.4   42.7   48.5   23.8 35.6   38.8   29.6
39.7%



OSTE
  2   74.5    46.6   75.9   56.6   20.7   59.3   87.7   84.4   27.0   64.3   60.6   58.2   53.9 47.5   48.4   22.5
55.5%




 Diff
        0.2   33.5   19.9   6.4    0.7    8.7    46.0   42.3   14.0   8.9    17.9   9.7    30.1 11.9   9.6    7.1
15.8%




                              N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
Conclusions

   OSTE: perceived positively by
    students and assessors
   Formative assessment 
   Summative assessment ?




N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
    Thanks for your attention



                     Any questions?



              N.D.Alinier@herts.ac.uk
               G.Alinier@herts.ac.uk

N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland
                              References
     Harden R M, Stevenson M, Downie W W, Wilson G M, "Assessment of clinical
      competence using an objective structured clinical examination", Medical
      Education, Vol 1, 1975, 447-451.
     Alinier G, "Nursing students’ and lecturers’ perspectives of OSCE incorporating
      simulation", Nurse Education Today, Vol 23, No 6, 2003, 419-426.
     Alinier G, Hunt W B, Gordon R, "Determining the effect of realistic simulation in
      nurse education: Study design and initial results", Nurse Education in Practice,
      Vol 4, No 3, 2004, 200-207.
     Ross M, Carroll G, Knight J, Chamberlain M, Fothergill-Boubonnais F, Linton J,
      "Using the OSCE to measure clinical skills performance in nursing", Journal of
      Advanced Nursing, Vol 13, 1988, 45-56.
     Hulett S, Gilder N, "The application and development of the Objective Structured
      Practical Examination in evaluating physiotherapy students’ performance at the
      University of Cape Town", South African Journal of Physiotherapy, Vol 42, No 2,
      1986, 40-43.
     Marshall G, Harris P, "A study of the role of an objective structured clinical
      examination (OSCE) in assessing clinical competence in third year student
      radiographers", Radiography, Vol 6, 2000, 117-122.
     Mossey P A, Newton J P, Stirrups D R, "Scope of the OSCE in the assessment
      of clinical skills in dentistry", British Dental Journal, Vol 190, No 6, 2001, 323-326.
     Townsend A H, Mcllvenny S, Miller C J, Dunn E V, "The use of an objective
      structured clinical examination (OSCE) for formative and summative assessment
      in a general practice clinical attachment and its relationship to final medical
      school examination performance", Medical Education, Vol 35, 2001, 841-846.
     Harden R M, "Twelve tips for organising an OSCE", Medical Teacher, Vol 12, No
      3/4, 1990, 259-264.
     Sloan D A, Donnelly M B, Schwarts R W, Strodel W E, “The Objective Structured
      Clinical Examination: The new gold standard for evaluating postgraduate clinical
      performance”, Annals of Surgery, Vol 222, No 6, 1995, 735-742.
    N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

				
DOCUMENT INFO
Categories:
Tags:
Stats:
views:4
posted:9/30/2011
language:English
pages:26