Docstoc

The State of the Art

Document Sample
The State of the Art Powered By Docstoc
					CAA in UK HE:
   The State of the Art

      Bill Warburton
University of Southampton
 Email: wiw@soton.ac.uk
                            Research &
                            Graduate School
                            of Education
The 2003 survey
   Introduction
   Demography
   Findings:
       Question banks
       CAA software
       Pros & Cons
       Question types
       VLEs & MLEs
       Interoperability
                           Research &
   Summary                Graduate School
                           of Education
The 2003 survey
   Why do a CAA survey now?
       Increasing prevalence of web-based delivery
       Pervasiveness of Virtual Learning Environments (VLEs)
       Emergence of Managed Learning Environments (MLEs)
       Emergence of Interoperability standards e.g. IMS/QTI
       Four year gap e.g. between 1999 & 1995 CAA surveys




                                                       Research &
                                                       Graduate School
                                                       of Education
The 2003 survey
   Similarities with the (1999) CAA Centre Survey*
       For congruence, a superset of the 1999 survey
   Dissimilarities with 1999 survey
       Extra questions on VLEs, MLEs & Interoperability
       Targeted CAA enthusiasts rather than non-users
       Delivered purely by web-based CAA system



                                                          Research &
                                                          Graduate School
                                                          of Education

*Bull, 1999; Bull & McKenna, 2000; Bull & Hesketh, 2001
The 2003 survey:            Who were the respondents?
   Targeted people who were likely to be using CAA
   Incentive- prize draw of two £25 book tokens
   Survey questionnaire was distributed using three JISC
    mailbase lists:
       CAA
       TOIA
       ALT
   50 responses received by initial deadline of 18th May
   Will be issued more widely
                                               Research &
                                               Graduate School
                                               of Education
The 2003 survey:                                       Who were the respondents?
   CAA Involvement:
         A constituency of enthusiasts


    80%

    70%

    60%

    50%

    40%

    30%

    20%

    10%

    0%
           Not currently     Currently      Intend to be     No longer      Research &
           involved with   involved with   involved with   dinvolved with   Graduate School
                                                                            of Education
               CAA             CAA              CAA             CAA
The 2003 survey: Job Function
   Little more than half the 2003 respondents were
    academics (1999: 77% were academics)
   An increased proportion of pedagogic & ICT support

     60%


     50%


     40%


     30%


     20%

                                                        Research &
     10%
                                                        Graduate School
                                                        of Education
     0%
           Academic   ICT support   other   pedagogic
            teacher                          support
The 2003 Survey: Who supports CAA now?
   CAA support is largely centralised
   Even where support has been devolved to faculties /
    departments, there is usually a central CAA service
     50%




     40%




     30%




     20%




     10%                                                                                 Research &
                                                                                         Graduate School
                                                                                         of Education
     0%
           Solely in a    Solely within     Solely in    In a central unit   Elsewhere
           central unit    faculties       academic         and within
                                          departments   faculties and/or
                                                          departments
The 2003 survey:        Who uses CAA now?
Usage by Subject : broadly the same as 1999
 2003                   1999
 1 Computing/IT         1 Computing/IT
 2 Biomedical Science   2 Biomedical Science
 3 Other science        3 Geological science
 4 Geological science   4 Maths/Engineering
 5 Business             5 Other science
 =8 Psychology          6 Modern languages
 =8 Leisure             7 Psychology
 =8 Study skills        8 Business
                        9 Social science
                        10 Humanities
                        11 Built environment
                                               Research &
                        12 Leisure             Graduate School
                        13 Study skills        of Education
2003 Survey: The use of question banks
   Nearly 60% said they now use question banks
   Nearly 90% would be interested in sharing questions

      Do you use question banks?   Would you be interested in sharing
                                   questions with academics at other
                                             institutions?


                                                 No
                                                 12%
                      No
                      41%
          Yes
          59%                                                  Research &
                                           Yes
                                           88%                 Graduate School
                                                               of Education
The use of question banks
   Pros of question sharing- shared loads:
       Authoring
       QA
       Editing
   Cons of question sharing:
       Initial time penalty
       difficulty of applying rigorous QA standards across multiple
        authors & institutions
       requirement for multiple formats to suit all users

                                                         Research &
                                                         Graduate School
                                                         of Education
The use of question banks
   Example in Science:
       The e3an project developed an item bank of more than
        1500 questions for use in Electronics and Computer Science
   Example in Humanities:
       A question bank has been developed to pre-course assess
        modern language students




                                                       Research &
                                                       Graduate School
                                                       of Education
2003 Survey:                      What CAA software is used in HE now?

   Commercial CAA systems are more dominant
   Many respondents used multiple CAA systems
                             What CAA software do you use?



     14

     12

     10

      8

      6

      4

      2

      0
          Perception Blackboard    WebCT    TRIADS     i-assess   Other
2003 Survey:        What CAA software is used in HE now?


   Split between users of Commercial and Open systems
   Commercial systems- traditionally seen as expensive
    but easier to use
   Non-commercial systems e.g. TRIADS- free, but
    support-intensive




                                                 Research &
                                                 Graduate School
                                                 of Education
2003 Survey:           Advantages of different CAA software

 2003 Advantages of commercial     2003 Advantages of Open systems:
 systems ranked by popularity:
 1 Flexibility                     “TRIADS is very flexible”
 2 Support for Interoperability:
 “QTI…which makes it               “AIM is tailored to
 futureproof and easier to
 share questions”
                                   mathematics. It is open
 3 Student tracking-               source, rather than a
 4 Range of Q. types               professional package”
 5 Reporting-
 “ability to produce wide range    “Ease of use (as I
 of reports“                       developed it)”
 =9 Support
 =9 Scalability
 =9 Robustness                                             Research &
                                                           Graduate School
 =9 Ease of use                                            of Education
2003 Survey: Disadvantages of different CAA software
  2003 Disadvantages of                    2003 Disadvantages of Open systems:
  commercial systems ranked by
  unpopularity:
  1 Limitations of quiz facilities built
  into VLEs
                                           “TRIADS is very complex”
  2 Technical issues: e.g. Windows
  applications only, poor use of the
  database, non - standard user
  interface, cost, poor integration with
  MIS, poor password encryption, ad
  hoc patches and fixes
  3 non-intuitive user interfaces
  =5 Inflexibility: “need to use the
  underlying html type code to make
  it do what i want”                                               Research &
  =5 Restricted feedback and                                       Graduate School
  statistical results                                              of Education
2003 Survey: Main benefits of CAA
   Little different from 1999, considering small sample
   1999 survey:
       Speed of feedback to students
       Statistical analysis of results
       Automatic marking
   2003 survey
     Speed of feedback:
    “Available 24/7 - with instant feedback”
     Speed of automatic marking:

    “50 pieces of coursework marked in a flash”
     Formative use promotes student learning:

    “students often retake quizzes and improve their score”
2003 Survey: Main disadvantages of CAA
   There appears to have been a shift- Learning curve
    and up-front investment are now seen to be the most
    prominent disadvantages
   1999 survey:
       access to and reliability of hardware and software
       amount of time needed create and organize delivery
       difficulty of writing good questions
   2003 survey:
       amount of time needed create and organize delivery
       access to and reliability of hardware and software
                                                          Research &
       difficulty of writing good questions              Graduate School
                                                           of Education
2003 Survey: Does CAA save time?
   The 2003 respondents were evenly split regarding the
    time saved by CAA
      If academic staff at your institution use
       CAA, are they saving their own time?




                                 No
              Yes                49%
              51%


                                                  Research &
                                                  Graduate School
                                                  of Education
2003 Survey:                     Does CAA save time?
   Surprisingly, a large majority of those who reported
    productivity benefits did not see a corresponding shift
    in workload to other staff
     Has this meant a significant shift
     in the workloads of other staff?




                               No
                      z
         Yes                   71%
         29%
                                                       Research &
                                                       Graduate School
                                                       of Education
2003 Survey: Formative vs. Summative testing
   There appears to be a definite shift towards formative
    use: less than half the 2003 tests were summative
    (1999: most tests were summative)
   Diagnostic use is emergent: other formative usage is
    long-established Type of assessment

         50%

         40%

         30%

         20%
                                                                 Research &
         10%                                                     Graduate School
                                                                 of Education
         0%
               diagnostic   formative      self-     summative
                                        assessment
2003 Survey: Question types
   1999: MCQ & MRQ most popular followed by text and
    numeric input, then graphic hot spot and other types
   Little appears to have changed in 2003: MCQ still top

                    What question types/formats do you use?



       60%

       50%

       40%

       30%

       20%                                                             Research &
                                                                       Graduate School
       10%
                                                                       of Education
        0%
             graphical   MCQs        multi-     other     text input
             hot spots             response
2003 Survey: Who is doing CAA?
   Moore's Gap has yet to be crossed: CAA appears still to be
    largely the province of „early adopters‟
   Will CAA follow VLEs and become mainstream technology?


       Which of the following describes the experience of the staff at your institution with
                                             CAA?



              80%

              60%

              40%

              20%
                                                                                      Research &
               0%                                                                     Graduate School
                     No-one      A small   20% or       Some no Some may,
                    uses CAA    group of    more      longer use but not
                                                                                      of Education
                               enthusiasts                CAA     aware
2003 Survey: The growth, influence and relevance of VLEs
   Have VLEs crossed Moore‟s Gap? Are now pervasive
   Most respondents (81%) had implemented site-wide VLEs
   It seems that VLE quiz tools give users a taste for CAA
   The uptake of VLEs may foreshadow the uptake of CAA

        Does your institution already
      provide an institution-wide VLE?


                       No
                       19%


                                                    Research &
                Yes                                 Graduate School
                81%                                 of Education
2003 Survey:          The growth, influence and relevance of VLEs
   VLEs are seen as a largely positive (60%) influence on the
    uptake of CAA
   Cautionary note: Two reports that VLEs apparently
    discouraged the use of CAA


     Do you think your VLE has encouraged or discouraged the use of CAA?



      45%
      40%
      35%
      30%
      25%
      20%
      15%                                                                  Research &
      10%                                                                  Graduate School
       5%                                                                  of Education
       0%
            Discouraged Mildly    Neutral      Mildy  Encouraged
                      discouraged           encouraged
2003 Survey: Interoperability & VLEs
   The majority of VLE quiz users wanted to export their
    quizzes to a dedicated CAA system
   One respondent wanted to import CAA items and
    tests into a VLE
              Would you like to be able to
            export quizzes from your VLE to
                    a CAA system?


                                I do not
                                 use my
                      Yes         VLE's
                      40%         quiz
                                 facility     Research &
                                   38%        Graduate School
                            No                of Education
                            22%
2003 Survey:                      tension between VLE quizzes and CAA

   Many respondents attempting to do CAA using VLE‟s
    Quiz engine
     Most commented unfavourably on VLE Quiz engines:
    “…testing facility is very limited, but is all we have available”
   Cultural & technical differences between VLEs & CAA:
       Words do not always have common meanings-
            Blackboard refers to a quiz as an „item‟
            In CAA an „item‟ is usually a single question
       VLE quiz items/tests often restricted to proprietary formats,
        & may be less secure
                                                             Research &
                                                             Graduate School
                                                             of Education
2003 Survey: The emergence of MLEs
   More than two thirds of respondents‟ institutions have
    stated an intention to develop an MLE
   However, the concept of an MLE is contended (JISC
    7/99 projects)
         To your knowledge, is your institution
              aiming to develop an MLE?



                              Don't
                              know
                               26%

                    Yes                No         Research &
                    68%                6%         Graduate School
                                                  of Education
2003 Survey: Interoperability
   The majority of HE users thought interoperability to
    be important

              How important is Interoperability to you?


    30

    25

    20

    15

    10
                                                              Research &
    5
                                                              Graduate School
                                                              of Education
    0
          Not at all    Not really   Moderately     Very
         important     important     important    important
2003 Survey: Interoperability
   Although most HE users thought interoperability to be
    important, awareness of interoperability standards
    varied widely


              How aware are you of interoperability standards?




        50%
        40%
        30%
        20%
                                                                 Research &
        10%                                                      Graduate School
         0%                                                      of Education
                 Not really   Moderately    Very aware
                  aware        aware
Summary
Superset   of 1999 survey + extra questions on VLEs, MLEs & Interoperability
Demography: More support staff involved than in 1999

Question banks: Desirable but rare

CAA software: Commercial CAA systems dominant but inflexible

Pros: Little changed from 1999- time savings

Cons: Learning curve now more significant than cost

Question types: MCQs & MRQs still predominant

VLEs: have crossed Moore‟s Gap and appear to drive CAA

MLEs: Most HEIs intend to implements MLEs (but concept of MLE contended)

Interoperability: thought to be important, but awareness of standards varies




                                                              Research &
                                                              Graduate School
                                                              of Education
End of presentation
   Questions?
       Question banks: desirable but how much effort?
       CAA software: What features do people most want?
       Pros & Cons: at what point do the pros outweigh the cons?
       What question types do people most use? Most want?
       What roles do Interoperability, VLEs & MLEs play in
        promoting CAA?
       Do Interoperability, VLEs & MLEs drive CAA or vice-versa?
       When will CAA cross Moore‟s Gap?


                                                     Research &
                                                     Graduate School
                                                     of Education

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:9
posted:4/6/2010
language:English
pages:32