PowerPoint Presentation - Gabrielle AACE Ed-Media Conference by ert554898


									    The Use of Systematically Designed
Technology-Mediated Instructional Strategies
      to Improve Cadet Performance
           at a Military Academy

                  AECT Orlando
                October 20, 2005
        Donna M. Gabrielle, Ph.D.
                                                Technology is not…
 The solution
 The only way to address the gaps in the
 The focus of this research

                                                                                                                 c             n
                                                                                                             Qu i k Ti me ™ a d a
                                                                                                   F               e      )           e
                                                                                                TIF (Un co m p r s s e d d e co m p r s s o r
             Qu i k Ti me ™ a d a
                               n                                                                               e             i    c
                                                                                                   a re n e e d d to s e e th s p i tu re .
    F      c       re    d       c
 TIF (Un o m p s s e ) d e o m p s so r re
                e                   i
    a re n e e d d to s e e th i s p c tu re.

                                                                         cTm       n
                                                                     Q ui ki e™ ad a
                                                        TI FF( U     r        e
                                                                ncom pessed) dcom pr esors
                                                                 e               s c
                                                            ar e neded t osee t hi pi t ur e.
      Gaps in the Literature
 L. Visser, Plomp, Amirault, and Kuiper (2002)
  examined proceedings of World Conferences
  of the International Council for Distance
  Education between 1988 and 1995. They
  found that less than one percent of the
  papers (only six of 801) focused on
  motivational issues.
 Motivational needs of learners are often
  overlooked in research and in practice.
       USMA Environment
 Highly structured environment
 Instructor experience varies widely
 “Thayer Method” encouraged
 Homogeneous population
 Ideal environment for experimental research
 Very little prior experimental research
  conducted at military academies
           Gabrielle Study
 N=784 undergraduates
 Double blind design experiment
 Randomization of sections for each of 17
 12 courses including English, history,
  math, computer sciences, chemistry,
  leadership, and wellness
          Literature Review
 J. Visser (1990) studied the impact of
  strategies designed using Keller’s ARCS
  model and delivered in a motivationally
  challenging environment.
 Song (1998) studied motivationally adaptive
  computer-aided instruction with middle school
 L. Visser (1998) used a motivational
  messages support system (MMSS) to
  encourage learners to persist in
  correspondence courses.
         Literature Review
 Preczewski (1997) used a cross-
  sectional survey approach to investigate
  self-directedness of cadets (N=723)
  enrolled in all four years (from freshmen
  to seniors) at the academy.
 Preczewski examined various
  demographics and concluded the
  undergraduate experience fails to
  positively affect SDL
      Keller’s ARCS Model
 Attention
 Relevance
 Confidence
 Satisfaction
  1. Obtain
                       Course description and rationale
                       Setting and delivery system
                       Instructor information
                                                                  10-Step ARCS
       2. Obtain                Entry skill levels
      audience                  Attitudes toward school or work
     information                Attitudes toward course

                                       Motivational profile
          3. Analyze                   Root causes
           audience                    Modifiable influences

               4. Analyze                  Positive features
                 existing                  Deficiencies or problems
                materials                  Related issues

                                                   Motivational design goals
                   5. List ojectives               Learner behaviors
                   & assessments                   Confirmation methods
                             Brainstorm list of tactics
6. List potential
                             Begninning, during, and end
                                                                      10-Step ARCS
      7. Select &                 Integrated tactics
     design tactics               Enhancement tactics
                                  Sustaining tactics

                                        Combine designs
            8. Integrate                Points of inclusion
          with instruction              Revisions to be made

                    9. Select &               Select available materials
                     develop                  Modify to the situation
                     materials                Develop new materials

                       10. Evaluate &             Obtain student reactions
                           revise                 Determine satisfaction level
                                                  Revise if necessary
Systematic Instructional Design
Keller (1999, p. 3) proposes three
   assumptions of systematic instructional
1. Learner motivation can be affected by
   external factors
2. Motivation is a method of affecting
3. Systematically designed instruction can
   “predictably and measurably influence
Conceptual Model of Design Experiment
         Instructional strategy (TMIS)
   Systematically designed strategies to assist
    learners in mastering specific course performance
    objectives. Each TMIS was developed with
    Keller’s ARCS model, delivered via email, and
    included three components:
       (1) motivational messages at the beginning and end of
        each TMIS;
       (2) a link to instructional content provided on
        technologies including PDAs, web, and CD-ROM;
       (3) the SDL survey to track participation and obtain
        feedback to improve future TMIS.
Systematic Design of TMIS
        Materials & Measures
 Keller’s Instructional Materials Motivation
  Survey (IMMS)
 Guglielmino’s Readiness for Self-Directed
  Learning Survey (SDLRS)
 Performance measures (homework, projects,
  examinations, and other grades)
 Post-strategy SDL survey
    Summary of Interventions
 Developed from performance objectives
 Designed with Keller’s ARCS model of
 Delivered via technology
 Blended learning model, using interventions
  as supplementary instructional material to
  encourage learners to self-direct outside the
  requirements of a course
    Technologies Employed
 PDAs
 Web
 CD-Rom
 Computer
 Streamed
Formative Evaluation of TMIS
             Hypothesis 1
Treatment group students who used TMIS
have significantly higher levels of academic
performance (measured by course
aggregate points including homework,
projects, papers, and examinations) than
control group students taught by traditional
             Hypothesis 2
The change in the mean pretest and posttest
SDLRS score of treatment students with
access to TMIS is significantly higher than the
change in the mean pretest and posttest
score of control group students taught by
traditional methods.
            Hypothesis 3
Treatment group students with access to
TMIS have significantly higher mean vector
scores for IMMS than control group students
taught by traditional methods.
Summary of Effect of TMIS on Academic
Performance- Control vs. Access to TMIS
 Treatment group students who used the TMIS had
 significantly higher levels of academic performance
 than control group students taught by traditional
 methods. (p=.0045)

                      Mean          dfn   dfd   FC     F        P
                N     Performance

Control         298   83.53         1     557   6.68   8.14   .0045
Treatment who   165   85.15
accessed TMIS
      Summary of SDLRS Analysis
The change in the mean pretest and posttest SDLRS
score of treatment students was significantly higher than
the change in the mean pretest and posttest score of
control group students taught by traditional methods.
                   SDLRS      SDLRS      Δ SDLRS
             N     pretest    posttest     score     SD      p
                    mean       mean

Control     91    214.07     210.77      - 3.30    14.52   .004

Treatment   104   215.72     218.54      + 2.83    15.07
  Nonparametric SDLRS Comparison
                 Wilcoxon Rank Sum Test- p=.001

            N       Mean Rank   Mann Whitney-U   Power   p

Control     91      82.91       3358.5             .96   .001

Treatment   104     110.39
          Summary of Pretest to Posttest
            SDLRS Analysis by Group
 Pretest to posttest paired samples t-test analysis for all groups

               N     SDLRS      SDLRS      Δ SDLRS   SD Δ     Power    p
                     pretest    posttest     score
                      mean       mean
Control        91   214.07     210.77      - 3.30    14.52   .48      .030

Treatment      23   212.26     201.70      -10.57    12.56   .98      .001
w/o Access

Treatment w/   81   216.70     223.32      + 6.62    13.53   .99      .000
     Comparison of ARCS Scores on
Instructional Materials Motivation Survey
                                                                                   8 5 .5
  140                                                                            .5 12
  120                                                                        11

   60           2 41 .9                               5
                                .26 1 .3
                                        8     .26 3 .9
   40                         27 3          31 3                        34
                                                                .83 18 .
        Attention         Relevance Confidence Satisfaction ARCS Total
                           Control                        Treatment

  Hotelling’s t-Squared Test and MANOVA- p<.0001
        Summary of Results
 Quantitative data showed positive effects of
  TMIS on motivation (p<.0001), performance
  (p=.0045), and SDL (p=.004).
 Qualitative feedback showed cadets
  appreciated the technology-mediated
  instructional strategies and felt that they
  were more self-regulated, motivated and
  they learned more as a result.
        Follow-up Interviews
 Follow-up interviews were conducted 18
  months after the study.
 All cadets reported that they didn’t share
  the interventions with control group cadets
 Most remembered the strategies and had
  positive feelings about the experience
 Several said they felt the study impacted
  their study habits even today
 Strategies are still being used at USMA
            Qualitative Feedback
   “I remember how interactive they were and how much
    fun they made learning.”
   “I think the strategies helped my learning the most but
    they also helped motivation too due to the change of
    learning styles.”
   “…I wish we would have more interactive material like
   “I had previously failed this class, so I also think that
    the strategies gave me confidence to succeed.”
   “I liked the PDA content. Since the class ended, I have
    stopped using the PDA, sadly.”
   “…the technology based strategies made it more
    interesting to learn.”
              Qualitative Feedback
   “I think I learned more simply due to a higher motivation
    to learn since I had technology.
   “I thought it was very helpful and I was glad I was part of
    the experimental group because I think it helped my
    success in the class.”
   “Participating in the study helped me learn my PDA. I
    enjoyed being able to down load other content.”
    “Having it on the PDA made it easier. I could take it
    anywhere. The PDA was easier than a textbook, or
    computer. I could take it with me easily and in case I had
    time, I could read it.”
   “I like anything that's interactive that helps me from doing
    the regular learning of reading from a textbook. I like to
    deal with things that grab my attention like moving
    demonstrations, things on the computer, etc.”
 Findings confirmed decades of research showing
  motivation is one of the most critical concerns in how
  and why people learn (Efklides, Kuhl, & Sorrentino,
  2001; Keller, 1979; Wlodkowski, 1999).
 Despite idiosyncratic nature of motivation, external
  factors can have a positive effect on learners
 Systematically designed TMIS can be positive means
  of affecting motivation, performance, and SDL
 These differences would expected to be more
  pronounced for a less homogeneous population
            Future Research
   Use same research design on more
    heterogeneous population
   Examine extensive demographic data that
    were collected
   Look at which strategies are most effective in
    which contexts
   Analyze extensive qualitative data that were
   Examine impact of TMIS over time to see if
    improved motivation, self-regulation, and
    academic performance continues to improve
    Gabrielle Contact Information
 Email: gabrielle@gabrielleconsulting.com
 Dissertation available for download at

To top