Presentation in ENGAGE community

Document Sample
Presentation in ENGAGE community Powered By Docstoc
					 Collaborative Group Work:
How Students Feel About the
          Robert Gervey
         Chia-Chiang Wang
        Mary O'Connor Drout
    Department of Rehabilitation Psychology
             & Special Education
       University of Wisconsin-Madison
Study Setting
   RP&SE 500
       Foundations of Rehabilitation Counseling
           An entry-level course in the Master’s program for
            rehabilitation counselors. It teaches:
               Legislative history of profession
               Trends in societal attitudes and behaviors toward
                individuals with disabilities
               Ethical issues of the profession
               Vocational rehabilitation systems and practices
               Roles and functions of rehabilitation counselor
Structure of the Course:
Pre-Collaborative Group Work
    Class has been structured for past several years as a:
        Hybrid in-class/web-based course
        Mix of guest speakers, didactic presentations, small group
         discussions and five in-class debates
    Five debate topics
        Is history an inevitable story of progress and improvement?
        Has the Americans with Disabilities Act been successful?
        Should a person who makes bad life choices be eligible for
         limited federal-state vocational rehabilitation funding?
        Should the feeding tube have been removed from Terri
        Should sheltered workshops be closed?
Rationale for Introducing
Collaborative Group Work Exercises
     To encourage students to provide constructive criticism
      to one another so as to enhance
         Each participant’s level of participation in debates
         Each participant’s quality of participation in debates
         The overall team performance of the debates
     To improve self-reflection and self-appraisal about group
      collaboration skills
     To provide students with effective strategies in working
      with a diverse group under a high stress (time limited),
      high demand (grade involved) situation that
      approximates case planning meetings and/or multi-
      agency, multi-disciplinary case treatment conferences
   27 students
       5 master students
       22 undergraduates
   Gender
       Male-5
       Female-22
   12 report prior experience participating in
    classroom debates
Method: Assignment to Collaborative
Work Groups
   Developed 4 teams so as to create small
    groups of 6-7 students to collaborate in 5
    class debates
       Randomly assigned students to collaborative work
       Group membership remained constant throughout
Method: Structure of Debates

   Debates were administered in two separate
    classrooms led by the Instructor or the TA
   Individual teams rotated so that each team had the
    opportunity to debate each other team and also be
    evaluated equally by the TA and Instructor
   Each team member had 5 minutes to present their
    initial argument and 2 minutes for rebuttal
   Instructor/TA and 1 or 2 members of each team
    served as judges for the debate
Collaboration: Defined
   Week prior to each class debate
       On-line threaded group discussion was
        mandated during which time students were to
        expected to plan, critique and refine their
        arguments for the upcoming in-class debate.
        Students needed to:
           Post early and often
           Comment on others postings
           Edit/modify their arguments based on comments
            from others
           Submit written argument
            document in dropbox prior to debate
Collaborative Group Work
   Self and Peer Assessments
       Students completed on-line (Learn at UW) assessment of
        collaborative group members’ behavior during week
        preceding class debate (5 separate assessments-one after
        each of the 5 debates)
   Class evaluation
       Students completed in-class paper-pencil self report of
        learning experience of just completed week of collaborative
        group work preparation and debate (5 separate evaluations-
        one after each of the 5 debates)
   Course evaluation
       Students completed pre-post in-class course evaluation
Collaborative Group Work
Monitoring: Fidelity
   Instructor/TA Monitoring and Prompts
       Instructor and TA comments made within the
        threaded discussions to prompt, probe and direct
        participation and guide argument formation and to
        provide constructive feedback about specific types
        of collaborative group work activity evidenced by
        group members
Collaborative Behavior: Feedback

   Report Card includes:
       Self- and Peer-Assessment of Collaborative Behavior
        During Week Preceding Class Debate
       Number of Postings Made
       Number of Posting Read
       Instructor grade for in-class debate argument
       Instructor grade for written argument submitted to Dropbox
       Peer written feedback about Collaborative Behavior across
        3 Weeks of Collaboration
   In-Class, Small Group, Face-to-Face Verbal
    Feedback from and to Collaborative Group
    Members (no Instructor or TA present)
Outcome Evaluations: Instructor
   Verbal Performance
       Instructor/TA ratings of individual in-class debate arguments (0-2
        scale) (10-15 minute per student)
   Written Performance
       Instructor/TA ratings of individual debate argument submitted to
        Learn at UW Dropbox prior to debate (0-2 scale) (1-2 pages in
   Participation
       Number/Percentage of Postings within Group Threaded
       Number/Percentage of Postings Read
       Timeliness of Postings (Latency of Response)
Enjoyed collaborative debate activity?

          I enjoyed the collaborative debate activity

         Pre   Debate 1 Debate 2 Debate 3 Debate 4      Post

Increased understanding of materials?

         Working in a group increased understanding of material

         Pre    Debate 1 Debate 2 Debate 3 Debate 4       Post

Helped understand the multiple
  perspectives on the issue?

         Working in a group helped me to understand the multiple
                        perspectives on the issue
         Pre    Debate 1 Debate 2 Debate 3 Debate 4       Post

Like the idea of rate self and teammates
in terms of collaboration process?

          I like the idea of rating myself and my teammates in terms
                           of our collaboration process
          Debate 1    Debate 2     Debate 3     Debate 4      Post

Feedback from teammates enhanced my
writing and the strength of my debate?

           The feedback from my group members enhanced my
                  writing and the strength of my debate
          Debate 1     Debate 2        Debate 3   Debate 4

On-line collaborative process more
efficient and productive than face-face?

           I found on-line collaborative group process more efficient
             and productive than having to meet my team face-face

          Debate 1    Debate 2     Debate 3     Debate 4       Post

Level of comfort
           Level of comfort with the teaching methods used in this
          Debate 1   Debate 2     Debate 3    Debate 4       Post

                                Level of comfort

Post: 62% in 2008 vs. 85% in 2007
Level of satisfaction

          level of satisfaction with the teaching methods used in this
          Debate 1    Debate 2      Debate 3      Debate 4     Post

                                 level of satisfaction

Post: 54% in 2008 vs. 90% in 2007
Examples of Qualitative Assessment:
Collaborative Work Group Member
Feedback to Alice
   I appreciate your hard work and effort in the group projects. I
    believe the group would benefit from an increased confidence in
    your work.
   Alice is generally very timely with her discussion posts and is
    very good about commenting on others’ write-ups.
   I feel she has shown to be very punctual and precise in her
    arguments. She brings a relaxed feel to the group which I
    appreciate. I feel she’s on the higher end in terms of postings
    and feedback.
Examples of Qualitative Assessment: Alice
   I thought you did a good job helping to
    organize our group and you gave good
    feedback. Way to be on top of things!
   You are always great about reading everyone’s
    write ups and giving constructive feedback.
    Very encouraging and helpful and willing to put
    in the extra work. Perhaps could post a little bit
Qualitative Assessments Related to Leadership:
Examples of Statements Made to Different
   I think you’re a really great leader! You do a really good job of
    posting early and providing lots of really helpful feedback for
    other group members.
   I think your leadership is great – I would suggest using this
    leadership to give more feedback on discussion boards.
   You take leadership where necessary. I like that you aren’t
    overwhelming and you get the job done.
   ______ did an excellent job starting off our group when no
    one knew exactly what was expected.
Qualitative Assessment: Organization

   The only suggestion I can think of might be to work on
    organizing your arguments a little more: You usually find
    really good points and good data for backing up what you’re
    saying, but sometimes I have a little trouble following your
    train of thought.

   I would like to see more organization in what you say in the
    actual debate. It always seems you have a lot of great ideas,
    but they are not broken down into simple arguments to follow
Qualitative Assessment: Strength of Argument
   I feel like you really understand the debate materials and
    topics and you always come up with really solid arguments.
   I have also noticed a few times where your arguments have
    kind of seemed to summarize the points other people included
    in their arguments, and I feel like this is somewhat unfair to
    some of the other members who worked hard on their own,
    original arguments.
   I appreciate that your debates are strong and that you do your
Qualitative Assessment-Timeliness
   You’re really good at getting your debate points
    posted early
   You are fantastic at very timely feedback on-line and
    that is very helpful when it comes to tightening up
    our points.
   I know things are really busy and everyone’s
    schedules are hectic, but sometimes I feel a little
    bothered by how late you post your debate points,
    particularly if you’re asking for feedback.
   Although I understand you have a very busy
    schedule it was difficult when you couldn’t’ make the
    debate on time.
Qualitative Assessment-Feedback
   You seem to provide a lot of helpful feedback
    to other group members.
   Good feedback but I think she could be a little
    more assertive when she has an idea
   You provided some great constructive
   I think maybe you could provide our team
    members with a little more constructive
    feedback rather than just the blanket
    statement of “good job”.
Qualitative Assessment-Feedback
   You’re really responsive to other people’s feedback to your
   I would have liked to see more discussion from you online,
    and I think you, along with the rest of us, could have benefited
    from incorporating some feedback into your arguments.
   She was helpful to others and not afraid to ask for help
    herself, which made me feel better about my own confusion
    with the material.
   She did not hesitate to revised her debates based on our
Other Student Comments
   Let someone else have the first word for once. I
    admire that you get your work done quickly, but
    others are slower at reading and response.
   I think your participation, responses, and leadership
    were all on the right level exactly! I’d love it if you
    could encourage and support some of the other
    group members to do the same!
   I think it would be great to see your increased
    confidence and initiative in these projects.
   Although you have great input, maybe holding off
    and letting others post first might be beneficial to the
Other Student Comments
   You seem like a really hard worker and I like the way you
    really get involved in the debates.
    I think she always has a positive attitude and willing to take
    on sections or arguments (when in the debate) that are
   I am pleased with the rotation of duties.
   I see great potential in your thoughts!
   You do a really great job and I think you’re a really strong
    team member.
Student Comments—Written vs. Verbal Feedback

   “Thanks for allowing us to have time to hash
    this all out face to face. I will feel much more
    comfortable presenting my constructive
    criticism in person. This will also provide an
    opportunity to collaborate as a team to
    stratagize (sic) ways in which we can support
    one another in achieving these suggestions.”
Students’ Assessment about Assessment

   “I don’t really have anything to say about
    anyone specific. I appreciate those who are
    able to lead our group and begin the
   “Overall, I am very pleased with my group
    and their effort. I know that this is not
    necessarily what Dr. Gervey was looking for;
    however, if there is nothing to say, why make
    up something?”
Self- vs. Peer- Assessment
    •62% of students (N=16) agreed that s/he was tougher
    when rating self than when rating teammates
       •Subjective data

Pattern for rating                Self-rating
     Self vs. Peers                   = Peer-ratings
 Below: 6 (23%)                   Debate 1: 11 (42%)

 Even: 11 (42%)                   Debate 2: 15 (60%)

 Above: 2 (8%)                    Debate 3: 13 (52%)

 No Pattern: 7 (27%)              Debate 4: 11 (44%)

   +/- 2.5 points allowed           +/- 2.5 points allowed
Post- Survey
   The requirement of having to give feedback to
    my group members was helpful
       42% (N=11) for agreement

   Written vs. Verbal feedback from my group
    members enhanced my class performance
       Written 58% vs. Verbal 73%

   I would like other courses that I attend to use
    collaborative projects as a learning method
       54% (N=14) for agreement
Relationship between Process and
Performance: Where is the Beef?
   Peer ratings of group collaboration and
    instructor generated performance ratings
       Inconsistent findings
           Significant correlations found only with
               Debate 2 (r = .435, p <.05)
               Debate 3 (r = .601, p <.01)
To Collaborate or Not:
Students’ Choice
   For final project, students were given a
    choice to collaborate
       Only 6 of 27 elected to work collaboratively
       Reasons given for working independently
           52% - more flexible and time-saving
           17% - the site I assessed near my home
           13% - learn more by doing it solo
            9% - hard to co-work with others
           9% - other

   Did we actually make the student experience of the
    class worse by adding the collaborative group work
    process ?
       Students’ overall satisfaction with course dropped
        significantly from year prior
       Students voiced quite a bit of displeasure with
        collaborative group work process
       Students did not particularly appreciate quantitative report
        card style feedback- reported too many numbers and
   Are we over-estimating the enjoyment and utility of
    Internet in terms of replacing or augmenting in-class,
    face-to-face interactions?
       Strong student preference voiced for face-to-face verbal
   Did we select the correct activity to teach and
    measure collaboration?
       Debates appeared to be perfectly suited
   The frequency and various methods of assessment
    seemed excessive for both students and
       Data overload
       Diminishing returns?
Implications/Future Research
   Do we need more upfront discussion about students’
    pre-class attitudes and experiences concerning
    collaborative group work?
   Should we expect collaborative group work
    exercises to yield the results that we had anticipated
       Enhanced class participation
       Enhanced quality of work
       Enhanced match between self and peer assessment of
        collaborative group work skills
   Providing students with an individual and group
    grade for their collaborative group work does not
    appear to resolve grading concerns

   Possible confounds
       Instructor/TA differences between years
       Cohort effect- different set of students
       Was it the debate or the collaborative group work
        activity that was the source of displeasure
   Rating instruments and methods not tested
    for reliability or validity
Thank you
 Special thanks go to DOIT and
 ENGAGE for providing all needed
 consultation and technical supports.

Shared By: