Docstoc

analysis

Document Sample
analysis Powered By Docstoc
					                                                                                            1


ONLINE ADMINISTRATION OF FCQ’S IN GRADUATE SECTIONS WITH 10 OR
FEWER STUDENTS

Perry Sailor
University of Colorado at Boulder
Office of Planning, Budget, and Analysis
September 2006

Summary
In response to anonymity concerns expressed by the United Government of Graduate
Students (UGGS), the Chancellor’s FCQ Advisory Committee recommended a pilot test
of online administration of FCQ’s in graduate-level sections with 10 or fewer students
enrolled. The pilot study was carried out in the spring semester of 2006, involving 1,274
forms from 273 sections. Results showed no meaningful difference (less than a tenth of a
point) in mean course ratings or instructor ratings, or in standard deviations, compared to
paper administration in similar sections in spring 2005. However, return rates were lower
by 8 percentage points when all graduate sections with fewer than 10 students were
compared, and 12 percentage points when the comparison was restricted to the same
courses taught by the same instructors in both semesters. The average response rate
remained over 70% overall.

When asked if online administration should continue to be used in graduate sections of 10
or fewer students, 94% of those who responded said “yes.” The most popular reason
cited in the open-ended follow-up item was ease/convenience, followed in order by not
using valuable class time; having more time flexibility and/or more time to make
thoughtful comments; and anonymity.

Detailed description of the study

Initial plan
The report of the Chancellor’s FCQ Advisory Committee (August 2005 revision), in the
context of a discussion of confidentiality, recommended testing the strategy of
“automatically flagging all graduate courses with 10 or fewer enrolled students to have
the FCQ process be done entirely online.” (p. 13). They noted that this was “an issue of
high importance to graduate students, brought to us by UGGS” (p. 43), and recommended
this be done on a pilot basis in 2005-06.

In an e-mail exchange between Mike Grant, the chair of the committee, and Lou
McClelland, the director of Institutional Analysis in the office of Planning, Budget, and
Analysis (PBA), which administers the FCQ program, it was decided that

   The pilot would be implemented in fall, 2005. Due to time pressures, this was later
    moved to spring, 2006.
   The pilot would include only sections with 10 or fewer enrolled in courses numbered
    5000 or higher; that is, it would not include sections where the primary/parent section
    is undergrad and there are some graduate students in it.
   Participation would be mandatory. Before each term’s administration, FCQ
    coordinators edit online course lists PBA provides. In the typical process, a few
C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                            2


    courses are pre-marked for online FCQ administration (typically only courses that are
    themselves online are marked). The coordinator can mark others and can unmark any
    that are pre-marked, thereby shifting to paper. It was decided to pre-mark all sections
    of courses numbered 5000 or above, with enrollment of 10 or less (that is, all selected
    sections), as online in the course lists coordinators get. Further, it was decided not to
    allow coordinators to unmark the online toggle for those courses, i.e., to make online
    administration mandatory, on the authority of the chancellor’s committee.


Notifying faculty and administrators
On January 24, 2006, a memo was e-mailed from committee chairman Grant to all
Boulder campus teaching and research faculty, deans, directors, department chairs, and
system administration. The memo informed recipients that spring 2006 FCQs for
graduate sections with 10 or fewer students would be administered online, that instructors
of the affected sections would be notified by e-mail in mid-February, and that
participation was mandatory. It also spelled out how students would be notified,. For the
text of this e-mail, see Appendix B. The gist of the message was also included in a Buff
Bulletin announcement, and posted on the PBA FCQ website.


Reactions to the plan
The day after the message went out, January 25, an assistant professor in journalism sent
an e-mail to Grant, saying she was “troubled” by the idea of administering the FCQ
online, because she believed fewer students would respond, and the respondents would be
disproportionately those who have very strong feelings, either positive or negative. A
lower response rate, she felt, would lessen the validity of the results, which are especially
important to the untenured.

Grant responded that (1) the present effort was merely a pilot, the results of which would
be examined carefully before any permanent changes are considered; (2) based on
previous experience with online FCQs, the committee and the FCQ office were also
concerned about a lower response rate, and would take steps to try to increase it; (3)
previous experience suggests that the distributions of student responses are very similar
in paper and online administrations; (4) the recommendation to pilot the online
administration only in graduate classes with fewer than 10 students came in response to a
very strong recommendation from the United Government of Graduate Students (UGGS),
which had high concerns about the need for confidentiality of students’ FCQ ratings.

Grant also forwarded the journalism professor’s e-mail to the entire committee, so they
could respond independently if they wished; to another journalism instructor who had
expressed similar concerns; to John Stevenson, associate dean of the graduate school; and
to the UGGS.

In mid-February, Grant received another e-mail from another journalism instructor. The
instructor had heard from a student reporter that if the pilot online administration with
graduate sections of fewer than 10 students was successful, all FCQs would be
administered online beginning in the fall of 2006, and wanted to know if this was true.
Grant responded that this was not true, that the pilot was not being considered as a
C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                                      3


stepping-stone to universal online administration, but only as a pilot for future
consideration to apply to that subset of classes only.

According to PBA’s FCQ coordinator, these were the only contacts from instructors with
concerns about this pilot online administration. FCQ Advisory Committee chair Mike
Grant also confirmed he received no others, either before, during, or after administration.

Online administration logistics

A brief summary of the logistics of online FCQ administration:

   Students enrolled in participating course sections get a customized e-mail invitation.
    with a link (URL) to a web form for the course.
   The FCQ online website opens a few days before regular (paper) FCQ administration
    begins, so that a weekend is always included, and closes just before finals begin. This
    corresponds closely with the paper FCQ administration period.
   The questions in the FCQ form used online are identical to those used on paper.
   Student responses are anonymous.
   A reminder e-mail is sent to each student who did not respond during the first week.
    When the number of responses is less than 50% of enrollment, we send another
    reminder just before the last day of collection.
   For this pilot administration, the open-ended question “What did you like and not like
    about this method of collecting course evaluations?” was replaced by a two-part
    question. The first, “This online FCQ administration is a pilot program for all spring
    2006 graduate sections with 10 or fewer students. Do you think that online FCQs
    should be used in the future?” was closed-ended, with a “yes/no” response button.
    The second, “Why or why not?” was open-ended.

A more detailed description of how online FCQs are administered is at

http://www.colorado.edu/pba/fcq/form/webprocs.html, and is also attached as Appendix
A.

Results
There were 273 graduate sections of 10 or fewer students that participated in online
administration. Section means, standard deviations, and return rates from these sections
were compared to those from the 286 graduate section of 10 or fewer students from
spring, 2005 that filled out paper FCQs in class. The unit of analysis was the section –
that is, the means and SDs were calculated over 273 section means, not over the 1,274
forms in the sections.

Measure                      2005 (Paper) Mean                      2006 (Online) Mean   Difference
Course Rating Section Mean         3.47                                   3.40             -.07
Instructor Rating Section Mean     3.61                                   3.54             -.07
Course Rating Section SD           0.54                                   0.59              .05
Instructor Rating Section SD       0.47                                   0.51              .04
Response Rate                      79%                                    71%            - 8% pts.

C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                                      4


None of the differences was statistically significant, except response rate. Moreover, the
effect sizes for course and instructor ratings (difference in means between online and
paper, divided by SD of the paper administration) were a minuscule -0.13 for both course
and instructor ratings.

There were 51 cases over the two terms in which the same course was taught by the same
instructor, once using paper FCQ administration, once using online. Here are the results
from these sections:

Measure                      2005 (Paper) Mean                      2006 (Online) Mean   Difference
Course Rating Section Mean         3.50                                   3.47             -.03
Instructor Rating Section Mean     3.64                                   3.62             -.02
Course Rating Section SD           0.52                                   0.60              .08
Instructor Rating Section SD       0.46                                   0.47              .01
Response Rate                      91%                                    79%            - 12% pts.

Again, only the difference in return rate was statistically significant. The effect sizes for
course and instructor ratings were even smaller than those for the overall population, -.06
and -.04 for course and instructor ratings, respectively.

In summary, administering the FCQ online had no effect on average ratings, but did
result in a somewhat lower response rate. Still, the average response rate was better than
70%, and 60% of the sections had response rates of 70% or better.

A box plot of response rates for 2005 (paper) and 2006 (online) is shown in Appendix C.
Course and instructor rating box plots are in Appendix D.

Student Reactions
Students completing the online FCQ were told at the end of the form, “this online FCQ
administration is a pilot program for all spring 2006 graduate sections with 10 or fewer
students,” then asked the following questions:

        Do you think that online FCQs should be used in these sections in the future?
         (Students could select “Yes” or “No”)

        Why or why not? (Open-ended response)

Of the 1,274 forms returned, 932 responded to the Yes/No question, with 873 (94%)
answering “Yes.” In addition, 614 forms had responses to the open-ended “why or why
not” question. The most common responses involved

   ease and/or convenience
   not using valuable classroom time
   some combination of ability to make better, more thoughtful comments; lack of time
    pressure to complete the form; or the ability to complete it on one’s own time

Anonymity, the reason that was the impetus for the pilot study, was only fourth among
reasons mentioned for continuing to use online forms.
C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                        5



Many students spontaneously commented that online administration should be extended
to all graduate courses or to all courses.


Reason for Continuing Online                                N (Percent)
Ease/Convenience                                            226 (37%)
Doesn’t use valuable class time                             146 (24%)
No time pressure/
More considered or thoughtful comments/
Ability to complete on own time                             130 (21%)
Anonymity                                                   118 (19%)
Speed/Efficiency                                            85 (14%)
Conserves paper                                             48 (8%)




C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                           6


APPENDIX A: Detailed Description of Online Administration Logistics

PBA sends students enrolled in participating course sections a customized e-mail
invitation with a link (URL) to a web form for the course and a unique, random 7-digit
"passcode" number. Students taking more than one course receive only one passcode,
which will work for all courses the student is taking that term. A passcode is good for one
term only.

To respond, each student must go to the appropriate URL (for a course-section) linked
directly from the e-mail, or to an alternate all-courses URL where he selects the course-
section from a list and enters the passcode by hand. The passcode is used to verify
enrollment in the course and to store the responses temporarily, to avoid the possibility of
multiple responses from one person. If multiple forms for a single course are submitted
from a single passcode, we take the last submitted form with any items answered.
Students who lose their passcodes should go to the all-courses URL. At that site the
student may verify enrollment by providing birth date and last four digits of the student
ID.

For regular term-long courses we open the FCQ online website a few days before regular
(paper) FCQ administration begins, so that a weekend is always included, and close it just
before finals begin. This corresponds closely with the paper FCQ administration period,
which also closes before final exams begin. It provides students with approximately eight
days, including a weekend, to respond. The website is always closed before grades are
posted.

The questions in the FCQ form used online are identical to those used on paper, with the
addition of a final narrative response question, "Comments for FCQ Administrators":
"What did you like and not like about this method of collecting course evaluations?"
While student ID numbers are used to obtain students' e-mail addresses, and names are
kept in case students contact us about having lost or not received their passcodes, they are
never associated with students' responses. Student ID's and names are never stored on the
web server where collection takes place. Student responses are anonymous.

After about one week of collection, the passcodes of the responses received are used to
mark students' records in our database. Each respondent's student ID, name, and e-mail
address are deleted from the database, breaking the association between the passcode and
an individual student. A reminder e-mail is then sent to each student who did not respond
during the first week. When the number of responses is less than 50% of enrollment, we
send a second reminder just before the last day of collection.

See sample e-mail messages and reminders.

When collection is complete, all student ID's, names, and e-mail addresses are deleted
from the database, so that no one can associate any response with a particular student, or
determine who did or did not respond. The pass codes are generated by a process that
ensures that no one, not even someone who knows the generation procedure, can link
student ID's to pass codes. The pass codes are 7-digit numbers, not consecutive, so would
be extremely tedious to guess.
C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                                  7



For reporting and data storage we then assign a sequence number to each set of
responses. Sequence numbers are used to keep ratings and comments from each student
together in our files, and are unrelated to the student's ID, so that they will not sort in the
same order as the instructor's list of the ID's.

For this pilot administration, the open-ended question “What did you like and not like
about this method of collecting course evaluations?” was replaced by a two-part question.
The first, “This online FCQ administration is a pilot program for all spring 2006 graduate
sections with 10 or fewer students. Do you think that online FCQs should be used in the
future?” was closed-ended, with a “yes/no” response button. The second, “Why or why
not?” was open-ended.




C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                        8


APPENDIX B: Text of Memo Notifying Instructors of Mandatory Online FCQ
Administration.


         SUBJECT: Online FCQs for small graduate classes

         Re:       Online FCQs for graduate sections of 10 or fewer students

         As a pilot program approved by Chancellor DiStefano, online FCQs will
         be provided for all spring 2006 graduate sections with 10 or fewer
         students. This program is intended to enhance the anonymity of
         graduate student responses on the FCQ, by excluding the possibility that
         students can be identified from their handwriting in these small sections.

         All graduate sections of 10 or fewer students will be provided with online
         FCQs, and instructors of the affected course-sections will be notified by e-
         mail in mid-February. Graduate sections only will be affected – sections
         that have a graduate/undergraduate mix will not be included. When it is
         time for FCQ administration, students in these sections will be notified
         with an e-mail containing instructions and a passcode-protected link to
         the online FCQ collection website. Students who do not respond also get
         at least one reminder e-mail. For full details on how online FCQs are
         administered, see http://www.colorado.edu/pba/fcq/form/webprocs.html

         This pilot program is being conducted per the August 2005
         recommendations of the Chancellor’s FCQ Advisory Committee approved
         last September by Chancellor DiStefano, and participation is mandatory.
         For more details on the Committee’s report, see
         http://www.colorado.edu/pba/fcq/fall06changes.html. For the Chancellor’s
         approval letter, see http://www.colorado.edu/pba/fcq/chancellor.pdf.

The gist of the above message was also included in a Buff Bulletin announcement, and
posted on the PBA FCQ website.




C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                        9

Appendix C: Box plots of response rates for 2005 (paper) and 2006 (online).

 – Entire population of sections.

            |
          1 +                 +-----+                |
            |                 |     |                |
            |                 |     |                |
            |                 |     |                |
        0.9 +                 *-----*             +-----+
            |                 |     |             |     |
            |                 |     |             |     |
            |                 |     |             |     |
        0.8 +                 | + |               |     |
            |                 |     |             |     |
            |                 +-----+             *-----*
            |                    |                |     |
        0.7 +                    |                | + |
            |                    |                |     |
            |                    |                |     |
            |                    |                |     |
        0.6 +                    |                |     |
            |                    |                |     |
            |                    |                +-----+
            |                    |                   |
        0.5 +                    |                   |
            |                    |                   |
            |                    |                   |
            |                    |                   |
        0.4 +                                        |
            |                                        |
            |                                        |
            |                      0                 |
        0.3 +                                        |
            |                                        |
            |                                        |
            |                                        |
        0.2 +                                        |
            |                                        |
            |
            |
        0.1 +
            |
            |
            |
          0 +               0           0
                ------------+-----------+-----------
YEAR                      2005        2006

Top edge of box = 75th percentile
Bottom edge of box = 25th percentile
Middle horizontal line = Median (50th percentile)
+ = Mean
Vertical lines = Extent of data (to 1.5 interquartile ranges)
0 = Data values 1.5 to 3 interquartile ranges


C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                        10


-- Same sections taught by same instructor both years
      |
          1 +                 *-----*             +-----+
            |                 |     |             |     |
            |                 |     |             |     |
            |                 | + |               |     |
        0.9 +                 +-----+             |     |
            |                    |                |     |
            |                    |                *-----*
            |                    |                |     |
        0.8 +                    |                | + |
            |                    |                |     |
            |                    |                |     |
            |                    0                |     |
        0.7 +                                     +-----+
            |                      0                 |
            |                                        |
            |                                        |
        0.6 +                                        |
            |                                        |
            |                                        |
            |                                        |
        0.5 +                                        |
            |                                        |
            |                                        |
            |                                        |
        0.4 +                                        |
            |                                        |
            |                                        |
            |                                        |
        0.3 +                                        |
            |                                        |
            |                                        |
            |
        0.2 +
            |                                          0
            |
            |
        0.1 +
            |
            |
            |
          0 +               *           0
                ------------+-----------+-----------
      YEAR                2005        2006


Top edge of box = 75th percentile
Bottom edge of box = 25th percentile
Middle horizontal line = Median (50th percentile)
+ = Mean
Vertical lines = Extent of data (to 1.5 interquartile ranges)
0 = Data values 1.5 to 3 interquartile ranges
* = Data values more extreme than 3 interquartile ranges



C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM
                                                                                        11

Appendix D:          Box plots of course and instructor ratings.

         |
       4 +            |           |        +-----+     +-----+
         |            |           |        |     |     |     |
         |         +-----+     +-----+     |     |     |     |
         |         |     |     |     |     |     |     |     |
    3.75 +         |     |     |     |     *-----*     |     |
         |         |     |     |     |     |     |     *-----*
         |         *-----*     |     |     | + |       |     |
         |         |     |     |     |     |     |     | + |
     3.5 +         |     |     *-----*     |     |     |     |
         |         | + |       |     |     |     |     |     |
         |         |     |     | + |       +-----+     |     |
         |         |     |     |     |        |        +-----+
    3.25 +         |     |     |     |        |           |
         |         +-----+     |     |        |           |
         |            |        +-----+        |           |
         |            |           |           |           |
       3 +            |           |           |           |
         |            |           |           |           |
         |            |           |           |           |
         |            |           |           |           |
    2.75 +            |           |           |           |
         |            |           |           |           |
         |            |           |           |           |
         |            |           |           |           |
     2.5 +            |           |           |           |
         |            |           |                       |
         |            |           |           0           |
         |            |           |           0           0
    2.25 +                        |           0           0
         |            0           |                       0
         |            0           |           0
         |                        |
       2 +            0           |           0           0
         |
         |                        0
         |                                    0           0
    1.75 +            0           0                       0
         |                                                0
         |                        0                       0
         |                        0
     1.5 +                        0
         |
         |
         |                        0
    1.25 +                        0
         |
         |
         |
       1 +            *                                   *
         |
         |
         |
    0.75 +
         |
         |
         |
     0.5 +                                                *
          ------------+-----------+-----------+-----------+-----------
YEARTERM             20051       20061       20051       20061
     var            Course      Course       Instr       Instr

Top edge of box = 75th percentile
Bottom edge of box = 25th percentile
Middle horizontal line = Median (50th percentile)
+ = Mean
Vertical lines = Extent of data (to 1.5 interquartile ranges)
0 = Data values 1.5 to 3 interquartile ranges
* = Data values more extreme than 3 interquartile ranges
C:\Docstoc\Working\pdf\188b021e-158a-47c6-9631-2f8bc9a5de5a.doc_8/27/2012_11:44:12 AM

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:8/27/2012
language:Unknown
pages:11