A User s Guide to Completing Your Scantron Job Using Scanbook and Webtron http darwin bio uci edu local Webtron webtron html Webtron program and docum by owp20669

VIEWS: 79 PAGES: 11

									   A User's Guide to Completing Your
    Scantron Job Using Scanbook and
                Webtron

http://darwin.bio.uci.edu/local/Webtron/webtron.html


               Webtron program and documentation by
                       Garr Updegraff, Ph.D.
                   School of Biological Sciences

              User's Guide Written and Compiled by
                        Keith M. Vogelsang
             Dept. of Ecology and Evolutionary Biology
                    Revised: 18 November 1999

                           With funds from
           Instructional Technology Mini-Grant for Faculty
                             to Ann Sakai
           Department of Ecology and Evolutionary Biology
                     University of California-Irvine
Table of Contents

Introduction                                   3

Diagram of Scantron Reader                     4

Preparing to read exam forms                   5

Dealing with form feed problems                8

Editing raw data files                         9

Using Webtron to score your edited data file   10

Appendix 1: The Webtron FAQ                    12

Appendix 2: Sample input file                  15

Appendix 3: Sample output file                 18




                                                    2
Introduction

         Many large lecture courses at UCI rely at least in part upon machine-graded
exams to evaluate students. In addition to scoring student exams, grading software can
also provide instructors with more information about their multiple choice tests, with
feedback on exam questions to determine whether individual questions were more or less
informative in assessing student knowledge. This information can be used to eliminate
poor questions and design better tests in the future by understanding what concepts and
types of reasoning are particularly difficult for students. This notebook documents how
to use the Scantron reader to turn student responses on Scantron forms into computer
files, and how to use the Webtron software with these files to analyze test results.
        Completing a Scantron job is a two-part process of entering data through the
student forms using the Scantron 8200 machine, and data analysis using the Webtron
program. The Scantron 8200 form reader is interfaced with an old AST PC in Bio Sci II
Room 5205, and a MSDOS program called Scanbook that is installed on the AST PC in
that room. Data analysis (computation of student scores on the exam, class means, etc.)
uses a Web-based program called Webtron.
         The Scantron reader and AST PC are used to process the completed examination
forms and save the information into a raw data file. This raw data file needs to be saved
onto a 3.5" floppy diskette in order to use Webtron and complete the data analysis portion
of the job. An earlier data analysis program called Scantrax required separate raw data
files for each version of an exam to be scored. Webtron does not have this limitation, that
is, all exam versions can be stacked on top of each other and processed as a single run.
Additionally, exams can be scored using any on-campus networked PC with Internet
access; all that is needed is the diskette containing the raw data file.
        Webtron offers several advantages, including the ability to easily rescore exams,
the ability to include multiple correct response questions, and the ability to easily delete
ambiguous questions from being scored. Output includes the following for each version
of the test as well as the class as a whole:
1. A list by ID of student exam scores (raw and percent of total) as well as individual
student responses on each question,
2. Item analysis. For each question, the number and percent of students who answered
the question correctly, as well as the percent of students in the top and bottom quarters of
the class who answered correctly and the number of students who chose each possible
answer. This information can be used to judge the quality of the individual question.
3. A histogram of the test scores with test mean and standard deviation, with the score,
the numbers of students obtaining that score, and the class ranking of students obtaining
that score.




                                                                                               3
Preparing to use the Scantron 8200 reader

        The two important hardware components for data collection are the Scantron 8200
reader and the AST PC. If these components are already powered up, it is best to initialize
both devices by switching the Scantron reader off for a few seconds and then switching it
back on, and rebooting the PC by holding down the keys [Ctrl]-[Alt]-[Delete] in unison.
Although this initialization routine is not critical, it will free up PC memory and clear the
interface buffers of any data or error messages remaining from previous jobs. When
rebooting the PC, be sure to remove any diskettes in the floppy drive (drive A).


Diagram of the Scantron 8200

             Holding tray      Tension Adj.       Power switch          Forms to be scanned



                                   SCANTRON


                                                              ID#                     Name
                                                              |||||||||||||||||||||||||||||||||||||




                                                   8200




- SCANTRON 8200 -    :: UNLD                  1            End

READY               ::

                                              2           Select

                                                                                                      Plastic guide to be placed on
                                                                                                      top of forms




The Scantron equipment is housed in BSII room 5205. Two of Bio Sci’s computer
support people, Barry Denlinger (x3507) and Marko Popovich (x4937), share this office.
Contact them if you experience technical difficulties with this equipment.




                                                                                                                       4
Preparing to read exam forms

     1. Stay organized. Webtron works the same regardless of how the data file is
        ordered. Although it is not necessary to sort the exams before processing
        them, sorting is highly recommended to facilitate subsequent auditing and
        form retrieval. Therefore, two levels of sorting are recommended. First, exams
        should be sorted according to their version. For example, if three exam
        versions were administered, then group all of version A forms together,
        version B forms together, and version C forms together. Within these groups,
        the forms may also be sorted by the first digit of the ID number. Since each
        form becomes a separate line in the raw data file, pre-sorting can be a useful
        way to track down problems. Regardless of your pre-sorting preferences, it is
        important to keep the forms organized according to how they were actually
        scanned, since this order corresponds directly to the raw data file.
     2. Webtron is very flexible in how multiple exam versions can be managed.
        Instructors can either write exams so that students must bubble in a response
        to a version indicator question, or exam versions can be indicated in some
        other way that will not be read by the Scantron machine. For example, the last
        question on the exam could instruct the student to fill in answer "B" for
        question 50, thus indicating one of five possible versions (the Scantron forms
        are lettered A-E), which makes question 50 the version indicator question.
        Version indicator questions are an elegant solution to the problem of grading
        multiple versions. However, instructors may not want to make students aware
        of a particular exam version, or instructors may prefer another sorting scheme
        that is not machine readable. Instructions for both schemes are included
        below.
     3. All exam keys must be completed using the ID# 199999999 for exam version
        one, ID# 299999999 for exam version two, and so on. These numbers must be
        completed in the space reserved for student ID#, regardless of whether an
        instructor uses a version indicator question.


        If you use a version indicator question:
     4. The key(s) can be placed anywhere in the stack of forms as long as they
        precede the scantron forms that refer to that key (Webtron does not otherwise
        care), but it is recommended that all keys appear first. Place all keys on top of
        your first stack of forms so that the Scantron reads the keys first and puts them
        at the beginning of your data file.
     5. On your version indicator question, either leave the column blank or fill in the
        letter that corresponds to that particular version. This field will be hand-edited
        later and replaced with a "v" in the data file, so filling in the column on the
        Scantron form serves as a convenient placeholder. What follows is a sample
        data file that uses the optional version indicator:




                                                                                         5
   Sample data file:

  19999999        DCDEBEBEDABBBAAADAEDDBBBDDEEBCCBABBAv
  29999999        EEBBEEBCBBBADCDEBDBEDDCDBEDDDCDBBBBAv
  39999999        BEEDABADCDDDBABBBAAAABBBABBBBDCDDDEBv
  47518242        AAAEBEBEBAAAABAACCDDEDBBEECEBCCAABCCD
  18149500        AEDCBBBCBABABBAAAAADEDABDDECACCBABCAD
  38363623        EAEEEEBEAABBAAAADCBEDBABDEACBECBBBCDB
  50799673        EEBBAABCAABACCDEBBBDDECDDEEDDEBBABBAB
  95801721        EAECABBDDBBACCDCEDEABACDBEBEDCDBABBBA
  02415874        BCCDABCECDDDCABBAAAAAABBABABBEEADDEBD

   The first three lines of text are the exam keys. Notice the "v" in column 37 of
   these first three lines. This character is added in the editing process to tell
   Webtron that a student's response to question 37 determines which key the
   exam will be compared to.

   If you elect not to use a version indicator question:
6. A version indicator is not necessary if there is only one version of the exam.
   If there is more than one version, it is still possible to skip a version indicator
   column by placing each key directly before the group of exams it corresponds
   to; however, this is not recommended and may not be supported in the future
   because it is easy to make mistakes. What follows is a sample data file that
   does not use a version indicator question:

  19999999        DCDEBEBEDABBBAAADAEDDBBBDDEEBCCBABBAB
  47518242        AAAEBEBEBAAAABAACCDDEDBBEECEBCCAABCCD
  18149500        AEDCBBBCBABABBAAAAADEDABDDECACCBABCAD
  38363623        EAEEEEBEAABBAAAADCBEDBABDEACBECBBBCDB
  50799673        EEBBAABCAABACCDEBBBDDECDDEEDDEBBABBAB
  51673722        EECABEBBEBAADCAEBDBEDDCDBABDDCDBAABAA
  76366898        EEABBAACEBBBCDDEEDDEBDCDDAEDDECCBAABA
  95801721        EAECABBDDBBACCDCEDEABACDBEBEDCDBABBBA

The key is followed by the exams it corresponds to. Notice that column 37 is used
as just another question.




                                                                                         6
Reading forms to create the raw data file

      1. On the Scantron machine, press [1] to lower the paper tray and load your
         forms. There is a limit of approximately 100 or so at a time.
      2. Place your forms on the tray. Be sure that no forms are folded or bent and that
         the tops contact the lower-left corner of the tray (refer to the diagram).
      3. On the Scantron machine, press [1] to raise the paper tray.
      4. Place the plastic paper guide on top of the forms. It is important not to place
         the paper guide on the forms when the tray is down because it may not be
         aligned correctly and could get caught as your raise the paper tray.
      5. You are now ready to use the AST PC and access the Scanbook program. The
         remaining commands are to be done on the PC.
      6. Select Scanbook from the text menu. If this option is not available, then either
         type "menu" at the prompt and hit [enter], or simply reboot the PC and the
         menu should load automatically.
      7. Place your formatted 3.5" diskette into the A: drive.
      8. Use the arrow keys to select "Process" once Scanbook loads.
      9. Select "Read Forms." At the top of your screen, you will see "Record Output
         File" and some path designation like "G:\Scanbook\xxxxx\xxx." You need to
         tell the PC to use the A: drive and give your raw data file a unique name. Back
         space over the default path and type in "A:\xxxxxxxx" (without the quotes)
         where the x's are your file name. Since this is a DOS-based PC, you are
         limited to eight characters, with no spaces in between. The file extension ".txt"
         will be added automatically to your name. Be sure to remember what you
         named the file, in case there are other files on the diskette.
      10. Press [F2] to direct Scanbook to download its form information. "Load File
          Name" will appear. Press [Enter] to see a sub menu where you can indicate
          your particular form type. Use the down arrow key to select "F-158.800" or
          just type in "10" and press [Enter]. Most exams are administered on this form;
          if not, then look on your particular form and be sure to select the appropriate
          one in this option.
      11. The program will find the files it needs and check the status of the Scantron
          machine. You should see the following:
         Zone 1 ok
         Zone 2 ok
         Zone 3 ok … and so on through Zone 8. At the bottom you should see the
         message "Finished downloading…and ready to read."
      12. Press [F3] to read and the Scantron machine should begin reading forms.
      13. If you get an error message instead of the "Zone ok" message, you can try
          selecting [F1] to reset (and then reload) the forms routine. Otherwise, you can
          try lowering and raising the paper tray on the Scantron machine (select [1]), or


                                                                                           7
           try powering the Scantron machine off and then back on. Be sure there are not
           too many forms in the tray; you will not necessarily receive an error message.
           Often it is unclear where the source of the problem lies. If this is the case, you
           may need to reload the Scanbook program or even reboot all components.
       14. As the forms are being read, you may experience paper jams or feed errors due
           to sticky forms. It is not a problem to read the same form twice. Scanbook will
           record both ID numbers, and you can edit out the duplicate line later. As such,
           you can rerun any forms that jammed. Press [1] to release the paper tray,
           correct the feeding problem, and then press [1] again to resume reading. With
           a bit of luck, you will only experience smooth scanning, but it is important to
           be aware of potential sources of error. You may be requested to prompt the PC
           to continue (i.e., "Push any key…") or you may have to press [2] or [CONT]
           on the Scantron machine to continue. After a batch runs out, you can press [1]
           to lower the paper tray, load another stack of forms, press [1] again to raise the
           tray, and then press [2] to continue processing the job.
       15. When all the forms have been read and you are ready to close your raw data
           file, press the [END] button on the Scantron machine (the PC refers to it as the
           "end of batch" button).
       16. SAVE YOUR WORK! Press [F10] as soon as all the forms have been read.
           The program will not prompt you. Verify that your file is being saved on the
           A: drive.
       17. After saving your raw data file, you can select the option to quit the Scanbook
           program.


Dealing with form feed problems
        Unfortunately, the Scantron reader is prone to paper feed errors. Sometimes two
or three forms are grabbed at one time, or a form becomes misaligned as it travels through
the paper guides. The plastic guide should help alleviate many of these problems, but if
errors occur too frequently, then try the following:
       1. Remove the plastic guide and put it aside
       2. Place your right thumb on top of the right side of the stack of forms, and cup
          your right hand around the rest of the feeder tray.
       3. Apply just enough firm, consistent pressure with your thumb so that only one
          form can be pulled at a time. Monitor the stack to see if lower forms are
          "creeping" towards the first set of pinch rollers, and if so, gently scoot them
          back into place. Creeping forms indicate that more than one form is about to
          be pulled through the reader. A variation of this technique is to place your
          thumb just under the top form and apply firm pressure to the rest of the stack,
          thus preventing multiple forms from being pulled.
       4. You may also need to turn the tension knob slightly, which changes the
          position of the pinch rollers.




                                                                                             8
Editing your raw data file

        Your raw data file probably needs to be edited before submitting it to Webtron.
Use a text editing program with a non-proportional font such as Notepad for this
procedure, rather than word processing programs, because the latter usually default to
using proportional fonts (where an “m” is wider than an “i”) and then the text columns
won’t line up very well. If you want to use Word, Excel, or Wordpad anyway, be sure to
change all the text to the non-proportional Courier font; then if you submit that file to
Webtron, be sure you save it as a *.txt ASCII text file because Webtron can’t read .doc
files.

       Editing allows you to correct your exam keys by throwing out ambiguous
questions or adding alternate acceptable answers. Editing is also necessary to denote
where multiple version entries should be marked in the exam. For example, exam
question 25 may direct the student to fill in reponse "C" for that column, which indicates
version three of the exam. In position 25 of the corresponding exam answer key, you
would insert the character "v" (for version). This version indicator tells Webtron to score
exams with an "A" in position 25 against the key 19999999, exams with a "B" in position
25 against the key 29999999, and so on.

        Alternate acceptable answers can also be added in the editing process. Consider
the following example key:
                19999999 DCDEBEBEDBBAABCAADEAEAACv
                19999999          A C
The second line indicates that in addition to answers "D" and "B" for questions three and
five, respectively, answers "A" and "C" are also acceptable. Additional acceptable
answers can be added with similar lines of text. The version character (“v”) appears for
question 25. After running Webtron, be sure to double check the answer key summary in
the output to make sure you marked the correct columns for your alternate answers!

        You can also correct student ID numbers that have extra digits, spaces, or question
marks. Question marks appear when the Scantron detects more than one response in a
column, such as when students select two answers to a question. If this happens, a blank
space should be inserted in place of the question mark; otherwise, Webtron will flag this
line as an error. Sometimes students change their mind and erase their initial response.
Responses must me erased completely or the Scantron will insert a "?". Locate the
original form to find out which response the student intended and make the correction.
Locating the form should be easy since the first line of the text file corresponds to the first
form that was scanned. All question marks need to be investigated and corrected before
submitting your data file for analysis

         If you experienced form feed errors during processing and had to feed some forms
through twice, you will want to edit out duplicate lines of data. Scanbook does not
recognize duplicate student ID numbers; forms are simply read and logged as another line
of text in the data file. Webtron will recognize duplicate ID numbers, and will notify you
that duplicates were discovered. These duplicate numbers will then be reassigned in your


                                                                                             9
analyzed output. It is best, however, to delete duplicate lines rather than allow them to be
analyzed. Otherwise, duplicates will contribute to the summary statistics and will still
have to be discarded if the output file is later imported into your grade book program.

      Warning: Do not insert tab characters into your data! Tabs indicate a different
number of blanks to different text editors, so Webtron forbids them.


Using Webtron to score your edited data file

        Once all the corrections have been made to the raw data file, you can use Webtron
to complete the data analysis portion of the job. Since your data file is on a 3.5" diskette,
you can use any PC on campus that has Web access. Point your web browser to
        http://darwin.bio.uci.edu/local/Webtron/webtron.html
This program is simple to use and generates a useful output that can be saved and/or
printed as a text file. Below is a figure of Webtron's user interface within a Netscape
window:




        All that is necessary to get an output is the file name of your raw data, which can
be located by clicking [Browse…] and then directing your browser to the proper drive,
directory, and file name (on PC’s, be sure to change the Browse button’s file selector
dialog box default from *.html to *.* so you can view all your files). The fields for
instructor's name, course title, year, and test are optional fields that will make it easier to
keep multiple output listings organized. Two fields are available to specify student ID
lengths. Most student ID numbers are eight digits and so 8 is the default for these fields.
However, some extension students are given a six digit ID, in which case you should
input 6 for the minimum field. Click [Process data] once you have entered your specific
information. If there are no errors, then you will receive an output similar to Appendix 3.


                                                                                              10
Errors in your data file will cause Webtron to halt processing and return a message
indicating the location and nature of the problem. Errors may include bad ID numbers or
question marks in a line of text. If Webtron encounters a duplicate ID number, you will be
warned about the problem, but it is not considered an error. Webtron will attempt to
reassign one of the duplicate ID numbers after which you can resubmit your file.
However, it is best to audit your data file to eliminate duplicate ID numbers rather than
allow multiple ID numbers to be analyzed as separate exams. Once you have addressed
the problems in your data file, be sure to save your edited file before attempting another
Webtron analysis.

        Just below the Student ID length fields is the “Truncate ID's in output listing to
final xxx digits” input field. You should use this if you plan to post Webtron’s results in a
public building or on the web. UC Irvine’s Registrar does not want full Student ID’s
posted in public with test scores or grades, but partial Student ID’s are acceptable, and
that’s what this input field is for. If you leave it blank, Webtron will print each student’s
full ID in the output listing, but if you enter, say, “5”, then Webtron will truncate each
student’s ID to the last five digits in the output listing (An aside: Because the Registrar is
now assigning new Student ID’s randomly, rather than alphabetically, it may be
acceptable to post full Student ID’s after the year 2003.)

       Webtron can process data and give results in four different ways, depending on
which option you select in the Output pull-down menu (just to the right of the Process
Data button). The default mode is to display “All analysis results”, complete with per
version summaries, combined scores listed by ID, and a histogram of score distributions.
However, you can click instead to “Just final section and # correct” to just get a
summary of raw scores listed by ID, “Just final section and % correct”" to get a
summary of scores listed by ID and percent correct, and “Just final section and
percentile” to get a summary of scores listed by ID and a student's overall ranking on the
exam. These last three options are convenient for importing your output text file into a
database or spreadsheet program.

        Specific questions about the Webtron program are addressed online in the
program's FAQ file at:
        http://darwin.bio.uci.edu/Webtron/WebtronFaq.html
or see Appendix 1 of this document. Appendix 2 is a sample input file with specific
examples of what Webtron looks for in a data file. Appendix 3 is a sample output file that
shows a histogram of all combined student scores and a summary of each student's rank
on the exam, regardless of their exam version. All Webtron output files have legends and
summary statistics that are easy to interpret. To save an output file, use your browser's
File Save feature and save the output as an ASCII text file. This text file can then be
imported into a spreadsheet program like Excel or a database program like Paradox or
Access to match up Scantron ID numbers with your records from the Registrar's office.




                                                                                           11

								
To top