Get documents about "A User Guide to Completing Your Scantron Job Using Scanbook and
Document Sample
A User's Guide to Completing Your
Scantron Job Using Scanbook and
Webtron
http://darwin.bio.uci.edu/local/Webtron/webtron.ht
ml
Webtron program and documentation by
Garr Updegraff, Ph.D.
School of Biological Sciences
User's Guide Written and Compiled by
Keith M. Vogelsang
Dept. of Ecology and Evolutionary Biology
Revised: 18 November 1999
With funds from
Instructional Technology Mini-Grant for Faculty
to Ann Sakai
Department of Ecology and Evolutionary Biology
University of California-Irvine
Table of Contents
Introduction 3
Diagram of Scantron Reader 4
Preparing to read exam forms 5
Dealing with form feed problems 8
Editing raw data files 9
Using Webtron to score your edited data file 10
Appendix 1: The Webtron FAQ 12
Appendix 2: Sample input file 15
Appendix 3: Sample output file 18
2
Introduction
Many large lecture courses at UCI rely at least in part upon machine-graded exams to
evaluate students. In addition to scoring student exams, grading software can also provide
instructors with more information about their multiple choice tests, with feedback on exam
questions to determine whether individual questions were more or less informative in assessing
student knowledge. This information can be used to eliminate poor questions and design better
tests in the future by understanding what concepts and types of reasoning are particularly
difficult for students. This notebook documents how to use the Scantron reader to turn student
responses on Scantron forms into computer files, and how to use the Webtron software with
these files to analyze test results.
Completing a Scantron job is a two-part process of entering data through the student
forms using the Scantron 8200 machine, and data analysis using the Webtron program. The
Scantron 8200 form reader is interfaced with an old AST PC in Bio Sci II Room 5205, and a
MSDOS program called Scanbook that is installed on the AST PC in that room. Data analysis
(computation of student scores on the exam, class means, etc.) uses a Web-based program
called Webtron.
The Scantron reader and AST PC are used to process the completed examination
forms and save the information into a raw data file. This raw data file needs to be saved onto a
3.5" floppy diskette in order to use Webtron and complete the data analysis portion of the job.
An earlier data analysis program called Scantrax required separate raw data files for each
version of an exam to be scored. Webtron does not have this limitation, that is, all exam
versions can be stacked on top of each other and processed as a single run. Additionally,
exams can be scored using any on-campus networked PC with Internet access; all that is
needed is the diskette containing the raw data file.
Webtron offers several advantages, including the ability to easily rescore exams, the
ability to include multiple correct response questions, and the ability to easily delete ambiguous
questions from being scored. Output includes the following for each version of the test as well as
the class as a whole:
1. A list by ID of student exam scores (raw and percent of total) as well as individual student
responses on each question,
2. Item analysis. For each question, the number and percent of students who answered the
question correctly, as well as the percent of students in the top and bottom quarters of the class
who answered correctly and the number of students who chose each possible answer. This
information can be used to judge the quality of the individual question.
3. A histogram of the test scores with test mean and standard deviation, with the score, the
numbers of students obtaining that score, and the class ranking of students obtaining that score.
3
Preparing to use the Scantron 8200 reader
The two important hardware components for data collection are the Scantron 8200
reader and the AST PC. If these components are already powered up, it is best to initialize both
devices by switching the Scantron reader off for a few seconds and then switching it back on,
and rebooting the PC by holding down the keys [Ctrl]-[Alt]-[Delete] in unison. Although this
initialization routine is not critical, it will free up PC memory and clear the interface buffers of any
data or error messages remaining from previous jobs. When rebooting the PC, be sure to
remove any diskettes in the floppy drive (drive A).
Diagram of the Scantron 8200
Holding tray Tension Adj. Power switch Forms to be scanned
SCANTRON
ID# Name
|||||||||||||||||||||||||||||||||||||
8200
- SCANTRON 8200 - :: UNLD 1 End
READY ::
2 Select
Plastic guide to be placed on
top of forms
The Scantron equipment is housed in BSII room 5205. Two of Bio Sci’s computer support
people, Barry Denlinger (x3507) and Marko Popovich (x4937), share this office. Contact them
if you experience technical difficulties with this equipment.
4
Preparing to read exam forms
1. Stay organized. Webtron works the same regardless of how the data file is ordered.
Although it is not necessary to sort the exams before processing them, sorting is
highly recommended to facilitate subsequent auditing and form retrieval. Therefore,
two levels of sorting are recommended. First, exams should be sorted according to
their version. For example, if three exam versions were administered, then group all
of version A forms together, version B forms together, and version C forms
together. Within these groups, the forms may also be sorted by the first digit of the
ID number. Since each form becomes a separate line in the raw data file, pre-
sorting can be a useful way to track down problems. Regardless of your pre-sorting
preferences, it is important to keep the forms organized according to how they were
actually scanned, since this order corresponds directly to the raw data file.
2. Webtron is very flexible in how multiple exam versions can be managed. Instructors
can either write exams so that students must bubble in a response to a version
indicator question, or exam versions can be indicated in some other way that will
not be read by the Scantron machine. For example, the last question on the exam
could instruct the student to fill in answer "B" for question 50, thus indicating one of
five possible versions (the Scantron forms are lettered A-E), which makes question
50 the version indicator question. Version indicator questions are an elegant solution
to the problem of grading multiple versions. However, instructors may not want to
make students aware of a particular exam version, or instructors may prefer another
sorting scheme that is not machine readable. Instructions for both schemes are
included below.
3. All exam keys must be completed using the ID# 199999999 for exam version one,
ID# 299999999 for exam version two, and so on. These numbers must be
completed in the space reserved for student ID#, regardless of whether an
instructor uses a version indicator question.
If you use a version indicator question:
4. The key(s) can be placed anywhere in the stack of forms as long as they precede
the scantron forms that refer to that key (Webtron does not otherwise care), but it
is recommended that all keys appear first. Place all keys on top of your first stack of
forms so that the Scantron reads the keys first and puts them at the beginning of
your data file.
5. On your version indicator question, either leave the column blank or fill in the letter
that corresponds to that particular version. This field will be hand-edited later and
replaced with a "v" in the data file, so filling in the column on the Scantron form
serves as a convenient placeholder. What follows is a sample data file that uses the
optional version indicator:
5
Sample data file:
19999999 DCDEBEBEDABBBAAADAEDDBBBDDEEBCCBABBAv
29999999 EEBBEEBCBBBADCDEBDBEDDCDBEDDDCDBBBBAv
39999999 BEEDABADCDDDBABBBAAAABBBABBBBDCDDDEBv
47518242 AAAEBEBEBAAAABAACCDDEDBBEECEBCCAABCCD
18149500 AEDCBBBCBABABBAAAAADEDABDDECACCBABCAD
38363623 EAEEEEBEAABBAAAADCBEDBABDEACBECBBBCDB
50799673 EEBBAABCAABACCDEBBBDDECDDEEDDEBBABBAB
95801721 EAECABBDDBBACCDCEDEABACDBEBEDCDBABBBA
02415874 BCCDABCECDDDCABBAAAAAABBABABBEEADDEBD
The first three lines of text are the exam keys. Notice the "v" in column 37 of these
first three lines. This character is added in the editing process to tell Webtron that a
student's response to question 37 determines which key the exam will be compared
to.
If you elect not to use a version indicator question:
6. A version indicator is not necessary if there is only one version of the exam. If there
is more than one version, it is still possible to skip a version indicator column by
placing each key directly before the group of exams it corresponds to; however, this
is not recommended and may not be supported in the future because it is easy to
make mistakes. What follows is a sample data file that does not use a version
indicator question:
19999999 DCDEBEBEDABBBAAADAEDDBBBDDEEBCCBABBAB
47518242 AAAEBEBEBAAAABAACCDDEDBBEECEBCCAABCCD
18149500 AEDCBBBCBABABBAAAAADEDABDDECACCBABCAD
38363623 EAEEEEBEAABBAAAADCBEDBABDEACBECBBBCDB
50799673 EEBBAABCAABACCDEBBBDDECDDEEDDEBBABBAB
51673722 EECABEBBEBAADCAEBDBEDDCDBABDDCDBAABAA
76366898 EEABBAACEBBBCDDEEDDEBDCDDAEDDECCBAABA
95801721 EAECABBDDBBACCDCEDEABACDBEBEDCDBABBBA
The key is followed by the exams it corresponds to. Notice that column 37 is used as
just another question.
6
Reading forms to create the raw data file
1. On the Scantron machine, press [1] to lower the paper tray and load your forms.
There is a limit of approximately 100 or so at a time.
2. Place your forms on the tray. Be sure that no forms are folded or bent and that the
tops contact the lower-left corner of the tray (refer to the diagram).
3. On the Scantron machine, press [1] to raise the paper tray.
4. Place the plastic paper guide on top of the forms. It is important not to place the
paper guide on the forms when the tray is down because it may not be aligned
correctly and could get caught as your raise the paper tray.
5. You are now ready to use the AST PC and access the Scanbook program. The
remaining commands are to be done on the PC.
6. Select Scanbook from the text menu. If this option is not available, then either type
"menu" at the prompt and hit [enter], or simply reboot the PC and the menu should
load automatically.
7. Place your formatted 3.5" diskette into the A: drive.
8. Use the arrow keys to select "Process" once Scanbook loads.
9. Select "Read Forms." At the top of your screen, you will see "Record Output File"
and some path designation like "G:\Scanbook\xxxxx\xxx." You need to tell the PC
to use the A: drive and give your raw data file a unique name. Back space over the
default path and type in "A:\xxxxxxxx" (without the quotes) where the x's are your
file name. Since this is a DOS-based PC, you are limited to eight characters, with
no spaces in between. The file extension ".txt" will be added automatically to your
name. Be sure to remember what you named the file, in case there are other files on
the diskette.
10. Press [F2] to direct Scanbook to download its form information. "Load File Name"
will appear. Press [Enter] to see a sub menu where you can indicate your particular
form type. Use the down arrow key to select "F-158.800" or just type in "10" and
press [Enter]. Most exams are administered on this form; if not, then look on your
particular form and be sure to select the appropriate one in this option.
11. The program will find the files it needs and check the status of the Scantron
machine. You should see the following:
Zone 1 ok
Zone 2 ok
Zone 3 ok … and so on through Zone 8. At the bottom you should see the
message "Finished downloading…and ready to read."
12. Press [F3] to read and the Scantron machine should begin reading forms.
13. If you get an error message instead of the "Zone ok" message, you can try selecting
[F1] to reset (and then reload) the forms routine. Otherwise, you can try lowering
7
and raising the paper tray on the Scantron machine (select [1]), or try powering the
Scantron machine off and then back on. Be sure there are not too many forms in the
tray; you will not necessarily receive an error message. Often it is unclear where the
source of the problem lies. If this is the case, you may need to reload the Scanbook
program or even reboot all components.
14. As the forms are being read, you may experience paper jams or feed errors due to
sticky forms. It is not a problem to read the same form twice. Scanbook will record
both ID numbers, and you can edit out the duplicate line later. As such, you can
rerun any forms that jammed. Press [1] to release the paper tray, correct the
feeding problem, and then press [1] again to resume reading. With a bit of luck, you
will only experience smooth scanning, but it is important to be aware of potential
sources of error. You may be requested to prompt the PC to continue (i.e., "Push
any key…") or you may have to press [2] or [CONT] on the Scantron machine to
continue. After a batch runs out, you can press [1] to lower the paper tray, load
another stack of forms, press [1] again to raise the tray, and then press [2] to
continue processing the job.
15. When all the forms have been read and you are ready to close your raw data file,
press the [END] button on the Scantron machine (the PC refers to it as the "end of
batch" button).
16. SAVE YOUR WORK! Press [F10] as soon as all the forms have been read. The
program will not prompt you. Verify that your file is being saved on the A: drive.
17. After saving your raw data file, you can select the option to quit the Scanbook
program.
Dealing with form feed problems
Unfortunately, the Scantron reader is prone to paper feed errors. Sometimes two or
three forms are grabbed at one time, or a form becomes misaligned as it travels through the
paper guides. The plastic guide should help alleviate many of these problems, but if errors occur
too frequently, then try the following:
1. Remove the plastic guide and put it aside
2. Place your right thumb on top of the right side of the stack of forms, and cup your
right hand around the rest of the feeder tray.
3. Apply just enough firm, consistent pressure with your thumb so that only one form
can be pulled at a time. Monitor the stack to see if lower forms are "creeping"
towards the first set of pinch rollers, and if so, gently scoot them back into place.
Creeping forms indicate that more than one form is about to be pulled through the
reader. A variation of this technique is to place your thumb just under the top form
and apply firm pressure to the rest of the stack, thus preventing multiple forms from
being pulled.
8
4. You may also need to turn the tension knob slightly, which changes the position of
the pinch rollers.
9
Editing your raw data file
Your raw data file probably needs to be edited before submitting it to Webtron. Use a
text editing program with a non-proportional font such as Notepad for this procedure, rather
than word processing programs, because the latter usually default to using proportional fonts
(where an “m” is wider than an “i”) and then the text columns won’t line up very well. If you
want to use Word, Excel, or Wordpad anyway, be sure to change all the text to the non-
proportional Courier font; then if you submit that file to Webtron, be sure you save it as a *.txt
ASCII text file because Webtron can’t read .doc files.
Editing allows you to correct your exam keys by throwing out ambiguous questions or
adding alternate acceptable answers. Editing is also necessary to denote where multiple version
entries should be marked in the exam. For example, exam question 25 may direct the student to
fill in reponse "C" for that column, which indicates version three of the exam. In position 25 of
the corresponding exam answer key, you would insert the character "v" (for version). This
version indicator tells Webtron to score exams with an "A" in position 25 against the key
19999999, exams with a "B" in position 25 against the key 29999999, and so on.
Alternate acceptable answers can also be added in the editing process. Consider the
following example key:
19999999 DCDEBEBEDBBAABCAADEAEAACv
19999999 A C
The second line indicates that in addition to answers "D" and "B" for questions three and five,
respectively, answers "A" and "C" are also acceptable. Additional acceptable answers can be
added with similar lines of text. The version character (“v”) appears for question 25. After
running Webtron, be sure to double check the answer key summary in the output to make sure
you marked the correct columns for your alternate answers!
You can also correct student ID numbers that have extra digits, spaces, or question
marks. Question marks appear when the Scantron detects more than one response in a column,
such as when students select two answers to a question. If this happens, a blank space should
be inserted in place of the question mark; otherwise, Webtron will flag this line as an error.
Sometimes students change their mind and erase their initial response. Responses must me
erased completely or the Scantron will insert a "?". Locate the original form to find out which
response the student intended and make the correction. Locating the form should be easy since
the first line of the text file corresponds to the first form that was scanned. All question marks
need to be investigated and corrected before submitting your data file for analysis
If you experienced form feed errors during processing and had to feed some forms
through twice, you will want to edit out duplicate lines of data. Scanbook does not recognize
duplicate student ID numbers; forms are simply read and logged as another line of text in the
data file. Webtron will recognize duplicate ID numbers, and will notify you that duplicates were
discovered. These duplicate numbers will then be reassigned in your analyzed output. It is best,
however, to delete duplicate lines rather than allow them to be analyzed. Otherwise, duplicates
10
will contribute to the summary statistics and will still have to be discarded if the output file is later
imported into your grade book program.
Warning: Do not insert tab characters into your data! Tabs indicate a different number
of blanks to different text editors, so Webtron forbids them.
Using Webtron to score your edited data file
Once all the corrections have been made to the raw data file, you can use Webtron to
complete the data analysis portion of the job. Since your data file is on a 3.5" diskette, you can
use any PC on campus that has Web access. Point your web browser to
http://darwin.bio.uci.edu/local/Webtron/webtron.html
This program is simple to use and generates a useful output that can be saved and/or printed as
a text file. Below is a figure of Webtron's user interface within a Netscape window:
All that is necessary to get an output is the file name of your raw data, which can be
located by clicking [Browse…] and then directing your browser to the proper drive, directory,
and file name (on PC’s, be sure to change the Browse button’s file selector dialog box default
from *.html to *.* so you can view all your files). The fields for instructor's name, course title,
year, and test are optional fields that will make it easier to keep multiple output listings
organized. Two fields are available to specify student ID lengths. Most student ID numbers are
eight digits and so 8 is the default for these fields. However, some extension students are given a
six digit ID, in which case you should input 6 for the minimum field. Click [Process data] once
you have entered your specific information. If there are no errors, then you will receive an
output similar to Appendix 3. Errors in your data file will cause Webtron to halt processing and
11
return a message indicating the location and nature of the problem. Errors may include bad ID
numbers or question marks in a line of text. If Webtron encounters a duplicate ID number, you
will be warned about the problem, but it is not considered an error. Webtron will attempt to
reassign one of the duplicate ID numbers after which you can resubmit your file. However, it is
best to audit your data file to eliminate duplicate ID numbers rather than allow multiple ID
numbers to be analyzed as separate exams. Once you have addressed the problems in your
data file, be sure to save your edited file before attempting another Webtron analysis.
Just below the Student ID length fields is the “Truncate ID's in output listing to final
xxx digits” input field. You should use this if you plan to post Webtron’s results in a public
building or on the web. UC Irvine’s Registrar does not want full Student ID’s posted in public
with test scores or grades, but partial Student ID’s are acceptable, and that’s what this input
field is for. If you leave it blank, Webtron will print each student’s full ID in the output listing,
but if you enter, say, “5”, then Webtron will truncate each student’s ID to the last five digits in
the output listing (An aside: Because the Registrar is now assigning new Student ID’s randomly,
rather than alphabetically, it may be acceptable to post full Student ID’s after the year 2003.)
Webtron can process data and give results in four different ways, depending on which
option you select in the Output pull-down menu (just to the right of the Process Data button).
The default mode is to display “All analysis results”, complete with per version summaries,
combined scores listed by ID, and a histogram of score distributions. However, you can click
instead to “Just final section and # correct” to just get a summary of raw scores listed by ID,
“Just final section and % correct”" to get a summary of scores listed by ID and percent
correct, and “Just final section and percentile” to get a summary of scores listed by ID and a
student's overall ranking on the exam. These last three options are convenient for importing your
output text file into a database or spreadsheet program.
Specific questions about the Webtron program are addressed online in the program's
FAQ file at:
http://darwin.bio.uci.edu/Webtron/WebtronFaq.html
or see Appendix 1 of this document. Appendix 2 is a sample input file with specific examples of
what Webtron looks for in a data file. Appendix 3 is a sample output file that shows a histogram
of all combined student scores and a summary of each student's rank on the exam, regardless of
their exam version. All Webtron output files have legends and summary statistics that are easy
to interpret. To save an output file, use your browser's File Save feature and save the output as
an ASCII text file. This text file can then be imported into a spreadsheet program like Excel or
a database program like Paradox or Access to match up Scantron ID numbers with your
records from the Registrar's office.
12
