A Web-Based Semi-Automatic Assessment Tool For Conceptual Database

							 A Web-Based Semi-Automatic Assessment Tool For
           Conceptual Database Model

                  Firat Batmaz, Roger Stone, Chris Hinde

              Department of Computer Science, Loughborough University
                              Loughborough, LE11 3TU
                     {f.batmaz, r.g.stone ,c.j.hinde} @lboro.ac.uk


ABSTRACT
The increased presence of diagram-type student work in higher education has
recently attracted researchers to look into the automation of diagram marking.
This project report introduces web-based diagram drawing and marking tools
for a new (semi-automatic) assessment approach. The approach reduces the
number of diagram components marked by the human marker and provides
detailed personalised feedback to students. The tools which have been used
in tutorials of a first year database module in the Computer Science
department at Loughborough University are described together with findings
from the usage of the tools.

KEYWORDS
Conceptual Database Diagram, Diagram-Type Student Work, Improving
Assessment and Student Feedback

BACKGROUND
Teachers tend to complain about the business of marking, especially when
the number of students in the class is high. If it is possible an experienced
tutor will arrange the questions so that the answers are easier to mark. Even
so, it is still necessary to process each student's work individually.

This research proposes a new marking approach for diagram-type work which
is called partial marking. The approach is that the computer should process all
the students' diagrams and present the work to the tutor, not on a student-by-
student basis, but on a component-by-component basis. Every node and
every connection that appears in all the diagrams can be marked
independently by the tutor.

How much work has the tutor done? Instead of marking every node and every
edge for every student the tutor has only had to make decisions once for
every different node and every different edge. This should be a significant
reduction in effort. What is more, if this exercise is repeated with a different
class, the tutor will only have to mark any new nodes or edges that have been
generated by the new class - hopefully only a trivial amount of work.

The obvious problem is how the computer is going to be able to 'read' the
student's diagrams. Our solution is to give the students a smart editor which
allows them to construct the diagram on screen and save their work so that it
can be submitted to the tutor in electronic form. This overcomes the biggest
obstacle to computer-based marking.

The smart editor should be quick, easy and satisfying to use with a minimal
learning effort. The editor should be attractive to use in comparison with
'pencil and paper'. From the tutor point of view it would be particularly good if
the editor actually contributed to the learning process in some way and of
course the editor should deliver data about the student diagrams in a form
that is convenient to 'mark'.

The immediate problem faced by the authors is that of marking Entity-
Relationship (ER) diagrams [1] and so the project uses ER diagrams as the
example context. The method can be extended to other graph-based diagram
work.

The marking tool developed for ER diagrams reduces the number of
components in the diagrams marked by the examiner. That requires finding
identical components in different student diagrams. The Assess By Computer
(ABC) [2] project, which has the same principle as this project, defines
identical components by using those component’s attributes (e.g. label, type,
adjacent boxes). In our research, identical components are defined by
references to the text describing the scenario (scenario referencing). A
similar approach is used for the intelligent tutoring system in KERMIT [3] and
ERM-VLE [4] projects. A detailed discussion of the scenario referencing and
component matching for this research can be found in [5].

AIM
The aim for this project is to reduce or remove as many of the repetitive tasks
in the marking phase of the assessment process as possible. This also
achieves consistency of both grades and feedback on the solutions as the
same tasks are performed fewer times (possibly only once) by the tutors.

OUTCOMES AND DELIVERABLES
An online diagram editor for semi-automatic assessment has been developed.
Lecturers and students in any UK University can access the tool on “http://co-
draw.lboro.ac.uk”. They can create an account by using their university email
address. At this stage the tool is designed to be used for conceptual database
diagrams only.

The online marking tool is developed to mark students’ work semi-
automatically. It is accessible to tutors on the same address “http://co-
draw.lboro.ac.uk”. They need to upgrade their ordinary accounts to a tutor
account to be able to use the marking tool. The account upgrading is achieved
by making a request to the project team.

The data set collected from using the tool consists of more than 200 student
diagrammatic solutions, which includes the design rationale for their solutions,
and tutor’s marks on the student solutions. The data captured from students'
work was analysed and the system’s marks have been compared with
instructor’s marks to assess the efficiency of the tool. The efficiency of the tool
reached 96 percent for some questions. The data set has been analysed. Two
academic papers reporting findings from using the diagram and marking tools
were written. One of them is published on ITALICS Volume 8 (2) June 2009.
The other will be presented on at ICSHEA-2009 Conference.

PUTTING IT INTO PRACTICE
The online diagram editor has been used with two scenarios by a first year
class of 200 students. This represents 4 separate sessions since the first year
class had to be split into four groups of 50 students. At the beginning of each
50min practical session a short demonstration of the system was given
showing how to make one entity, one attribute and one relationship. The
students were then asked to create ER diagrams for the first scenario (hitherto
unseen) and if possible go on to do the same for the second scenario. By the
end of the session most students had finished the first scenario and some had
finished both.

Scenario   Total Component Component      Diversity   Auto marked       Manual marked   Efficiency
No         #                Group #       Rate %      Group #           Group #         Rate %
1          5356             708           13          468               240             96
2          3707             607           16          317               290             92
                         Table 1 Summary of diagram marking for two scenarios


The tutor marked the student diagrams using the marking tool in less than two
hours. Table 1 shows the summary of the marking task. The tutor marked 240
components out of 5356. That makes the efficiency of the tool for first
scenario approximately 96 percent. The efficiency for the second scenario is
92 percent. This might be interpreted such that the system’s efficiency
increases when more students draw ER diagrams for the same scenario. The
table also shows the diversity rate. This rate is calculated by using both
component group and total component numbers. If student diagrams are
similar to each other then the diversity rate decreases otherwise it increases.
Diversity rate could be used as a feedback for the scenarios. Student
solutions for the similar scenarios should have the same rate. If they are very
different, the tutor may analyse this and revise the scenario text to find out the
reason for the diversity.

The marking tool produces a detailed report for any chosen scenario. The
report has a list of every distinct element, how it was marked and the number
of students whose diagrams included that element. This reveals for example
how many students made the 'same mistake'. So for example it was clear that
something in the way the first scenario was worded caused 50 students to
wrongly identify "Consultant Name" as an entity and go on to make related
mistakes with attributes. The report contains aggregate marks e.g. entities
(green 82%, amber 7%, red 11%) attributes (green 82%, amber 10%, red 8%)
and relationships (green 34%, amber 59%, red 7%) showing that it is the
precise identification of relationships (amber 59%) that caused the most
problems.

In feedback sessions, the students were able to see their marked diagrams.
The colour coding of tutor comments was extremely well received by the
students and led to lively, positive discussion of the principles involved with
interpreting the scenarios which was very beneficial. A simple questionnaire
about the editor and the associated marking feedback has been given to
students at the end of term. The results from 70 returns show that the
students were favourably disposed to the editor and they liked the coloured
feedback.

ISSUES AND DEBATES
Learning issues

Q) Can the students use any diagram editors to draw diagrams and submit
their works to your marking systems in order to get personalised feedback?

A) The marking tool only accepts the submission from the online diagram
editor developed for this project.

Q) Does the student need to learn how to use the editor?

A) Yes, they need to learn. In our experiment, a short demonstration of the
system was sufficient for the students to start using the editor.

Q) Does the student see their feedback immediately?

A) The student diagrams are marked as a batch after the deadline for the
submission. Feedback is revealed to students once marking the batch is
completed. In this case, students don't get their feedback immediately.
However it is possible to give feedback instantly for some part of their work if
the system has stored previously marked solutions for the same questions.

Q) How much detailed feedback can be given to individual students?

A) The system writes standard comments on marked components. The tutor
may give additional detailed feedback on the same components. Instead of
commenting on every component for every student the tutor has only had to
make decisions once for every different component. The system uses these
comments to generate personalised feedback for every student.

Teaching issues

Q) How can the tutor upload questions for the database modelling to your
system?

A) The tool has two sample questions to practice the editor. If the tutor wishes
to upload their questions then they need to email questions to the project
team for uploading.

Q) Who will mark the student works?

A) Each tutor account holder can mark the questions. Our research team will
mark the students’ work for sample questions. We expect the tutor who
provides a particular question will mark the student work for that question.
Required support will be given to the tutor for marking.

Q) How different is partial marking from traditional marking?

A) In the partial marking style, the tutor needs to mark every component
independently that appears in all the diagrams. This marking style is quite
different from the traditional marking. However it is observed that the marking
process is intuitive and easy to learn. Example slides to demonstrate the
partial marking will be available to the tutors on the project site.

Q) Can the tool be used for the assessment of different diagrams other than
ER Diagrams?

A) The tool currently supports only ER diagrams. However the semi automatic
approach to marking can be applied to many other graph-based diagram
works. The development of support for UML diagrams is already planned.

RESOURCES
A set of practical scenarios for database modelling are available on the site
“http://co-draw.lboro.ac.uk”. Students can practise their design skills on the
sites and get personalised feedback. The number of scenarios will be
expected to increase when academics are aware of the availability of the tool.

BIBLIOGRAPHY

[1] Chen, P.P., The Entity-Relationship Model - Toward a Unified View of Data, ACM
    Transactions on Database Systems 1 (1), 9-36, (1976).
[2] Tselonis, C., Sargeant, J. and McGee Wood, M., Diagram matching for HCC assessment,
    Proc. 9th CAA Conference, Loughborough UK, 2005
[3] Suraweera, P., and Mitrovic, A., 2002, KERMIT: A constraint-based tutor for database
    modelling. Proc ITS’2002, LCNS 2363, 2002, 377-387.
[4] Hall, L. and Gordon, A., 1998, A virtual learning environment for entity relationship
    modelling. Proc 29th SIGCSE, Atlanta, Georgia, US, 1998, 345-349.
[5] Batmaz, F. and Hinde, C.J., 2007, A_Web-Based Semi-Automatic Assessment Tool for
    Conceptual Database Diagram, Proceedings of the sixth Web-Based Education
    conference, pp. 427-432.