Get documents about "Reviewers comment ICALT 2005
Document Sample
Reviewers comments
on our paper submitted to 35th ASEE/IEEE
Frontiers In Education Conference
K. Schützler, K. Bothe
Introduction
• In 2003 first presentations of current JCSE
project state:
– Information Technology Interfaces, Cavtat, Croatia
– Eurocon, Ljubljana, Slovenia
– 1st Balkan Conference in Informatics, Thessaloniki,
Greece
• 2005: submitted a new paper with current state
to 35th IEEE/ASEE Frontiers In Education
Conference, Indianapolis
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 2
Homepage of FIE 2005
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 3
Topics of interest on FIE 2005
• Accreditation and assessment • Innovative uses of technology in
• Active learning the classroom
• Capstone and senior design • K-12 initiatives and partnerships
experiences • Laboratory experiences: On-site
• Computer and Web-based and at a distance
software • Learning models
• Creative design experiences • Lifelong learning
• CSET educational research • Nontraditional students
• Distance learning: Methods, • Partnerships (industry,
technologies, and assessment government, university,
• Diversity: Valuing it, achieving it, international)
and teaching it • Service learning
• Entrepreneurship programs • Software engineering
• Ethics: Creative ways to teach • Student retention and persistence
and assess it • Teaming
• Faculty development • Undergraduate research
• First-year courses and programs experiences
• Globalization: Preparing faculty • Women in CSET education
and students • Other (You may submit abstracts
• Innovative degree programs and and proposals on other topics that
curricula address issues at the frontiers in
CSET education.)
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 4
K. Bothe, K. Schützler, Z. Budimac, K. Zdravkova:
Collaborative Development of a Joint Web-Based Software
Engineering Course Across Countries
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 5
Aims of this presentation
• Document the serious and detailed
review process of FIE 2005
• Make reviewers comments available
for all project participants
• Draw some conclusions
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 6
January 10, 2005:
Abstract submission
February 20, 2005:
Acceptance/Rejection of
abstract
March 21, 2005:
Full paper submission
April 30, 2005:
Notification on review
results
May 30, 2005:
Submission of revised
paper with modification
notes
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 7
Excerpt from first accept
notification email
Dear Klaus,
The final recommendation for FIE paper 1106 is:
ACCEPT with major changes
That means: Each comment of the reviewers must be addressed
(even if considered inappropriate)
In the comments below, you will see numerical scores for different
aspects of your presentation. The scale was:
1 - poor 2 - needs work 3 - good 4 - very good 5 - excellent
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 8
Note from Heidi Diefes-Dux
(from the organizer‘s site):
Two reviewers raise a number of questions/issues
that require greater revision than the reviewers
indicated - thus my recommendation. As you do
have an amount of space left in the paper to
addresses these, I recommend that you focus on
those that would broaden the usefulness of the
paper to a wider audience. In an email to me,
please indicate how and where the issues raised
by the reviewers are addressed. If you feel that
the comments are not appropriate, please
indicate that to me as well.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 9
Reviewer 1
Reviewer's recommendation: accept_as_is
New ideas: yes
Comments on new ideas:
It addresses collaborative development of course materials across
several countries.
Soundness of work: yes
Contribution: 3
Comments on work contribution:
project management issues
Presentation quality: 4, Writing quality: 4, References: yes
Gender neutral language: yes, Should anything be deleted: no
Completeness of work: no
Comments on completeness:
still somewhat of a work in progress
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 10
Reviewer 2
Reviewer's recommendation: accept_revise
New ideas: yes
Comments on new ideas:
Describes experience with a shared course in
software engineering with participants in multiple
countries. Approach could be useful for
technical education in many locations around
the world.
Soundness of work: yes
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 11
Reviewer 2 (cont.)
Contribution: 4
Comments on work contribution:
Describes a successful project in which many
participants not sharing a language use common course
materials to support a course in software engineering.
Insight and experience from this project could be of
broad general interest, particularly in developing
countries and countries partnering with them in
education.
Presentation quality: 4, Writing quality: 4, References: yes
Gender neutral language: yes
Should anything be deleted: no, Completeness of work: yes
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 12
Reviewer 2 comments
Writing quality
This is interesting work and the quality of the writing is fairly
good. My questions address relatively minor points, but I
encourage the authors to read over the paper again carefully
to catch a few remaining awkward sentences, typos, and
misused words.
We went through the paper as requested and indeed found a
few mistakes and even one incomplete sentence.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 13
Reviewer 2 comments
Web-based course?
You describe the course as web-based, but that applies more
to the instructors than student participants. (That is, students
apparently attend lectures where they see powerpoint slides
rather than use machines on a network.) This seems to be
inconsistent with the usage of the term in my experience.
skip
We agree with reviewer's description of our course, but we still
call the course web-based. In section "Overview of project
management and course materials" we added "Although
students are expected to attend regular lectures, the course as
a whole could be called web-based. All documents the
students need are published at the course website: the slides
in PDF-format, case studies, assignments, tool information,
literature recommendations, schedule, etc.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 14
Reviewer 2 comments
Contributions from all participants?
The abstract mentions a "plethora of interesting contributions" from the diff-
erent participants across various universities and countries. Could you give
some examples? I'm curious how the course has evolved over time. You
allude to such changes but give essentially no concrete examples or details.
Figure 3: Contributions to the project
We added a paragraph in
Translate materials to Use slides in lectures and give usage
section that gives an example English reports
Add new slides Design and
how the contributions and other administrate the
materials project web-site
changed the course.
Include new topics Contribute to the
Course
Besides that, we would like to development assignments
draw the attention to Figure 3 Write lecture notes
Contribute to the
pool of case
for additional information on Write review studies
reports
the contributions of all project Develop slide style Translate materials to native
partners. guides language
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 15
Reviewer 2 comments
One comment – many questions:
examinations, tools, literature, duration
You don't say much about some of the links under course materials,
specifically the examinations, tools, literature, and duration links. Are copies
of all examinations shared with other participants? What about translation
overhead? What kind of tools are currently available, or do you plan to make
available in the future? Will the literature link simply give citations of relevant
papers, or does this describe internal documents for the course? What
duration information is included: just the time required in class to cover each
topic or to cover each lecture?
skip
We extended section "Course materials" with the new subsections "V.
Literature" and "VI. Tools". It covers now all relevant aspects of the materials
on the project website. In section "Overview of project management and
course materials" we extended the information concerning the Duration link.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 16
Reviewer 2 comments
Assignments
Are the assignments linked in (under course materials) labs or
homework style problems? What kinds of assignments have been
included in the course so far, and what kind of tools or documentation
do they require?
Again, the missing information is addressed in two new paragraphs in
subsection "IV. Assignments" of section "Course materials".
"Typical tasks of the assignments include: reviews of requirements
specifications, object-oriented modeling practices, performing cost
estimations, applying test case determination methods, automated
regression testing, and evaluating software metrics."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 17
Reviewer 2 comments
Concurrent updates?
It seems challenging to let all participants update course mate-
rials but to store only a single current file. What if you don't like
the changes someone else makes?
skip
To address this question we added an extra paragraph at the
end of subsection "IV. Update management" in section "Project
management to organize collaborative work":
"Of course, it is necessary to assess the results of the update
process. Modified topics are reviewed by their original author. If
there are any issues raised, they will be discussed and solved in
a cooperative way between author and modifier."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 18
Reviewer 2 comments
Collaboration success measured?
Do you have any way of measuring the effectiveness of the
class or of this approach to managing a class? To what extent
has it reduced the overhead for new participants in creating a
new class and getting it up and running?
skip
The "Conclusions and future work" section has been extended:
"Our original goals to disseminate experience and to reduce efforts in
creating a new class on software engineering were reached
successfully. Participants at several universities could introduce an
approved and mature course with all of its parts (lectures,
assignments, case studies, etc.) Both, the many years of experience
and the produced materials could be reused.
The partner universities in southeast Europe experienced significant
reduces (compared to earlier attempts of building new courses from
scratch) of the managerial efforts for introducing the course, a benefit
that can hardly be overestimated."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 19
Reviewer 3
Reviewer's recommendation: accept_revise
New ideas: no
Comments on new ideas:
Multi-national collaboration and material sharing.
Soundness of work: yes
Contribution: 2
Comments on work contribution:
Giving confidence to others to follow this pattern of
working.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 20
Reviewer 3 (cont.)
Presentation quality: 5
Writing quality: 4
References: yes
Gender neutral language: yes
Should anything be deleted: no
Completeness of work: no
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 21
Reviewer 3 comments
Missing information
Extra commentaries by the participants and a retrospective an-
alysis of the effectiveness of the transfer process would have
been useful.
As mentioned above, we discuss the effectiveness of the tran-
sfer in our new version of the paper. We do not think that com-
mentaries would fit in the structure of the paper. In our opinion,
the paper as a whole presents the common experiences of all
project partners.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 22
Reviewer 3 comments
Influence of related work
The paper was well prepared and presented, expressed in
good English and easy to read, with sufficient reference to
earlier activities and relevant publications. But how was the
direction and management of this project influenced by the
earlier work?
To clarify, why our project could not be a simple repetition of
the referenced projects, we added a paragraph:
"Compared to similar approaches [2, 3, 5, 6] for collaborative
preparation and usage of joint course materials, our project
contains a couple of unique features: multi-lingual project
partners, a strict style for lecture notes, and a strong strategy
for project management with review reports, update manage-
ment, style guides, etc."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 23
skip
Reviewer 3 comments
Addressed the right issues?
I was disappointed in the lack of critical appraisal, or pedagogic
discussion concerning the effectiveness of the transfer process.
I do not think, if I were to embark on a similar multi-national col-
laboration, that I would rush to reread the paper for advice and
guidance. There is little here which challenges common sense,
and no experiential commentary.
As mentioned in section "Conclusion and future work", more
and more partners joined us over the years and will join us in
the future. From our experience the questions raised by new
partners are indeed the ones addressed in our paper. Thus, we
do not find this comment appropriate.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 24
Reviewer 3 comments
Key information of the paper
The topic and information about how to run similar projects
could be of genuine value for many teachers in HE across
Europe, and may interest US academics too. What key
information do the authors hope to relate to readers? What are
the principal benefits to be gained from reading this paper?
We added a paragraph in section "Conclusions and future
work" to explicitly state that our project could serve as an
guiding example to similar settings:
"Besides that, our project shows that the process of transfer-
ring an existing course to other environments (even with dif-
ferent languages) can be performed successfully, as long as
the issues we presented here are addressed appropriately."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 25
Reviewer 3 comments
Students cooperation
Are students at the partner institutions
encouraged to communicate or even
cooperate on case studies and assignments?
"Up to now, students do not communicate or
cooperate across universities."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 26
Reviewer 3 comments
Effectiveness of course at partner sites
Was any appraisal undertaken to gauge the differences between the
effectiveness of the course material when used in the original insti-
tution, and that achieved in the acceptor universities? Did student
feedback get compared? Were graded assignments and exam ques-
tions shared? How was success/failure judged?
skip
We extended our paper with another paragraph:
"Up to now, students do not communicate or cooperate across univer-
sities. However, there is already some kind of indirect cross-university
influence from the students. Their solutions to the assignments are
compared throughout the whole project and they are evaluating their
respective course using a project-wide feedback form. The compari-
son of these feedback forms is subject to future work."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 27
Reviewer 3 comments
Translation to English
How was the course material translated from German to English? Who
paid for this time consuming work? How were the results checked? How
long did it take? Presumably it was an irksome task because of the later
reference to automating the work.
We changed the first paragraph of section "Project management to
organize collaborative work" to cover more details on the managerial
background of the project:
"Because our project received substantial funding from DAAD, yearly
workshops and coordination meetings were possible. Partners from SEE
were enabled to stay at the German university several times in order to
work on project tasks like translating existing and preparing new materials."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 28
Reviewer 3 comments
Mobility issues and material reuse
How much travel, face-to-face meetings for academics, or guest-lecturing
was possible during this project?
skip
The change of the first paragraph of section "Project management to
organize collaborative work" does apply to this question, too.
skip
As the courses were ACM compliant, would it have been possible to use
an alternative course, already available in the English language?
We do not think that using other existing courses would have been
appropriate, because the partners explicitly expressed their interest in the
particular course available at Humboldt University. We changed the
respective sentence in the first section to reflect this situation more.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 29
skip
Reviewer 3 comments
Misunderstandings
"the aims changed over time" - how, why and in what manner?
We think, that this issue has been addressed already in the directly fol-
lowing sentences. We made it a bit clearer now, by putting a colon instead of
a dot.
"…over time, the targets … of the project have changed considerably: With
the contributions from participating partners, new materials have been
developed as a response to the feedback and suggestions of professors and
their students. Due to the additional information the project had to face and
overcome organizational challenges to coordinate the work."
"special course material problems" - what were these?
We assume that this results from a misunderstanding of our bad English.
We changed it to "special problems concerning course materials" and again
replaced a dot by a colon.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 30
Reviewer 3 comments
Paper from one partner only?
The paper appears to have been produced by the senior part-
ner, with little evidence of reported experience from the junior,
accepting institutions. Views more clearly from all the instituti-
ons would have been welcome.
We do not consider this comment as being appropriate. We
are not able to imagine what kind of evidence the reviewer is
requesting.
The paper has been produced together by the senior partner
and two of the junior partners and contains experience of both
sides.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 31
Reviewer 3 comments
Case studies
Well chosen case studies are often the most useful material,
"borrowed" by colleagues from different universities. Little was
said on this subject. Were the two original scenarios specially
drafted, or inherited from the original, parent course? Mention
is made of the tutors' commercial experience being brought to
bear locally for the benefit of the students. How can this
resource be distributed successfully to other sites?
In subsection "III. Case studies" we added a paragraph:
"Presently, we are including two case studies reused from the
original course, one from an administrative application and the
second from a technical application. The project participants
contributed additional well suited case studies. These case
studies are not included in the course so far, because teaching
materials for them still have to be prepared."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 32
skip
Reviewer 3 comments
Technical problems and recommended literature
For the reviewer, the project URL, reference [7], did not work
when accessed. An email was dispatched to the principal
author.
We fixed that problem.
Instead of ".../index.html" it has to be ".../index.htm".
What text books were recommended for use by students?
As mentioned above, we added subsection "V. Literature" to
section "Course materials" to give more information on the
literature recommendations in our course.
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 33
Reviewer 3 comments
Configuration management
Was any project source file control system (CVS?) employed
to maintain project consistency?
This question is answered in the last paragraph of subsec-
tion "IV. Update management" in section "Project manage-
ment to organize collaborative work“:
"Since most materials are in binary formats and are not
changed concurrently there was no urge to apply a complete
configuration management. However, previous versions are
archived to be used as fallbacks in case of problems."
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 34
Final note from Heidi Diefes-Dux
(from the organizer‘s site):
Klaus Bothe -
Thank you for making those changes. I am very satisfied
with your responses to the reviewers and I am looking
forward to seeing you and your presentation in Indy!
Heidi
*************************************************************
Heidi A. Diefes-Dux, Assistant Professor
Purdue University
Department of Engineering Education
*************************************************************
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 35
Conclusions
• Our project is interesting and useful for third parties
• FIE reviewers are interested in quantifiable results
• Commonly used real-life case studies are
challenging
– Still a somewhat open question in our project
• Cooperation between students across countries is to
be considered seriously
• Feedback from partner universities is highly
recommended – especially critiques
Kay Schützler, Baile Herculane, 28.08. - 03.09.05 36
