by al3alem29


									                       Can we teach Empirical Software Engineering?

                                  Letizia Jaccheri and Thomas Østerlie
                        Department of Computer and Information Science (IDI)
                       Norwegian University of Science and Technology (NTNU)
                       letizia.jaccheri@idi.ntnu.no, thomas.osterlie@idi.ntnu.no

                      Abstract                             course for teaching ESE to software engineering
                                                           researchers in this paper. We believe that our findings
    We report about an empirical software engineering      are equally applicable for teaching ESE to
course for PhD students. We introduce its syllabus and     practitioners.
two different pedagogical strategies. The first strategy       There are some fundamental challenges for an ESE
is based on individual learning and presentations. The     research project to succeed. First, researchers in
second relies also on social activities to support         general and PhD students in particular must be well
learning and knowledge sharing. The syllabus, which        acquainted with existing methods. Second, ESE
has been used for three iterations of the course, is       research is a major undertaking and it is a cooperative
available at our web site together with student essays,    activity within a research group. Third, the research
evaluation data, and other documentation produced          needs to be relevant outside the research community.
during course runs.                                        The research group must therefore have access,
                                                           knowledge of and familiarity with the software
1. Introduction                                            industry in order to study concrete and real situations,
                                                           and to generate industry relevant research questions.
                                                           Lastly, the research problems must also be relevant
   Empirical software engineering (ESE) is a sub field     within the academic field of software engineering.
of software engineering which aims at applying             Research problems therefore have to be significant to
empirical theories and methods for the measuring,          both the international research community and the
understanding, and improvement of the software             local industry. This means that research problems and
development process in organizations. ESE is by its        questions must be shared and understood by the all
nature a multi-disciplinary field as software engineers,   members of the research team and by the industrial
industry actors, statisticians, pedagogues, and            actors. These challenges need to be addressed by a
psychologists have traditionally been cooperating.         PhD level course in empirical software engineering.
   In this paper, we report on a PhD level course in           Our course is an attempt to make our PhD students
empirical software engineering that has been run three     acquainted with the state of the art within ESE as well
times. The course held during the autumn of 2002 was       as reflect on investigations done by others and in
based on individual presentation. During the spring of     which they have possibly participated.
2003 the course was based on group work held. This             Our course has been run three times and its
was replicated when the course was held in 2004.           syllabus, program, and evaluation is available at [16].
   The main objective of empirical software                We have evaluated the pedagogical effects of the
engineering education is to train software engineers in    course by exploiting Bloom´s taxonomy of learning
empirical evaluation of the tools, techniques, and         (which is well known and used by the software
technologies used in software engineering. It is in this   engineering community) and qualitative methods for
context, that we see the importance in discussing the      data collection and analysis [29] as applied in the ESE
strategy for teaching empirical software engineering.      field.
   We are of the opinion that ESE is relevant for both         This paper is structured as follows. Section 2
practitioners and researchers. For practitioners it is     introduces the aspects of software engineering
about evaluating tools and techniques for use in           education which have been relevant to our work and
concrete cases [18]. While it is equally important to      some learning issues in research education. Section 3
teach ESE to each of these two groups., we report on a     describes our course, its syllabus, pedagogical goals
and the two different strategies. Section 4 is about the       In this sense, their approaches can be classified as
evaluation of the course. Discussion and conclusions       addressing short-term requirement vs. addressing long-
are given in section 5.                                    term requirements. Guidelines for Software
                                                           Engineering Education [4] adopt a middle ground
2 Background                                               approach to education contents. They address the long-
   Software engineering, as a field, has, among others,    term issues and are based on the body of knowledge
two supporting disciplines–software engineering            for software engineering [1]. This body of knowledge,
education and ESE. Software engineering education          however, is based upon expert opinions within the
focuses on training "software professionals" for the       industry.
industry [25], while ESE focuses on evaluating the             Along the second axis, there are two strategies: the
tools and techniques used, developed, and intended for     first strategy is based on lectures and individual
use in the industry through empirical validation. There    learning. The second strategy is based on learning by
is an element of training required in making the           doing, also known as project-based learning. Both
transition from trained software professional, with or     Meyer [25] and [4] favor project-based learning.
without working experience, to become an ESE               Unlike the education content, they argue in favor of
researcher.                                                project-based learning to "prepare our students for the
                                                           real challenges they will face" ([25] p.33). They also
2.1 Software Engineering Education                         argue that it is easier to learn from personal mistakes
                                                           rather than mistakes related by a lecturer.
    The software engineering education literature
moves along a dual axis; one axis for education            2.2 Learning issues in research education
content and one for pedagogy. We use the two axes to
reflect on the current state within software engineering      Provided that it is possible to teach somebody how
literature in this section.                                to become a good researcher, there are three kinds of
    The education content axis is delimited by two         courses that can be offered as part of research
extremes: industry driven and principles driven. Meyer     education:
[25] argues that the contents of software engineering      1. General courses on research methods at both
education must be driven by the principles on which            undergraduate and post graduate level are usually
software engineering is based:                                 offered by social science faculties. These courses
    "What matters is teaching [the students]                   address research issues such as scientific method
fundamental modes of thought that will accompany               and nature of evidence, advocacy versus evidence-
them throughout their careers and help them grow in            based approaches, writing and reviewing research
this ever-changing field. The ones who blossom are             proposals, how to use bibliographies and citation
those who can rise beyond the tools of the moment in           searches, project planning, selecting results and
harmony with the progress of the discipline. ([25]             places to publish, outlining and structuring
p.29)"                                                         research papers, the peer review process,
    On the other end is the work of Lethbridge [24].           presenting posters and papers at conferences,
Based on a survey of 168 software engineers, he finds          publishing in academic and engineering journals,
significant differences between curricula taught at            etc..[5]
colleges and universities and the actual knowledge         2. Courses on research methods in computer science
required in the industry. Lethbridge argues for aligning       address some of the research issues above, such as
existing curricula with skills required by the industry.       scientific method and nature of evidence,
Where Meyer [25] is specific on the need to distance           customized to the IT field. At our department, for
education from the industry’s immediate, short-term            example, there is a common introductory course
requirement, Lethbridge writes little of the long-term         for all PhD students, which addresses general
requirements that Meyer addresses with his principles.         research issues in IT like those discussed for
                                                               example in [12].
Table 1 Bloom’s taxonomy of learning customized to ESE
Level           Definition                Sample        Sample behavior       Sample behavior ESE
Knowledge       Student recalls or        Write         The student will      The students will be able to
                recognizes information,   List          define the 6 levels   define the content of the different
                ideas, and principles     Label         of Bloom's            papers.
                in the approximate        Name          taxonomy of the
                form in which they        State         cognitive domain.
                were learned.             Define
Comprehension   Student translates,       Explain       The student will      The students will explain the
                comprehends, or           Summarize     explain the           purpose of the give methods and
                interprets information    Paraphrase    purpose of            investigations.
                based on prior            Describe      Bloom's
                learning.                 Illustrate    taxonomy of the
                                                        cognitive domain.
Application     Student selects, trans-   Use           The student will      The student will be able to use
                fers, and uses data       Compute       write an              one method for experimentation.
                and principles to         Solve         instructional
                complete a problem        Demonstrate   objective for each
                or task with a mini-      Apply         level of Bloom's
                mum of direction.         Construct     taxonomy.
Analysis        Student distinguishes,    Analyze       The student will      The students will compare and
                classifies, and relates   Categorize    compare and           contrast different methods and
                the assumptions,          Compare       contrast the          investigations.
                hypotheses, evidence,     Contrast      cognitive and
                or structure of a         Separate      affective domains.
                statement or question.
Synthesis       Student originates,       Create        The student will      The students will design an
                integrates, and           Design        design a              investigation by choosing and
                combines ideas into a     Hypothesize   classification        perhaps combining different
                product, plan or          Invent        scheme for            methods.
                proposal that is new      Develop       writing
                to him or her.                          educational
                                                        objectives that
                                                        combines the
                                                        affective, and
Evaluation      Student appraises,        Judge         The student will      The students will judge the
                assesses, or critiques    Recommend     judge the             effectiveness of using empirical
                on a basis of specific    Critique      effectiveness of      software engineering methods.
                standards and criteria.   Justify       writing objectives
                                                        using Bloom's
                                                            3.1 Common characteristics
3.   Courses like the one we describe in this paper,
     which address empirical research methods in            In the following section we discuss the common
     software engineering. These courses do not exist       characteristics for the course.
     in isolation but in education and research context
     that may or may not include general research           3.1.1 Students
     courses or courses specific for IT research. As far
     as we know, there is at least one paper that reports   The empirical software engineering course counts as
     on teaching ESE [15] as part of a software             7.5 European Credit Transfer System (ECTS). This is
     engineering course. Undergraduate students work        equivalent to 12 hours work for 15 weeks, 2 hours in
     in projects, and the teachers play the role of the     class and 10 hours of other learning activities, for a
     customer. In one project the customer is a             total of 180 hours. The course was run once during the
     hypothetical company wanting the students to           autumn of 2002 at the Norwegian University of
     perform empirical studies (mainly experiments) to      Science and Technology (abbreviated NTNU) for PhD
     evaluate different alternative techniques, e.g.,       students by the software engineering group, and once
     different kinds of reviews. Students are not asked     during the summer of 2003 at Simula Research
     to plan the studies but only to perform them, as       Laboratory in Oslo for both University of Oslo and
     planning and running would take too much of the        NTNU PhD students.
     course. The syllabus of the empirical software            Some of the students worked in empirical software
     engineering part is [32].                              engineering research projects, while others worked in
                                                            other kinds of software engineering projects. All
2.2 Software engineering             education      and     students had a general course about research methods
research education at IDI                                   in IT as part of their curriculum. Some students even
                                                            had a course on research methods in general. The
   The software engineering group at IDI has thirty         students age, gender, nationality, and scientific
years experience with project based software                background differed.
engineering education. One of the author of this paper,
Jaccheri, has more than ten years experience with           3.1.2 Syllabus
teaching project based software engineering, software
quality and metrics issues to software engineering          The course has had the same syllabus throughout.
students, both in Italy and in Norway, as well as           Here, we refer to a consolidated bulk of literature,
reflecting and writing about this [17] [18] [19]. IDI has   which is used as syllabus for the course.
7 years experience with an introductive course for IT          The syllabus is divided into three parts: motivation,
researchers, which is mandatory for all new PhD             method, and actual investigations.
students. Jaccheri has had the main responsibility for      • Motivation: In [33] and in [31], motivations for
this course for one year. IDI graduates twenty PhD              the existence of the ESE field are given. These
candidates each year and the software engineering               papers provide also a classification of existing
group has graduated a total of 19 software engineering          software engineering research papers according to
PhD students.                                                   the kind of empirical method used in the
                                                                respective research.
3. Empirical software engineering course                    • Methods: [32] provides an introduction to the
                                                                field, with special emphasis on experiments. [22],
   The course reported in this paper is offered to PhD          [9], [28], [3], [8], [29], [2], and [14], provide
students within software engineering. While the course          concrete methods for performing, and analyzing
has been run three times, we mainly report on the first         investigations. [27] and [6] are about Data
two times as the third time we ran the course as a              Analysis Methods.
replication of the second time. The first and the second    • Actual studies: [13], [23], [11], and [28] are about
times had some common characteristics and some                  concrete investigations.
different characteristic. In the following section we       The course ended with a final oral exam with the
discuss the common characteristics and afterwards we        teacher and an ESE expert outside of the university as
introduce the characteristics that were different for the   examiner.
3.2 Differing characteristics                                 students to performance work and to stimulate
Here we introduce the characteristics that differed.          cooperation among students.
                                                         f.   Production of a five minutes movie advertising the
3.2.1 Pedagogical goals and teaching strategy for             field of empirical software engineering.
Autumn 2002                                              g.   Choosing an actual study in the syllabus,
                                                              characterize the investigation presented in the
In the autumn 2002 iteration of the course, the goal          chosen study according to motivations, method,
was to make the students acquainted with the contents         measurements, and data analysis method. The goal
of the syllabus. The course was held in a classroom           of this exercise was to simulate the planning and
context where students were met two hours a week for          execution of an empirical investigation.
13 weeks (totally 26 hours in class).                    h.   Group work with the goal of obtaining a deep and
   At each meeting, one student presented one paper           critical understanding of the issues presented in
from the syllabus. The teachers responsible for the           the syllabus, also in light of the experience
course provided feedback and stimulated discussion            acquired during the previous exercises. Each
around the paper.                                             group had to: choose one research hypothesis and
                                                              its motivation (business, education, others); list
3.2.2 Pedagogical goals and teaching strategy for             and comment investigation planning and risks
Summer 2003                                                   (what can go wrong); define (or reuse) guidelines
                                                              for     designing      and    running     empirical
In the summer 2003 version of our course, the                 investigations; choose one actual investigation.
pedagogical goals were:                                  i.   Essay writing. Each student had to write an
1. given our syllabus which introduces a possible             individual essay. The essay assignment was
       overview of empirical software engineering             “Extract from [22] and from the other papers in
       knowledge, let our students know about the             the Methods part of the syllabus, the most
       syllabus at a level with is as high as possible        important guidelines for designing and running
       according to the Bloom´s taxonomy [7].                 empirical investigations. Characterize the
2. establish a Norwegian network of young                     investigation presented in the chosen actual study,
       researchers within the field of software               according to these guidelines. This means that you
       engineering.                                           have to comment about the motivations, the
   Bloom’s taxonomy is reported in Table 1. The last          method, the measurements, and the data analysis
column of the table (Sample behavior ESE) gives our           method”. The essay was supposed to be handed in
interpretation of the taxonomy when applied to the            two weeks after the seminar and one week before
ESE domain.                                                   the oral exam. Recalling that students have 180
The second iteration of the course was organized as a         hours allocated to this course, we expected that
three days event (21 hours). Here we list in                  students worked full time writing the essay during
chronological order the main tasks of the course.             these two weeks.
a. Students were informed about the syllabus and the
     course web page [16]. One month before course       4. Course evaluation
     start and were asked to read the entire syllabus
     before the course started.                             Based on running the course twice we wanted to
b. Short introduction to the course content by the       evaluate it in order to plan its third iteration. One goal
     teacher.                                            of this evaluation was to get general suggestions for
c. Introduction to the field of empirical software       improvement. We wanted to reflect about the two ways
     engineering by discussing examples of interaction   of teaching ESE–individual presentations or group
     with the Norwegian software industry by two         assignments.
     Norwegian research managers.                           Educational evaluation is a sub field of educational
d. Group work with the goal of extracting the main       research for which there is an extensive bibliography
     issues from the syllabus. Each group had to find    along with national and international standards [26].
     and list research hypotheses, data, and their       To the best of our knowledge our course is one if the
     analysis. Moreover each group had to summarize      few offered internationally within the field. The
     one method to plan and perform investigations,      number of new software engineering PhD students in
     and to summarize one investigation.                 Norway is of the magnitude of ten. While there is a
e. Practical exercises coordinated by a performance      need to teach such a course and to reflect over its
     and theater instructor with the goal to introduce
learning effects and the benefits and risk of different         •     In your own words, what were the primary
pedagogical strategies, we are aware that time is not                objectives of the Empirical software engineering
mature for a formal evaluation of the course involving               course?
professional pedagogues and psychologists of the                    We did not ask directly what the students had
education service at our university.                            learnt, but what they thought were the primary
    However, we have designed and run an evaluation             objectives of the course. We did this to make sure the
of the two first iterations of the course. Our evaluation       students answered what they had learnt from the
attempts to reflect about how much the students had             course contents, not what the teachers were supposed
learnt from attending the course.                               to teach them. To see how well the course succeeded
    We decided to implement our investigation as an e-          in teaching the students the overall goal, to run
mail-based questionnaire [29] that we circulated to all         empirical evaluations, we asked the following:
students attending the courses.                                 • On the basis of what you have learned in this
    We decided to use Bloom’s taxonomy. The                          course, would you be able to plan and run an
columns Level, Definition, Sample verbs, and Sample                  empirical investigation?
behavior are taken directly from the Bloom taxonomy                 Finally, we formulated the final question:
of educational objectives [7]. Column “Sample                   • Do you have further comments on the course?
behavior ESE” is our contribution.                                  If we had asked the wrong questions, we hoped the
    We decomposed the level of learning reached by              students would tell us so in answering this question.
students with respect to the Bloom taxonomy in three            We were also interested in feedback on how to better
categories, one for each part of the syllabus:                  the course, and thought this question provided an
motivation, method, and actual studies (see section             opportunity for such feedback.
3.1.2 Syllabus). We wanted to know how much each                    Based on lists of the participants in the ESE course,
student has learnt from each part of the syllabus.              we circulated the questionnaire to all participants by e-
    The questionnaire had to be formulated in such a            mail. We included an introductory letter explaining
way that we could assess what the students have learnt          that we wanted to use the data for both improving the
and how well they have learnt. To this end we                   course and as material for a paper, that all responses
formulated the two questions:                                   would be treated confidentially, and a date within
• When did you take the Empirical software                      which we would like to receive the responses by. The
    engineering course? (Autumn 2002 or Summer                  responses to our questionnaire were on free form and
    2003?)                                                      are available at [16].
• Have you ever participated in an Empirical                        We e-mailed the survey to the seventeen students
    software engineering investigation? If yes, prior or        attending the two courses. In the end, due to data
    after attending the course?                                 discard, we had ten responses to analyze.
    Our goal was to measure how much of the course                  The data from the questionnaire and our analysis
contents the students had learnt from attending the             can be found at [16].
empirical software engineering course. We formulated            • The “stories” written by students provide a
the following question in order to measure how much                   valuable piece of knowledge for those evaluating
of the course contents the students had learnt:                       our course. To the question “In your own words,
                                                                      what were the primary objectives of the Empirical
                                                                      software engineering course?".
Table 2: Coding conventions.
                 Word/phrase in free form question          Translated meaning
Coding key 1     Learn                                      • Bloom’s taxonomy: knowledge level
                 insight                                    • Course topic: all
Coding key 2     Run experiment                             •   Bloom’s taxonomy: application level
                                                            •   Course topic: actual studies
Coding key 3     Plan and run experiment                    •   Bloom’s taxonomy: synthesis level
                                                            •   Course topic: actual studies
Coding key 4     Tradition                                  •   Bloom’s taxonomy: depends on other coding keys
                                                            •   Course topic: motivation
   To the question “On the basis of what you have         subjects (10) and the measurement scale (ordinal), it
learned in this course, would you be able to plan and     does not make sense to try statistical generalization.
run an empirical investigation?” students write answers
like:                                                     5. Discussion and Conclusions
•         “Yes. But more concrete skills about how to
design questionnaire and how to analyze data should           The ESE course is now mandatory for all new
be learned”.                                              software engineering PhD students at NTNU. We had
•         “Not without more input, but I would have a     only 4 new software engineering PhD students in
better starting point for the planning phase. A lot of    2004. For this reason, the teacher (Jaccheri) decided to
the work in the planning phase would consist of           organize the 4 students as a group with which she
deciding how the investigation should be formed, and I    interacted as a performing member for the third
think I have a better understanding about the basic       iteration of the course.
concepts and where I could find out more about                The transition from 2003 to 2004 benefited from the
them.”                                                    evaluation we report in this paper. The 2004 iteration
   To analyze our data, we employed the method of         is organized in a similar way to 2003 and it is based on
coding [25] for extracting qualitative data and map       group activities. The most important change with
them into Bloom’s taxonomy of learning. Based on the      respect to 2003 is the introduction of a “summary
responses and our interpretation of Bloom’s taxonomy,     writing” activity that proceeds the seminar. This is an
we therefore formulated a set of coding keys to ensure    individual activity during which each student is
identical coding of free form replies for both the        supposed to write a structured summary of the main
taxonomy and the course topics (Table 2).                 characteristics of the content part. Students get the
•         Coding key 1 enabled us to translate            assignment one month before the seminar. In this way,
sentences like “Provide some insight into typical         the 2004 iteration is more oriented toward individual
methods used in the field” into level knowledge for       learning that the 2003 iteration but still much more
topic method.                                             group oriented than the 2002 one. The learning goal of
•         Coding key 2 enabled us to code sentences       this summary activity is that students must have
like “Yes - it would give me an important starting        reached a knowledge level of the Bloom´s taxonomy
point, but I would always need to confer and discuss      before seminar starts. The next version (fourth) of the
afterwards possible investigations and their plans with   course will run during autumn 2005.
people with experience” into level application for            The main contribution of this work is the
topic actual studies.                                     description of a course in the field of empirical
•         Coding key 3 says that whenever a student       software engineering. First, the presented syllabus [16]
declares to have knowledge to plan an experiment, the     can be used as a basis for a dialog in the ESE
level for actual studies is synthesis since Create,       community about which topics are of most importance
Design, Hypothesize, Invent, and Develop are sample       when educating the future researchers in the field.
verbs for synthesis level in the Bloom taxonomy.          Second, the two presented pedagogical strategies can
Example is a sentence like: “Yes, I can design an         be discussed further to find out which one or which
empirical study now”.                                     combination of the two is better suited for which
•         Coding key 4 says that whenever a student       context. Another contribution is our customization of
acknowledges that the goals of the course encompass       Bloom´s taxonomy of learning to the ESE field.
the traditions of the field, we assigned a value to the       Like every other course, there are at least two axes:
motivation field. Example: in response to the question    one axis for education content and one for pedagogy.
“In your own words, what were the primary objectives      These two axes can be used also to reflect over this
of the empirical software engineering course?” one        course or similar ones. Which part of the content are
student replied “To given an overview of the main         we satisfied with and why? Which pedagogical
methods and traditions of empirical software              strategy work better in which situation? According to
engineering”.                                             our evaluation, students are generally satisfied with
   Along with the coding keys, we made extra rules        syllabus and the way it is structured into motivation,
for coding that can be found at [16] together with the    methods, and actual studies. Students appreciate
data set.                                                 examples provided by the papers in the investigation
   There is a slight difference between the levels        part. Concerning the method part, there is need to
obtained for autumn 2002 and summer 2003.                 focus more attention on case studies and interpretative
However, taken in consideration the number of             studies. Teachers and examiners are satisfied with the
essays and we believe that the guidelines provided in        6. Acknowldge
[22] are a good tool to characterize investigations
    Other ESE courses are offered at other universities,     We thank all PhD students who participated to the courses
like the one documented in [15]. However, if one asks        and who answered our questionnaire. We are in debt to
colleagues about how they learnt to become                   Reidar Conradi and Monica Divitini who set up the first
researchers in software engineering, or more specific        version of the syllabus and organized the first version of the
empirical software engineering or measurement, the           course. We thank Tore Dybå for useful discussions and
answer is often vague. Experienced researchers rely on       suggestions on the syllabus part. Special thanks go to Dag
                                                             Sjøberg and the Simula center group for giving us the
tacit knowledge which is learnt and shared locally by
                                                             possibilities to run the second and third version of our
research groups and internationally by research              course.
communities. A contribution of this paper is that it can
be an example for those who provide and want to
provide similar courses.
                                                             7. References
    Concerning our evaluation, we are aware that the
                                                             [1]       A. Abran, J. W. Moore, P. Bourque, R.
evaluation of the two pedagogical strategies has
                                                                       Dupuis, and L. L. Tripp, "Guide to the
limitations if one regards it as a piece of educational
                                                                       Software Engineering Body of Knowledge,
research evaluation. First the sample size (10) is too
                                                                       Trial Version SWEBOK, A Project of the
limited. Second we ask students to evaluate their own
                                                                       Software Engineering Coordinating
learning perception. Third, though we map the initial
                                                                       Committee," IEEE, May 2001.
knowledge of each student, we do not take gender,
                                                             [2]       E. Arisholm, B. Anda, M. Jørgensen, and D.
age, language, and country of origin into
                                                                       I. K. Sjøberg, "Guidelines on Conducting
consideration. Students have different age, gender,
                                                                       Software Process Improvement Studies in
nationality, and scientific background. While
                                                                       Industry," presented at 22nd IRIS
heterogeneity enhances learning challenges and
                                                                       (Information Systems Research Seminar In
possibilities, it poses serious problems to a formal
                                                                       Scandinavia) Conference, Keruu, Finland,
evaluation of the learning effects of the course.
    However, the evaluation, in addition to content and
                                                             [3]       D. E. Avison, F. Lau, M. D. Myers, and P. A.
pedagogy, are relevant to the ESE community and to
                                                                       Nielsen, " Action research," Communications
the software engineering community. This because
                                                                       of the ACM, vol. 42, num. 1, pp. 94-97, 1999.
despite several empirical studies being conducted with
                                                             [4]       D. J. Bagert, T. B. Hilburn, G. Hislop, M.
students as subjects [10], there is a lack of frameworks
                                                                       Lutz, M. McCracken, and S. Mengel,
for evaluating the learning effects of software
                                                                       "Guidelines for Software Engineering
engineering courses. Our work is of interest for the
                                                                       Education Version 1.0 (1999)," CMU/SEI
measurement community as it can serve as an example
                                                                       CMU/SEI-99-TR-032, 1999.
for those who want to evaluate the pedagogical effects
                                                             [5]       J. Bell, Doing your research project.
of courses where such experiments take place.
                                                                       Buckingham . Philadelphia: Open University
    One of the goals of our course is that of establishing
                                                                       Press, 230 pp., 2000.
a Norwegian network of young researcher in the field
                                                             [6]       A. Birk, T. Dingsøyr, and T. Stålhane,
of software engineering. From the students’
                                                                       "Postmortem: Never Leave a Project without
questionnaire answers and for living and working
                                                                       It," IEEE Software, vol. 19, num. 3, pp. 357-
together with these students, we evaluate the project
                                                                       390, 2000.
based version of the course to be successful to respect
                                                             [7]       B. S. Bloom, Taxonomy of educational
to this goal.
                                                                       objectives: The classification of educational
    There remain three open questions. First, that of
                                                                       goals: Handbook I, cognitive domain. New
comparing our course with other similar courses
                                                                       York, Toronto: Longmans, Green. 1956.
offered internationally. Secondly, one question which
                                                             [8]       L. Briand, E. Arisholm, S. Counsell, F.
is not attacked by our work is: is it possible to teach
                                                                       Houdek, and P. Thevenod-Fosse, "Empirical
PhD students how to cooperate with external actors?
                                                                       Studies of Object-Oriented Artifacts, Methods
The final issue that we have not discussed is empirical
                                                                       and Processes: State of the Art and Future
software engineering education for practitioners and
                                                                       Directions," Empirical Software Engineering,
not only for researchers.
                                                                       num. 4, pp. 385-402, 1999.
                                                             [9]       L. C. Briand, S. Morasca, and V. R. Basili,
                                                                       "An operational process for goal-driven
       definition of measures," IEEE Transactions             10th ACM\IEEE-CS Conference on Software
       on Software Engineering, vol. 12, num. 4, pp.          Engineering Education and Training, Virginia
       1106- 1125, 2002.                                      Beach.
[10]   J. Carver, L. Jaccheri, S. Morasca, and F.      [20]   J. Kirk and M. L. Miller, Reliability and
       Shull, "Issues in Using Students in Empirical          Validity in Qualitative Research, vol. 1.
       Studies in Software Engineering Education,"            Newbury Park, California: SAGE
       presented at International Software Metrics            Publications Inc., 1986.
       Symposium, Sydney, Australia., 2003.            [21]   B. Kitchenham, "DESMET: A Method for
[11]   R. Conradi and T. Dybå, "An Empirical Study            Evaluating Software Engineering Methods
       on the Utility of Formal Routines to Transfer          and Tools," Keele University, Technical
       Knowledge and Experience," presented at 8th            Report TR96-09, August 1996.
       European software engineering conference        [22]   B. A. Kitchenham, S. L. Pfleeger, L. M.
       held jointly with 9th ACM SIGSOFT                      Pickard, P. W. Jones, D. C. Hoaglin, K. El
       international symposium on Foundations of              Emam, and J. Rosenberg, " Preliminary
       software engineering, Vienna, Austria, 2001.           guidelines for empirical research in software
[12]   P. J. Denning, "Computer Science: The                  engineering," IEEE Transactions on Software
       Discipline," Encyclopedia of Computer                  Engineering, vol. 28, num. 8, pp. 721 -734,
       Science, 2000.                                         2002.
[13]   T. Dybå, "An Instrument for Measuring the       [23]   O. Laitenberger and J.-M. DeBaud,
       Key Factors of Success in Software Process             "Perspective-based Reading of Code
       Improvement," Empirical Software                       Documents at Robert Bosch GmbH,"
       Engineering, Kluwer Academic Publishers,               Information and Software Technology, vol.
       vol. 5, pp. 357-390, 2000.                             31, num. 11, pp. 781-791, 1997.
[14]   N. Fenton, "Software Measurement: A             [24]   C. T. Lethbridge, "On the Relevance of
       Necessary Scientific Basis," IEEE                      software education: A survey and some
       Transactions on Software Engineering, vol.             Recommendations," Annals of Software
       20, num. 3, pp. 199-205, 1994.                         Engineering, vol. 6, pp. 91/110, 1998.
[15]   M. Höst, "Introducing Empirical Software        [25]   B. Meyer, "Software Engineering in the
       Engineering Methods in Education,"                     Academy," IEEE Computer, vol. 34, num. 5,
       presented at Conference on Software                    pp. 28-35, 2001.
       Engineering Education and Training              [26]   Meredith D. Gall, Walter R. Borg, Joyce P.
       CSEE&T, Covington, Northern Kentucky,                  Gall, ISBN: 0-321-08189-7, Allyn & Bacon,
       USA., 2002.                                            2003, 656 pp
[16]   L. Jaccheri and T. Østerlie, "Empirical         [27]   J. Miller, J. Daly, M. Wood, M. Roper, and A.
       software engineering Course," Trondheim -              Brooks, "Statistical Power and its
       Oslo, Norway,                                          Subcomponents: Missing and Misunderstood
       http://www.idi.ntnu.no/emner/empse/, last              Concepts in Empirical Software Engineering
       accessed February 2004.                                Research," Information and Software
[17]   L. Jaccheri, A software quality and software           Technology, vol. 39, num. 4, pp. 285-295,
       process improvement course based on                    1995.
       interaction with the local software software    [28]   M. Morisio, M. Ezran, and C. Tully, "Success
       industry, in the Tutorial section of Computer          and Failure Factors in Software Reuse," IEEE
       Applications in Engineering Education                  Transactions on Software Engineering, vol.
       Journal, John Wiley and Sons, Inc. page 265-           28, num. 4, pp. 340-357, 2002.
       272, Mar 2002.                                  [29]   C. B. Seaman, "Qualitative Methods in
[18]   L. Jaccheri and P. Lago, Metrics aspects of a          Empirical Studies of Software Engineering,"
       software engineering course project,                   IEEE Transactions on Software Engineering,
       September 1997, in Proc. of Inspire II, 2nd            vol. 25, num. 4, pp. 557-572, 1999.
       International Conference on Software Process    [30]   D. I. K. Sjøberg, B. Anda, T. Dybå, M.
       Improvement - Research into Education &                Jørgensen, A. Karahasanovic, E. F. Koren,
       Training, Sweden.                                      and M. Vokac, "Conducting Realistic
[19]   L. Jaccheri and P. Lago, "Applying Software            Experiments in Software Engineering,"
       Process Modeling and Improvement in                    presented at First International Symposium on
       Academic Setting", April 1997, Proc. of the
       Empirical Software Engineering, Nara, Japan,
[31]   W. F. Tichy, "Should Computer Scientists
       Experiment More?," IEEE Computer, vol. 31,
       num. 5, pp. 32-40, 1998.
[32]   C. Wohlin, P. Runeson, M. Host, M. C.
       Ohlsson, B. Regnell, and A. Wesslen,
       Experimentation in Software Engineering: An
       Introduction: Kluwer Academic Publishers.
[33]   M. V. a. Zelkowitz and D. R. Wallace, "
       Experimental Models for Validating
       Technology," IEEE Computer, vol. 31, num.
       5, pp. 23-31, 1998.

To top