Can we teach Empirical Software Engineering?
Letizia Jaccheri and Thomas Østerlie
Department of Computer and Information Science (IDI)
Norwegian University of Science and Technology (NTNU)
Abstract course for teaching ESE to software engineering
researchers in this paper. We believe that our findings
We report about an empirical software engineering are equally applicable for teaching ESE to
course for PhD students. We introduce its syllabus and practitioners.
two different pedagogical strategies. The first strategy There are some fundamental challenges for an ESE
is based on individual learning and presentations. The research project to succeed. First, researchers in
second relies also on social activities to support general and PhD students in particular must be well
learning and knowledge sharing. The syllabus, which acquainted with existing methods. Second, ESE
has been used for three iterations of the course, is research is a major undertaking and it is a cooperative
available at our web site together with student essays, activity within a research group. Third, the research
evaluation data, and other documentation produced needs to be relevant outside the research community.
during course runs. The research group must therefore have access,
knowledge of and familiarity with the software
1. Introduction industry in order to study concrete and real situations,
and to generate industry relevant research questions.
Lastly, the research problems must also be relevant
Empirical software engineering (ESE) is a sub field within the academic field of software engineering.
of software engineering which aims at applying Research problems therefore have to be significant to
empirical theories and methods for the measuring, both the international research community and the
understanding, and improvement of the software local industry. This means that research problems and
development process in organizations. ESE is by its questions must be shared and understood by the all
nature a multi-disciplinary field as software engineers, members of the research team and by the industrial
industry actors, statisticians, pedagogues, and actors. These challenges need to be addressed by a
psychologists have traditionally been cooperating. PhD level course in empirical software engineering.
In this paper, we report on a PhD level course in Our course is an attempt to make our PhD students
empirical software engineering that has been run three acquainted with the state of the art within ESE as well
times. The course held during the autumn of 2002 was as reflect on investigations done by others and in
based on individual presentation. During the spring of which they have possibly participated.
2003 the course was based on group work held. This Our course has been run three times and its
was replicated when the course was held in 2004. syllabus, program, and evaluation is available at .
The main objective of empirical software We have evaluated the pedagogical effects of the
engineering education is to train software engineers in course by exploiting Bloom´s taxonomy of learning
empirical evaluation of the tools, techniques, and (which is well known and used by the software
technologies used in software engineering. It is in this engineering community) and qualitative methods for
context, that we see the importance in discussing the data collection and analysis  as applied in the ESE
strategy for teaching empirical software engineering. field.
We are of the opinion that ESE is relevant for both This paper is structured as follows. Section 2
practitioners and researchers. For practitioners it is introduces the aspects of software engineering
about evaluating tools and techniques for use in education which have been relevant to our work and
concrete cases . While it is equally important to some learning issues in research education. Section 3
teach ESE to each of these two groups., we report on a describes our course, its syllabus, pedagogical goals
and the two different strategies. Section 4 is about the In this sense, their approaches can be classified as
evaluation of the course. Discussion and conclusions addressing short-term requirement vs. addressing long-
are given in section 5. term requirements. Guidelines for Software
Engineering Education  adopt a middle ground
2 Background approach to education contents. They address the long-
Software engineering, as a field, has, among others, term issues and are based on the body of knowledge
two supporting disciplines–software engineering for software engineering . This body of knowledge,
education and ESE. Software engineering education however, is based upon expert opinions within the
focuses on training "software professionals" for the industry.
industry , while ESE focuses on evaluating the Along the second axis, there are two strategies: the
tools and techniques used, developed, and intended for first strategy is based on lectures and individual
use in the industry through empirical validation. There learning. The second strategy is based on learning by
is an element of training required in making the doing, also known as project-based learning. Both
transition from trained software professional, with or Meyer  and  favor project-based learning.
without working experience, to become an ESE Unlike the education content, they argue in favor of
researcher. project-based learning to "prepare our students for the
real challenges they will face" ( p.33). They also
2.1 Software Engineering Education argue that it is easier to learn from personal mistakes
rather than mistakes related by a lecturer.
The software engineering education literature
moves along a dual axis; one axis for education 2.2 Learning issues in research education
content and one for pedagogy. We use the two axes to
reflect on the current state within software engineering Provided that it is possible to teach somebody how
literature in this section. to become a good researcher, there are three kinds of
The education content axis is delimited by two courses that can be offered as part of research
extremes: industry driven and principles driven. Meyer education:
 argues that the contents of software engineering 1. General courses on research methods at both
education must be driven by the principles on which undergraduate and post graduate level are usually
software engineering is based: offered by social science faculties. These courses
"What matters is teaching [the students] address research issues such as scientific method
fundamental modes of thought that will accompany and nature of evidence, advocacy versus evidence-
them throughout their careers and help them grow in based approaches, writing and reviewing research
this ever-changing field. The ones who blossom are proposals, how to use bibliographies and citation
those who can rise beyond the tools of the moment in searches, project planning, selecting results and
harmony with the progress of the discipline. ( places to publish, outlining and structuring
p.29)" research papers, the peer review process,
On the other end is the work of Lethbridge . presenting posters and papers at conferences,
Based on a survey of 168 software engineers, he finds publishing in academic and engineering journals,
significant differences between curricula taught at etc..
colleges and universities and the actual knowledge 2. Courses on research methods in computer science
required in the industry. Lethbridge argues for aligning address some of the research issues above, such as
existing curricula with skills required by the industry. scientific method and nature of evidence,
Where Meyer  is specific on the need to distance customized to the IT field. At our department, for
education from the industry’s immediate, short-term example, there is a common introductory course
requirement, Lethbridge writes little of the long-term for all PhD students, which addresses general
requirements that Meyer addresses with his principles. research issues in IT like those discussed for
example in .
Table 1 Bloom’s taxonomy of learning customized to ESE
Level Definition Sample Sample behavior Sample behavior ESE
Knowledge Student recalls or Write The student will The students will be able to
recognizes information, List define the 6 levels define the content of the different
ideas, and principles Label of Bloom's papers.
in the approximate Name taxonomy of the
form in which they State cognitive domain.
were learned. Define
Comprehension Student translates, Explain The student will The students will explain the
comprehends, or Summarize explain the purpose of the give methods and
interprets information Paraphrase purpose of investigations.
based on prior Describe Bloom's
learning. Illustrate taxonomy of the
Application Student selects, trans- Use The student will The student will be able to use
fers, and uses data Compute write an one method for experimentation.
and principles to Solve instructional
complete a problem Demonstrate objective for each
or task with a mini- Apply level of Bloom's
mum of direction. Construct taxonomy.
Analysis Student distinguishes, Analyze The student will The students will compare and
classifies, and relates Categorize compare and contrast different methods and
the assumptions, Compare contrast the investigations.
hypotheses, evidence, Contrast cognitive and
or structure of a Separate affective domains.
statement or question.
Synthesis Student originates, Create The student will The students will design an
integrates, and Design design a investigation by choosing and
combines ideas into a Hypothesize classification perhaps combining different
product, plan or Invent scheme for methods.
proposal that is new Develop writing
to him or her. educational
Evaluation Student appraises, Judge The student will The students will judge the
assesses, or critiques Recommend judge the effectiveness of using empirical
on a basis of specific Critique effectiveness of software engineering methods.
standards and criteria. Justify writing objectives
3.1 Common characteristics
3. Courses like the one we describe in this paper,
which address empirical research methods in In the following section we discuss the common
software engineering. These courses do not exist characteristics for the course.
in isolation but in education and research context
that may or may not include general research 3.1.1 Students
courses or courses specific for IT research. As far
as we know, there is at least one paper that reports The empirical software engineering course counts as
on teaching ESE  as part of a software 7.5 European Credit Transfer System (ECTS). This is
engineering course. Undergraduate students work equivalent to 12 hours work for 15 weeks, 2 hours in
in projects, and the teachers play the role of the class and 10 hours of other learning activities, for a
customer. In one project the customer is a total of 180 hours. The course was run once during the
hypothetical company wanting the students to autumn of 2002 at the Norwegian University of
perform empirical studies (mainly experiments) to Science and Technology (abbreviated NTNU) for PhD
evaluate different alternative techniques, e.g., students by the software engineering group, and once
different kinds of reviews. Students are not asked during the summer of 2003 at Simula Research
to plan the studies but only to perform them, as Laboratory in Oslo for both University of Oslo and
planning and running would take too much of the NTNU PhD students.
course. The syllabus of the empirical software Some of the students worked in empirical software
engineering part is . engineering research projects, while others worked in
other kinds of software engineering projects. All
2.2 Software engineering education and students had a general course about research methods
research education at IDI in IT as part of their curriculum. Some students even
had a course on research methods in general. The
The software engineering group at IDI has thirty students age, gender, nationality, and scientific
years experience with project based software background differed.
engineering education. One of the author of this paper,
Jaccheri, has more than ten years experience with 3.1.2 Syllabus
teaching project based software engineering, software
quality and metrics issues to software engineering The course has had the same syllabus throughout.
students, both in Italy and in Norway, as well as Here, we refer to a consolidated bulk of literature,
reflecting and writing about this   . IDI has which is used as syllabus for the course.
7 years experience with an introductive course for IT The syllabus is divided into three parts: motivation,
researchers, which is mandatory for all new PhD method, and actual investigations.
students. Jaccheri has had the main responsibility for • Motivation: In  and in , motivations for
this course for one year. IDI graduates twenty PhD the existence of the ESE field are given. These
candidates each year and the software engineering papers provide also a classification of existing
group has graduated a total of 19 software engineering software engineering research papers according to
PhD students. the kind of empirical method used in the
3. Empirical software engineering course • Methods:  provides an introduction to the
field, with special emphasis on experiments. ,
The course reported in this paper is offered to PhD , , , , , , and , provide
students within software engineering. While the course concrete methods for performing, and analyzing
has been run three times, we mainly report on the first investigations.  and  are about Data
two times as the third time we ran the course as a Analysis Methods.
replication of the second time. The first and the second • Actual studies: , , , and  are about
times had some common characteristics and some concrete investigations.
different characteristic. In the following section we The course ended with a final oral exam with the
discuss the common characteristics and afterwards we teacher and an ESE expert outside of the university as
introduce the characteristics that were different for the examiner.
3.2 Differing characteristics students to performance work and to stimulate
Here we introduce the characteristics that differed. cooperation among students.
f. Production of a five minutes movie advertising the
3.2.1 Pedagogical goals and teaching strategy for field of empirical software engineering.
Autumn 2002 g. Choosing an actual study in the syllabus,
characterize the investigation presented in the
In the autumn 2002 iteration of the course, the goal chosen study according to motivations, method,
was to make the students acquainted with the contents measurements, and data analysis method. The goal
of the syllabus. The course was held in a classroom of this exercise was to simulate the planning and
context where students were met two hours a week for execution of an empirical investigation.
13 weeks (totally 26 hours in class). h. Group work with the goal of obtaining a deep and
At each meeting, one student presented one paper critical understanding of the issues presented in
from the syllabus. The teachers responsible for the the syllabus, also in light of the experience
course provided feedback and stimulated discussion acquired during the previous exercises. Each
around the paper. group had to: choose one research hypothesis and
its motivation (business, education, others); list
3.2.2 Pedagogical goals and teaching strategy for and comment investigation planning and risks
Summer 2003 (what can go wrong); define (or reuse) guidelines
for designing and running empirical
In the summer 2003 version of our course, the investigations; choose one actual investigation.
pedagogical goals were: i. Essay writing. Each student had to write an
1. given our syllabus which introduces a possible individual essay. The essay assignment was
overview of empirical software engineering “Extract from  and from the other papers in
knowledge, let our students know about the the Methods part of the syllabus, the most
syllabus at a level with is as high as possible important guidelines for designing and running
according to the Bloom´s taxonomy . empirical investigations. Characterize the
2. establish a Norwegian network of young investigation presented in the chosen actual study,
researchers within the field of software according to these guidelines. This means that you
engineering. have to comment about the motivations, the
Bloom’s taxonomy is reported in Table 1. The last method, the measurements, and the data analysis
column of the table (Sample behavior ESE) gives our method”. The essay was supposed to be handed in
interpretation of the taxonomy when applied to the two weeks after the seminar and one week before
ESE domain. the oral exam. Recalling that students have 180
The second iteration of the course was organized as a hours allocated to this course, we expected that
three days event (21 hours). Here we list in students worked full time writing the essay during
chronological order the main tasks of the course. these two weeks.
a. Students were informed about the syllabus and the
course web page . One month before course 4. Course evaluation
start and were asked to read the entire syllabus
before the course started. Based on running the course twice we wanted to
b. Short introduction to the course content by the evaluate it in order to plan its third iteration. One goal
teacher. of this evaluation was to get general suggestions for
c. Introduction to the field of empirical software improvement. We wanted to reflect about the two ways
engineering by discussing examples of interaction of teaching ESE–individual presentations or group
with the Norwegian software industry by two assignments.
Norwegian research managers. Educational evaluation is a sub field of educational
d. Group work with the goal of extracting the main research for which there is an extensive bibliography
issues from the syllabus. Each group had to find along with national and international standards .
and list research hypotheses, data, and their To the best of our knowledge our course is one if the
analysis. Moreover each group had to summarize few offered internationally within the field. The
one method to plan and perform investigations, number of new software engineering PhD students in
and to summarize one investigation. Norway is of the magnitude of ten. While there is a
e. Practical exercises coordinated by a performance need to teach such a course and to reflect over its
and theater instructor with the goal to introduce
learning effects and the benefits and risk of different • In your own words, what were the primary
pedagogical strategies, we are aware that time is not objectives of the Empirical software engineering
mature for a formal evaluation of the course involving course?
professional pedagogues and psychologists of the We did not ask directly what the students had
education service at our university. learnt, but what they thought were the primary
However, we have designed and run an evaluation objectives of the course. We did this to make sure the
of the two first iterations of the course. Our evaluation students answered what they had learnt from the
attempts to reflect about how much the students had course contents, not what the teachers were supposed
learnt from attending the course. to teach them. To see how well the course succeeded
We decided to implement our investigation as an e- in teaching the students the overall goal, to run
mail-based questionnaire  that we circulated to all empirical evaluations, we asked the following:
students attending the courses. • On the basis of what you have learned in this
We decided to use Bloom’s taxonomy. The course, would you be able to plan and run an
columns Level, Definition, Sample verbs, and Sample empirical investigation?
behavior are taken directly from the Bloom taxonomy Finally, we formulated the final question:
of educational objectives . Column “Sample • Do you have further comments on the course?
behavior ESE” is our contribution. If we had asked the wrong questions, we hoped the
We decomposed the level of learning reached by students would tell us so in answering this question.
students with respect to the Bloom taxonomy in three We were also interested in feedback on how to better
categories, one for each part of the syllabus: the course, and thought this question provided an
motivation, method, and actual studies (see section opportunity for such feedback.
3.1.2 Syllabus). We wanted to know how much each Based on lists of the participants in the ESE course,
student has learnt from each part of the syllabus. we circulated the questionnaire to all participants by e-
The questionnaire had to be formulated in such a mail. We included an introductory letter explaining
way that we could assess what the students have learnt that we wanted to use the data for both improving the
and how well they have learnt. To this end we course and as material for a paper, that all responses
formulated the two questions: would be treated confidentially, and a date within
• When did you take the Empirical software which we would like to receive the responses by. The
engineering course? (Autumn 2002 or Summer responses to our questionnaire were on free form and
2003?) are available at .
• Have you ever participated in an Empirical We e-mailed the survey to the seventeen students
software engineering investigation? If yes, prior or attending the two courses. In the end, due to data
after attending the course? discard, we had ten responses to analyze.
Our goal was to measure how much of the course The data from the questionnaire and our analysis
contents the students had learnt from attending the can be found at .
empirical software engineering course. We formulated • The “stories” written by students provide a
the following question in order to measure how much valuable piece of knowledge for those evaluating
of the course contents the students had learnt: our course. To the question “In your own words,
what were the primary objectives of the Empirical
software engineering course?".
Table 2: Coding conventions.
Word/phrase in free form question Translated meaning
Coding key 1 Learn • Bloom’s taxonomy: knowledge level
insight • Course topic: all
Coding key 2 Run experiment • Bloom’s taxonomy: application level
• Course topic: actual studies
Coding key 3 Plan and run experiment • Bloom’s taxonomy: synthesis level
• Course topic: actual studies
Coding key 4 Tradition • Bloom’s taxonomy: depends on other coding keys
• Course topic: motivation
To the question “On the basis of what you have subjects (10) and the measurement scale (ordinal), it
learned in this course, would you be able to plan and does not make sense to try statistical generalization.
run an empirical investigation?” students write answers
like: 5. Discussion and Conclusions
• “Yes. But more concrete skills about how to
design questionnaire and how to analyze data should The ESE course is now mandatory for all new
be learned”. software engineering PhD students at NTNU. We had
• “Not without more input, but I would have a only 4 new software engineering PhD students in
better starting point for the planning phase. A lot of 2004. For this reason, the teacher (Jaccheri) decided to
the work in the planning phase would consist of organize the 4 students as a group with which she
deciding how the investigation should be formed, and I interacted as a performing member for the third
think I have a better understanding about the basic iteration of the course.
concepts and where I could find out more about The transition from 2003 to 2004 benefited from the
them.” evaluation we report in this paper. The 2004 iteration
To analyze our data, we employed the method of is organized in a similar way to 2003 and it is based on
coding  for extracting qualitative data and map group activities. The most important change with
them into Bloom’s taxonomy of learning. Based on the respect to 2003 is the introduction of a “summary
responses and our interpretation of Bloom’s taxonomy, writing” activity that proceeds the seminar. This is an
we therefore formulated a set of coding keys to ensure individual activity during which each student is
identical coding of free form replies for both the supposed to write a structured summary of the main
taxonomy and the course topics (Table 2). characteristics of the content part. Students get the
• Coding key 1 enabled us to translate assignment one month before the seminar. In this way,
sentences like “Provide some insight into typical the 2004 iteration is more oriented toward individual
methods used in the field” into level knowledge for learning that the 2003 iteration but still much more
topic method. group oriented than the 2002 one. The learning goal of
• Coding key 2 enabled us to code sentences this summary activity is that students must have
like “Yes - it would give me an important starting reached a knowledge level of the Bloom´s taxonomy
point, but I would always need to confer and discuss before seminar starts. The next version (fourth) of the
afterwards possible investigations and their plans with course will run during autumn 2005.
people with experience” into level application for The main contribution of this work is the
topic actual studies. description of a course in the field of empirical
• Coding key 3 says that whenever a student software engineering. First, the presented syllabus 
declares to have knowledge to plan an experiment, the can be used as a basis for a dialog in the ESE
level for actual studies is synthesis since Create, community about which topics are of most importance
Design, Hypothesize, Invent, and Develop are sample when educating the future researchers in the field.
verbs for synthesis level in the Bloom taxonomy. Second, the two presented pedagogical strategies can
Example is a sentence like: “Yes, I can design an be discussed further to find out which one or which
empirical study now”. combination of the two is better suited for which
• Coding key 4 says that whenever a student context. Another contribution is our customization of
acknowledges that the goals of the course encompass Bloom´s taxonomy of learning to the ESE field.
the traditions of the field, we assigned a value to the Like every other course, there are at least two axes:
motivation field. Example: in response to the question one axis for education content and one for pedagogy.
“In your own words, what were the primary objectives These two axes can be used also to reflect over this
of the empirical software engineering course?” one course or similar ones. Which part of the content are
student replied “To given an overview of the main we satisfied with and why? Which pedagogical
methods and traditions of empirical software strategy work better in which situation? According to
engineering”. our evaluation, students are generally satisfied with
Along with the coding keys, we made extra rules syllabus and the way it is structured into motivation,
for coding that can be found at  together with the methods, and actual studies. Students appreciate
data set. examples provided by the papers in the investigation
There is a slight difference between the levels part. Concerning the method part, there is need to
obtained for autumn 2002 and summer 2003. focus more attention on case studies and interpretative
However, taken in consideration the number of studies. Teachers and examiners are satisfied with the
essays and we believe that the guidelines provided in 6. Acknowldge
 are a good tool to characterize investigations
Other ESE courses are offered at other universities, We thank all PhD students who participated to the courses
like the one documented in . However, if one asks and who answered our questionnaire. We are in debt to
colleagues about how they learnt to become Reidar Conradi and Monica Divitini who set up the first
researchers in software engineering, or more specific version of the syllabus and organized the first version of the
empirical software engineering or measurement, the course. We thank Tore Dybå for useful discussions and
answer is often vague. Experienced researchers rely on suggestions on the syllabus part. Special thanks go to Dag
Sjøberg and the Simula center group for giving us the
tacit knowledge which is learnt and shared locally by
possibilities to run the second and third version of our
research groups and internationally by research course.
communities. A contribution of this paper is that it can
be an example for those who provide and want to
provide similar courses.
Concerning our evaluation, we are aware that the
 A. Abran, J. W. Moore, P. Bourque, R.
evaluation of the two pedagogical strategies has
Dupuis, and L. L. Tripp, "Guide to the
limitations if one regards it as a piece of educational
Software Engineering Body of Knowledge,
research evaluation. First the sample size (10) is too
Trial Version SWEBOK, A Project of the
limited. Second we ask students to evaluate their own
Software Engineering Coordinating
learning perception. Third, though we map the initial
Committee," IEEE, May 2001.
knowledge of each student, we do not take gender,
 E. Arisholm, B. Anda, M. Jørgensen, and D.
age, language, and country of origin into
I. K. Sjøberg, "Guidelines on Conducting
consideration. Students have different age, gender,
Software Process Improvement Studies in
nationality, and scientific background. While
Industry," presented at 22nd IRIS
heterogeneity enhances learning challenges and
(Information Systems Research Seminar In
possibilities, it poses serious problems to a formal
Scandinavia) Conference, Keruu, Finland,
evaluation of the learning effects of the course.
However, the evaluation, in addition to content and
 D. E. Avison, F. Lau, M. D. Myers, and P. A.
pedagogy, are relevant to the ESE community and to
Nielsen, " Action research," Communications
the software engineering community. This because
of the ACM, vol. 42, num. 1, pp. 94-97, 1999.
despite several empirical studies being conducted with
 D. J. Bagert, T. B. Hilburn, G. Hislop, M.
students as subjects , there is a lack of frameworks
Lutz, M. McCracken, and S. Mengel,
for evaluating the learning effects of software
"Guidelines for Software Engineering
engineering courses. Our work is of interest for the
Education Version 1.0 (1999)," CMU/SEI
measurement community as it can serve as an example
for those who want to evaluate the pedagogical effects
 J. Bell, Doing your research project.
of courses where such experiments take place.
Buckingham . Philadelphia: Open University
One of the goals of our course is that of establishing
Press, 230 pp., 2000.
a Norwegian network of young researcher in the field
 A. Birk, T. Dingsøyr, and T. Stålhane,
of software engineering. From the students’
"Postmortem: Never Leave a Project without
questionnaire answers and for living and working
It," IEEE Software, vol. 19, num. 3, pp. 357-
together with these students, we evaluate the project
based version of the course to be successful to respect
 B. S. Bloom, Taxonomy of educational
to this goal.
objectives: The classification of educational
There remain three open questions. First, that of
goals: Handbook I, cognitive domain. New
comparing our course with other similar courses
York, Toronto: Longmans, Green. 1956.
offered internationally. Secondly, one question which
 L. Briand, E. Arisholm, S. Counsell, F.
is not attacked by our work is: is it possible to teach
Houdek, and P. Thevenod-Fosse, "Empirical
PhD students how to cooperate with external actors?
Studies of Object-Oriented Artifacts, Methods
The final issue that we have not discussed is empirical
and Processes: State of the Art and Future
software engineering education for practitioners and
Directions," Empirical Software Engineering,
not only for researchers.
num. 4, pp. 385-402, 1999.
 L. C. Briand, S. Morasca, and V. R. Basili,
"An operational process for goal-driven
definition of measures," IEEE Transactions 10th ACM\IEEE-CS Conference on Software
on Software Engineering, vol. 12, num. 4, pp. Engineering Education and Training, Virginia
1106- 1125, 2002. Beach.
 J. Carver, L. Jaccheri, S. Morasca, and F.  J. Kirk and M. L. Miller, Reliability and
Shull, "Issues in Using Students in Empirical Validity in Qualitative Research, vol. 1.
Studies in Software Engineering Education," Newbury Park, California: SAGE
presented at International Software Metrics Publications Inc., 1986.
Symposium, Sydney, Australia., 2003.  B. Kitchenham, "DESMET: A Method for
 R. Conradi and T. Dybå, "An Empirical Study Evaluating Software Engineering Methods
on the Utility of Formal Routines to Transfer and Tools," Keele University, Technical
Knowledge and Experience," presented at 8th Report TR96-09, August 1996.
European software engineering conference  B. A. Kitchenham, S. L. Pfleeger, L. M.
held jointly with 9th ACM SIGSOFT Pickard, P. W. Jones, D. C. Hoaglin, K. El
international symposium on Foundations of Emam, and J. Rosenberg, " Preliminary
software engineering, Vienna, Austria, 2001. guidelines for empirical research in software
 P. J. Denning, "Computer Science: The engineering," IEEE Transactions on Software
Discipline," Encyclopedia of Computer Engineering, vol. 28, num. 8, pp. 721 -734,
Science, 2000. 2002.
 T. Dybå, "An Instrument for Measuring the  O. Laitenberger and J.-M. DeBaud,
Key Factors of Success in Software Process "Perspective-based Reading of Code
Improvement," Empirical Software Documents at Robert Bosch GmbH,"
Engineering, Kluwer Academic Publishers, Information and Software Technology, vol.
vol. 5, pp. 357-390, 2000. 31, num. 11, pp. 781-791, 1997.
 N. Fenton, "Software Measurement: A  C. T. Lethbridge, "On the Relevance of
Necessary Scientific Basis," IEEE software education: A survey and some
Transactions on Software Engineering, vol. Recommendations," Annals of Software
20, num. 3, pp. 199-205, 1994. Engineering, vol. 6, pp. 91/110, 1998.
 M. Höst, "Introducing Empirical Software  B. Meyer, "Software Engineering in the
Engineering Methods in Education," Academy," IEEE Computer, vol. 34, num. 5,
presented at Conference on Software pp. 28-35, 2001.
Engineering Education and Training  Meredith D. Gall, Walter R. Borg, Joyce P.
CSEE&T, Covington, Northern Kentucky, Gall, ISBN: 0-321-08189-7, Allyn & Bacon,
USA., 2002. 2003, 656 pp
 L. Jaccheri and T. Østerlie, "Empirical  J. Miller, J. Daly, M. Wood, M. Roper, and A.
software engineering Course," Trondheim - Brooks, "Statistical Power and its
Oslo, Norway, Subcomponents: Missing and Misunderstood
http://www.idi.ntnu.no/emner/empse/, last Concepts in Empirical Software Engineering
accessed February 2004. Research," Information and Software
 L. Jaccheri, A software quality and software Technology, vol. 39, num. 4, pp. 285-295,
process improvement course based on 1995.
interaction with the local software software  M. Morisio, M. Ezran, and C. Tully, "Success
industry, in the Tutorial section of Computer and Failure Factors in Software Reuse," IEEE
Applications in Engineering Education Transactions on Software Engineering, vol.
Journal, John Wiley and Sons, Inc. page 265- 28, num. 4, pp. 340-357, 2002.
272, Mar 2002.  C. B. Seaman, "Qualitative Methods in
 L. Jaccheri and P. Lago, Metrics aspects of a Empirical Studies of Software Engineering,"
software engineering course project, IEEE Transactions on Software Engineering,
September 1997, in Proc. of Inspire II, 2nd vol. 25, num. 4, pp. 557-572, 1999.
International Conference on Software Process  D. I. K. Sjøberg, B. Anda, T. Dybå, M.
Improvement - Research into Education & Jørgensen, A. Karahasanovic, E. F. Koren,
Training, Sweden. and M. Vokac, "Conducting Realistic
 L. Jaccheri and P. Lago, "Applying Software Experiments in Software Engineering,"
Process Modeling and Improvement in presented at First International Symposium on
Academic Setting", April 1997, Proc. of the
Empirical Software Engineering, Nara, Japan,
 W. F. Tichy, "Should Computer Scientists
Experiment More?," IEEE Computer, vol. 31,
num. 5, pp. 32-40, 1998.
 C. Wohlin, P. Runeson, M. Host, M. C.
Ohlsson, B. Regnell, and A. Wesslen,
Experimentation in Software Engineering: An
Introduction: Kluwer Academic Publishers.
 M. V. a. Zelkowitz and D. R. Wallace, "
Experimental Models for Validating
Technology," IEEE Computer, vol. 31, num.
5, pp. 23-31, 1998.