Can we teach Empirical Software Engineering? Letizia Jaccheri and Thomas Østerlie Department of Computer and Information Science (IDI) Norwegian University of Science and Technology (NTNU) email@example.com, firstname.lastname@example.org Abstract course for teaching ESE to software engineering researchers in this paper. We believe that our findings We report about an empirical software engineering are equally applicable for teaching ESE to course for PhD students. We introduce its syllabus and practitioners. two different pedagogical strategies. The first strategy There are some fundamental challenges for an ESE is based on individual learning and presentations. The research project to succeed. First, researchers in second relies also on social activities to support general and PhD students in particular must be well learning and knowledge sharing. The syllabus, which acquainted with existing methods. Second, ESE has been used for three iterations of the course, is research is a major undertaking and it is a cooperative available at our web site together with student essays, activity within a research group. Third, the research evaluation data, and other documentation produced needs to be relevant outside the research community. during course runs. The research group must therefore have access, knowledge of and familiarity with the software 1. Introduction industry in order to study concrete and real situations, and to generate industry relevant research questions. Lastly, the research problems must also be relevant Empirical software engineering (ESE) is a sub field within the academic field of software engineering. of software engineering which aims at applying Research problems therefore have to be significant to empirical theories and methods for the measuring, both the international research community and the understanding, and improvement of the software local industry. This means that research problems and development process in organizations. ESE is by its questions must be shared and understood by the all nature a multi-disciplinary field as software engineers, members of the research team and by the industrial industry actors, statisticians, pedagogues, and actors. These challenges need to be addressed by a psychologists have traditionally been cooperating. PhD level course in empirical software engineering. In this paper, we report on a PhD level course in Our course is an attempt to make our PhD students empirical software engineering that has been run three acquainted with the state of the art within ESE as well times. The course held during the autumn of 2002 was as reflect on investigations done by others and in based on individual presentation. During the spring of which they have possibly participated. 2003 the course was based on group work held. This Our course has been run three times and its was replicated when the course was held in 2004. syllabus, program, and evaluation is available at . The main objective of empirical software We have evaluated the pedagogical effects of the engineering education is to train software engineers in course by exploiting Bloom´s taxonomy of learning empirical evaluation of the tools, techniques, and (which is well known and used by the software technologies used in software engineering. It is in this engineering community) and qualitative methods for context, that we see the importance in discussing the data collection and analysis  as applied in the ESE strategy for teaching empirical software engineering. field. We are of the opinion that ESE is relevant for both This paper is structured as follows. Section 2 practitioners and researchers. For practitioners it is introduces the aspects of software engineering about evaluating tools and techniques for use in education which have been relevant to our work and concrete cases . While it is equally important to some learning issues in research education. Section 3 teach ESE to each of these two groups., we report on a describes our course, its syllabus, pedagogical goals and the two different strategies. Section 4 is about the In this sense, their approaches can be classified as evaluation of the course. Discussion and conclusions addressing short-term requirement vs. addressing long- are given in section 5. term requirements. Guidelines for Software Engineering Education  adopt a middle ground 2 Background approach to education contents. They address the long- Software engineering, as a field, has, among others, term issues and are based on the body of knowledge two supporting disciplines–software engineering for software engineering . This body of knowledge, education and ESE. Software engineering education however, is based upon expert opinions within the focuses on training "software professionals" for the industry. industry , while ESE focuses on evaluating the Along the second axis, there are two strategies: the tools and techniques used, developed, and intended for first strategy is based on lectures and individual use in the industry through empirical validation. There learning. The second strategy is based on learning by is an element of training required in making the doing, also known as project-based learning. Both transition from trained software professional, with or Meyer  and  favor project-based learning. without working experience, to become an ESE Unlike the education content, they argue in favor of researcher. project-based learning to "prepare our students for the real challenges they will face" ( p.33). They also 2.1 Software Engineering Education argue that it is easier to learn from personal mistakes rather than mistakes related by a lecturer. The software engineering education literature moves along a dual axis; one axis for education 2.2 Learning issues in research education content and one for pedagogy. We use the two axes to reflect on the current state within software engineering Provided that it is possible to teach somebody how literature in this section. to become a good researcher, there are three kinds of The education content axis is delimited by two courses that can be offered as part of research extremes: industry driven and principles driven. Meyer education:  argues that the contents of software engineering 1. General courses on research methods at both education must be driven by the principles on which undergraduate and post graduate level are usually software engineering is based: offered by social science faculties. These courses "What matters is teaching [the students] address research issues such as scientific method fundamental modes of thought that will accompany and nature of evidence, advocacy versus evidence- them throughout their careers and help them grow in based approaches, writing and reviewing research this ever-changing field. The ones who blossom are proposals, how to use bibliographies and citation those who can rise beyond the tools of the moment in searches, project planning, selecting results and harmony with the progress of the discipline. ( places to publish, outlining and structuring p.29)" research papers, the peer review process, On the other end is the work of Lethbridge . presenting posters and papers at conferences, Based on a survey of 168 software engineers, he finds publishing in academic and engineering journals, significant differences between curricula taught at etc.. colleges and universities and the actual knowledge 2. Courses on research methods in computer science required in the industry. Lethbridge argues for aligning address some of the research issues above, such as existing curricula with skills required by the industry. scientific method and nature of evidence, Where Meyer  is specific on the need to distance customized to the IT field. At our department, for education from the industry’s immediate, short-term example, there is a common introductory course requirement, Lethbridge writes little of the long-term for all PhD students, which addresses general requirements that Meyer addresses with his principles. research issues in IT like those discussed for example in . Table 1 Bloom’s taxonomy of learning customized to ESE Level Definition Sample Sample behavior Sample behavior ESE verbs Knowledge Student recalls or Write The student will The students will be able to recognizes information, List define the 6 levels define the content of the different ideas, and principles Label of Bloom's papers. in the approximate Name taxonomy of the form in which they State cognitive domain. were learned. Define Comprehension Student translates, Explain The student will The students will explain the comprehends, or Summarize explain the purpose of the give methods and interprets information Paraphrase purpose of investigations. based on prior Describe Bloom's learning. Illustrate taxonomy of the cognitive domain. Application Student selects, trans- Use The student will The student will be able to use fers, and uses data Compute write an one method for experimentation. and principles to Solve instructional complete a problem Demonstrate objective for each or task with a mini- Apply level of Bloom's mum of direction. Construct taxonomy. Analysis Student distinguishes, Analyze The student will The students will compare and classifies, and relates Categorize compare and contrast different methods and the assumptions, Compare contrast the investigations. hypotheses, evidence, Contrast cognitive and or structure of a Separate affective domains. statement or question. Synthesis Student originates, Create The student will The students will design an integrates, and Design design a investigation by choosing and combines ideas into a Hypothesize classification perhaps combining different product, plan or Invent scheme for methods. proposal that is new Develop writing to him or her. educational objectives that combines the cognitive, affective, and psychomotor domains. Evaluation Student appraises, Judge The student will The students will judge the assesses, or critiques Recommend judge the effectiveness of using empirical on a basis of specific Critique effectiveness of software engineering methods. standards and criteria. Justify writing objectives using Bloom's taxonomy. 3.1 Common characteristics 3. Courses like the one we describe in this paper, which address empirical research methods in In the following section we discuss the common software engineering. These courses do not exist characteristics for the course. in isolation but in education and research context that may or may not include general research 3.1.1 Students courses or courses specific for IT research. As far as we know, there is at least one paper that reports The empirical software engineering course counts as on teaching ESE  as part of a software 7.5 European Credit Transfer System (ECTS). This is engineering course. Undergraduate students work equivalent to 12 hours work for 15 weeks, 2 hours in in projects, and the teachers play the role of the class and 10 hours of other learning activities, for a customer. In one project the customer is a total of 180 hours. The course was run once during the hypothetical company wanting the students to autumn of 2002 at the Norwegian University of perform empirical studies (mainly experiments) to Science and Technology (abbreviated NTNU) for PhD evaluate different alternative techniques, e.g., students by the software engineering group, and once different kinds of reviews. Students are not asked during the summer of 2003 at Simula Research to plan the studies but only to perform them, as Laboratory in Oslo for both University of Oslo and planning and running would take too much of the NTNU PhD students. course. The syllabus of the empirical software Some of the students worked in empirical software engineering part is . engineering research projects, while others worked in other kinds of software engineering projects. All 2.2 Software engineering education and students had a general course about research methods research education at IDI in IT as part of their curriculum. Some students even had a course on research methods in general. The The software engineering group at IDI has thirty students age, gender, nationality, and scientific years experience with project based software background differed. engineering education. One of the author of this paper, Jaccheri, has more than ten years experience with 3.1.2 Syllabus teaching project based software engineering, software quality and metrics issues to software engineering The course has had the same syllabus throughout. students, both in Italy and in Norway, as well as Here, we refer to a consolidated bulk of literature, reflecting and writing about this   . IDI has which is used as syllabus for the course. 7 years experience with an introductive course for IT The syllabus is divided into three parts: motivation, researchers, which is mandatory for all new PhD method, and actual investigations. students. Jaccheri has had the main responsibility for • Motivation: In  and in , motivations for this course for one year. IDI graduates twenty PhD the existence of the ESE field are given. These candidates each year and the software engineering papers provide also a classification of existing group has graduated a total of 19 software engineering software engineering research papers according to PhD students. the kind of empirical method used in the respective research. 3. Empirical software engineering course • Methods:  provides an introduction to the field, with special emphasis on experiments. , The course reported in this paper is offered to PhD , , , , , , and , provide students within software engineering. While the course concrete methods for performing, and analyzing has been run three times, we mainly report on the first investigations.  and  are about Data two times as the third time we ran the course as a Analysis Methods. replication of the second time. The first and the second • Actual studies: , , , and  are about times had some common characteristics and some concrete investigations. different characteristic. In the following section we The course ended with a final oral exam with the discuss the common characteristics and afterwards we teacher and an ESE expert outside of the university as introduce the characteristics that were different for the examiner. two. 3.2 Differing characteristics students to performance work and to stimulate Here we introduce the characteristics that differed. cooperation among students. f. Production of a five minutes movie advertising the 3.2.1 Pedagogical goals and teaching strategy for field of empirical software engineering. Autumn 2002 g. Choosing an actual study in the syllabus, characterize the investigation presented in the In the autumn 2002 iteration of the course, the goal chosen study according to motivations, method, was to make the students acquainted with the contents measurements, and data analysis method. The goal of the syllabus. The course was held in a classroom of this exercise was to simulate the planning and context where students were met two hours a week for execution of an empirical investigation. 13 weeks (totally 26 hours in class). h. Group work with the goal of obtaining a deep and At each meeting, one student presented one paper critical understanding of the issues presented in from the syllabus. The teachers responsible for the the syllabus, also in light of the experience course provided feedback and stimulated discussion acquired during the previous exercises. Each around the paper. group had to: choose one research hypothesis and its motivation (business, education, others); list 3.2.2 Pedagogical goals and teaching strategy for and comment investigation planning and risks Summer 2003 (what can go wrong); define (or reuse) guidelines for designing and running empirical In the summer 2003 version of our course, the investigations; choose one actual investigation. pedagogical goals were: i. Essay writing. Each student had to write an 1. given our syllabus which introduces a possible individual essay. The essay assignment was overview of empirical software engineering “Extract from  and from the other papers in knowledge, let our students know about the the Methods part of the syllabus, the most syllabus at a level with is as high as possible important guidelines for designing and running according to the Bloom´s taxonomy . empirical investigations. Characterize the 2. establish a Norwegian network of young investigation presented in the chosen actual study, researchers within the field of software according to these guidelines. This means that you engineering. have to comment about the motivations, the Bloom’s taxonomy is reported in Table 1. The last method, the measurements, and the data analysis column of the table (Sample behavior ESE) gives our method”. The essay was supposed to be handed in interpretation of the taxonomy when applied to the two weeks after the seminar and one week before ESE domain. the oral exam. Recalling that students have 180 The second iteration of the course was organized as a hours allocated to this course, we expected that three days event (21 hours). Here we list in students worked full time writing the essay during chronological order the main tasks of the course. these two weeks. a. Students were informed about the syllabus and the course web page . One month before course 4. Course evaluation start and were asked to read the entire syllabus before the course started. Based on running the course twice we wanted to b. Short introduction to the course content by the evaluate it in order to plan its third iteration. One goal teacher. of this evaluation was to get general suggestions for c. Introduction to the field of empirical software improvement. We wanted to reflect about the two ways engineering by discussing examples of interaction of teaching ESE–individual presentations or group with the Norwegian software industry by two assignments. Norwegian research managers. Educational evaluation is a sub field of educational d. Group work with the goal of extracting the main research for which there is an extensive bibliography issues from the syllabus. Each group had to find along with national and international standards . and list research hypotheses, data, and their To the best of our knowledge our course is one if the analysis. Moreover each group had to summarize few offered internationally within the field. The one method to plan and perform investigations, number of new software engineering PhD students in and to summarize one investigation. Norway is of the magnitude of ten. While there is a e. Practical exercises coordinated by a performance need to teach such a course and to reflect over its and theater instructor with the goal to introduce learning effects and the benefits and risk of different • In your own words, what were the primary pedagogical strategies, we are aware that time is not objectives of the Empirical software engineering mature for a formal evaluation of the course involving course? professional pedagogues and psychologists of the We did not ask directly what the students had education service at our university. learnt, but what they thought were the primary However, we have designed and run an evaluation objectives of the course. We did this to make sure the of the two first iterations of the course. Our evaluation students answered what they had learnt from the attempts to reflect about how much the students had course contents, not what the teachers were supposed learnt from attending the course. to teach them. To see how well the course succeeded We decided to implement our investigation as an e- in teaching the students the overall goal, to run mail-based questionnaire  that we circulated to all empirical evaluations, we asked the following: students attending the courses. • On the basis of what you have learned in this We decided to use Bloom’s taxonomy. The course, would you be able to plan and run an columns Level, Definition, Sample verbs, and Sample empirical investigation? behavior are taken directly from the Bloom taxonomy Finally, we formulated the final question: of educational objectives . Column “Sample • Do you have further comments on the course? behavior ESE” is our contribution. If we had asked the wrong questions, we hoped the We decomposed the level of learning reached by students would tell us so in answering this question. students with respect to the Bloom taxonomy in three We were also interested in feedback on how to better categories, one for each part of the syllabus: the course, and thought this question provided an motivation, method, and actual studies (see section opportunity for such feedback. 3.1.2 Syllabus). We wanted to know how much each Based on lists of the participants in the ESE course, student has learnt from each part of the syllabus. we circulated the questionnaire to all participants by e- The questionnaire had to be formulated in such a mail. We included an introductory letter explaining way that we could assess what the students have learnt that we wanted to use the data for both improving the and how well they have learnt. To this end we course and as material for a paper, that all responses formulated the two questions: would be treated confidentially, and a date within • When did you take the Empirical software which we would like to receive the responses by. The engineering course? (Autumn 2002 or Summer responses to our questionnaire were on free form and 2003?) are available at . • Have you ever participated in an Empirical We e-mailed the survey to the seventeen students software engineering investigation? If yes, prior or attending the two courses. In the end, due to data after attending the course? discard, we had ten responses to analyze. Our goal was to measure how much of the course The data from the questionnaire and our analysis contents the students had learnt from attending the can be found at . empirical software engineering course. We formulated • The “stories” written by students provide a the following question in order to measure how much valuable piece of knowledge for those evaluating of the course contents the students had learnt: our course. To the question “In your own words, what were the primary objectives of the Empirical software engineering course?". Table 2: Coding conventions. Word/phrase in free form question Translated meaning Coding key 1 Learn • Bloom’s taxonomy: knowledge level insight • Course topic: all overview aware Coding key 2 Run experiment • Bloom’s taxonomy: application level • Course topic: actual studies Coding key 3 Plan and run experiment • Bloom’s taxonomy: synthesis level • Course topic: actual studies Coding key 4 Tradition • Bloom’s taxonomy: depends on other coding keys • Course topic: motivation To the question “On the basis of what you have subjects (10) and the measurement scale (ordinal), it learned in this course, would you be able to plan and does not make sense to try statistical generalization. run an empirical investigation?” students write answers like: 5. Discussion and Conclusions • “Yes. But more concrete skills about how to design questionnaire and how to analyze data should The ESE course is now mandatory for all new be learned”. software engineering PhD students at NTNU. We had • “Not without more input, but I would have a only 4 new software engineering PhD students in better starting point for the planning phase. A lot of 2004. For this reason, the teacher (Jaccheri) decided to the work in the planning phase would consist of organize the 4 students as a group with which she deciding how the investigation should be formed, and I interacted as a performing member for the third think I have a better understanding about the basic iteration of the course. concepts and where I could find out more about The transition from 2003 to 2004 benefited from the them.” evaluation we report in this paper. The 2004 iteration To analyze our data, we employed the method of is organized in a similar way to 2003 and it is based on coding  for extracting qualitative data and map group activities. The most important change with them into Bloom’s taxonomy of learning. Based on the respect to 2003 is the introduction of a “summary responses and our interpretation of Bloom’s taxonomy, writing” activity that proceeds the seminar. This is an we therefore formulated a set of coding keys to ensure individual activity during which each student is identical coding of free form replies for both the supposed to write a structured summary of the main taxonomy and the course topics (Table 2). characteristics of the content part. Students get the • Coding key 1 enabled us to translate assignment one month before the seminar. In this way, sentences like “Provide some insight into typical the 2004 iteration is more oriented toward individual methods used in the field” into level knowledge for learning that the 2003 iteration but still much more topic method. group oriented than the 2002 one. The learning goal of • Coding key 2 enabled us to code sentences this summary activity is that students must have like “Yes - it would give me an important starting reached a knowledge level of the Bloom´s taxonomy point, but I would always need to confer and discuss before seminar starts. The next version (fourth) of the afterwards possible investigations and their plans with course will run during autumn 2005. people with experience” into level application for The main contribution of this work is the topic actual studies. description of a course in the field of empirical • Coding key 3 says that whenever a student software engineering. First, the presented syllabus  declares to have knowledge to plan an experiment, the can be used as a basis for a dialog in the ESE level for actual studies is synthesis since Create, community about which topics are of most importance Design, Hypothesize, Invent, and Develop are sample when educating the future researchers in the field. verbs for synthesis level in the Bloom taxonomy. Second, the two presented pedagogical strategies can Example is a sentence like: “Yes, I can design an be discussed further to find out which one or which empirical study now”. combination of the two is better suited for which • Coding key 4 says that whenever a student context. Another contribution is our customization of acknowledges that the goals of the course encompass Bloom´s taxonomy of learning to the ESE field. the traditions of the field, we assigned a value to the Like every other course, there are at least two axes: motivation field. Example: in response to the question one axis for education content and one for pedagogy. “In your own words, what were the primary objectives These two axes can be used also to reflect over this of the empirical software engineering course?” one course or similar ones. Which part of the content are student replied “To given an overview of the main we satisfied with and why? Which pedagogical methods and traditions of empirical software strategy work better in which situation? According to engineering”. our evaluation, students are generally satisfied with Along with the coding keys, we made extra rules syllabus and the way it is structured into motivation, for coding that can be found at  together with the methods, and actual studies. Students appreciate data set. examples provided by the papers in the investigation There is a slight difference between the levels part. Concerning the method part, there is need to obtained for autumn 2002 and summer 2003. focus more attention on case studies and interpretative However, taken in consideration the number of studies. Teachers and examiners are satisfied with the essays and we believe that the guidelines provided in 6. Acknowldge  are a good tool to characterize investigations Other ESE courses are offered at other universities, We thank all PhD students who participated to the courses like the one documented in . However, if one asks and who answered our questionnaire. We are in debt to colleagues about how they learnt to become Reidar Conradi and Monica Divitini who set up the first researchers in software engineering, or more specific version of the syllabus and organized the first version of the empirical software engineering or measurement, the course. We thank Tore Dybå for useful discussions and answer is often vague. Experienced researchers rely on suggestions on the syllabus part. Special thanks go to Dag Sjøberg and the Simula center group for giving us the tacit knowledge which is learnt and shared locally by possibilities to run the second and third version of our research groups and internationally by research course. communities. A contribution of this paper is that it can be an example for those who provide and want to provide similar courses. 7. References Concerning our evaluation, we are aware that the  A. Abran, J. W. Moore, P. Bourque, R. evaluation of the two pedagogical strategies has Dupuis, and L. L. Tripp, "Guide to the limitations if one regards it as a piece of educational Software Engineering Body of Knowledge, research evaluation. First the sample size (10) is too Trial Version SWEBOK, A Project of the limited. Second we ask students to evaluate their own Software Engineering Coordinating learning perception. Third, though we map the initial Committee," IEEE, May 2001. knowledge of each student, we do not take gender,  E. Arisholm, B. Anda, M. Jørgensen, and D. age, language, and country of origin into I. K. Sjøberg, "Guidelines on Conducting consideration. Students have different age, gender, Software Process Improvement Studies in nationality, and scientific background. While Industry," presented at 22nd IRIS heterogeneity enhances learning challenges and (Information Systems Research Seminar In possibilities, it poses serious problems to a formal Scandinavia) Conference, Keruu, Finland, evaluation of the learning effects of the course. 1999. However, the evaluation, in addition to content and  D. E. Avison, F. Lau, M. D. Myers, and P. A. pedagogy, are relevant to the ESE community and to Nielsen, " Action research," Communications the software engineering community. This because of the ACM, vol. 42, num. 1, pp. 94-97, 1999. despite several empirical studies being conducted with  D. J. Bagert, T. B. Hilburn, G. Hislop, M. students as subjects , there is a lack of frameworks Lutz, M. McCracken, and S. Mengel, for evaluating the learning effects of software "Guidelines for Software Engineering engineering courses. Our work is of interest for the Education Version 1.0 (1999)," CMU/SEI measurement community as it can serve as an example CMU/SEI-99-TR-032, 1999. for those who want to evaluate the pedagogical effects  J. Bell, Doing your research project. of courses where such experiments take place. Buckingham . Philadelphia: Open University One of the goals of our course is that of establishing Press, 230 pp., 2000. a Norwegian network of young researcher in the field  A. Birk, T. Dingsøyr, and T. Stålhane, of software engineering. From the students’ "Postmortem: Never Leave a Project without questionnaire answers and for living and working It," IEEE Software, vol. 19, num. 3, pp. 357- together with these students, we evaluate the project 390, 2000. based version of the course to be successful to respect  B. S. Bloom, Taxonomy of educational to this goal. objectives: The classification of educational There remain three open questions. First, that of goals: Handbook I, cognitive domain. New comparing our course with other similar courses York, Toronto: Longmans, Green. 1956. offered internationally. Secondly, one question which  L. Briand, E. Arisholm, S. Counsell, F. is not attacked by our work is: is it possible to teach Houdek, and P. Thevenod-Fosse, "Empirical PhD students how to cooperate with external actors? Studies of Object-Oriented Artifacts, Methods The final issue that we have not discussed is empirical and Processes: State of the Art and Future software engineering education for practitioners and Directions," Empirical Software Engineering, not only for researchers. num. 4, pp. 385-402, 1999.  L. C. Briand, S. Morasca, and V. R. Basili, "An operational process for goal-driven definition of measures," IEEE Transactions 10th ACM\IEEE-CS Conference on Software on Software Engineering, vol. 12, num. 4, pp. Engineering Education and Training, Virginia 1106- 1125, 2002. Beach.  J. Carver, L. Jaccheri, S. Morasca, and F.  J. Kirk and M. L. Miller, Reliability and Shull, "Issues in Using Students in Empirical Validity in Qualitative Research, vol. 1. Studies in Software Engineering Education," Newbury Park, California: SAGE presented at International Software Metrics Publications Inc., 1986. Symposium, Sydney, Australia., 2003.  B. Kitchenham, "DESMET: A Method for  R. Conradi and T. Dybå, "An Empirical Study Evaluating Software Engineering Methods on the Utility of Formal Routines to Transfer and Tools," Keele University, Technical Knowledge and Experience," presented at 8th Report TR96-09, August 1996. European software engineering conference  B. A. Kitchenham, S. L. Pfleeger, L. M. held jointly with 9th ACM SIGSOFT Pickard, P. W. Jones, D. C. Hoaglin, K. El international symposium on Foundations of Emam, and J. Rosenberg, " Preliminary software engineering, Vienna, Austria, 2001. guidelines for empirical research in software  P. J. Denning, "Computer Science: The engineering," IEEE Transactions on Software Discipline," Encyclopedia of Computer Engineering, vol. 28, num. 8, pp. 721 -734, Science, 2000. 2002.  T. Dybå, "An Instrument for Measuring the  O. Laitenberger and J.-M. DeBaud, Key Factors of Success in Software Process "Perspective-based Reading of Code Improvement," Empirical Software Documents at Robert Bosch GmbH," Engineering, Kluwer Academic Publishers, Information and Software Technology, vol. vol. 5, pp. 357-390, 2000. 31, num. 11, pp. 781-791, 1997.  N. Fenton, "Software Measurement: A  C. T. Lethbridge, "On the Relevance of Necessary Scientific Basis," IEEE software education: A survey and some Transactions on Software Engineering, vol. Recommendations," Annals of Software 20, num. 3, pp. 199-205, 1994. Engineering, vol. 6, pp. 91/110, 1998.  M. Höst, "Introducing Empirical Software  B. Meyer, "Software Engineering in the Engineering Methods in Education," Academy," IEEE Computer, vol. 34, num. 5, presented at Conference on Software pp. 28-35, 2001. Engineering Education and Training  Meredith D. Gall, Walter R. Borg, Joyce P. CSEE&T, Covington, Northern Kentucky, Gall, ISBN: 0-321-08189-7, Allyn & Bacon, USA., 2002. 2003, 656 pp  L. Jaccheri and T. Østerlie, "Empirical  J. Miller, J. Daly, M. Wood, M. Roper, and A. software engineering Course," Trondheim - Brooks, "Statistical Power and its Oslo, Norway, Subcomponents: Missing and Misunderstood http://www.idi.ntnu.no/emner/empse/, last Concepts in Empirical Software Engineering accessed February 2004. Research," Information and Software  L. Jaccheri, A software quality and software Technology, vol. 39, num. 4, pp. 285-295, process improvement course based on 1995. interaction with the local software software  M. Morisio, M. Ezran, and C. Tully, "Success industry, in the Tutorial section of Computer and Failure Factors in Software Reuse," IEEE Applications in Engineering Education Transactions on Software Engineering, vol. Journal, John Wiley and Sons, Inc. page 265- 28, num. 4, pp. 340-357, 2002. 272, Mar 2002.  C. B. Seaman, "Qualitative Methods in  L. Jaccheri and P. Lago, Metrics aspects of a Empirical Studies of Software Engineering," software engineering course project, IEEE Transactions on Software Engineering, September 1997, in Proc. of Inspire II, 2nd vol. 25, num. 4, pp. 557-572, 1999. International Conference on Software Process  D. I. K. Sjøberg, B. Anda, T. Dybå, M. Improvement - Research into Education & Jørgensen, A. Karahasanovic, E. F. Koren, Training, Sweden. and M. Vokac, "Conducting Realistic  L. Jaccheri and P. Lago, "Applying Software Experiments in Software Engineering," Process Modeling and Improvement in presented at First International Symposium on Academic Setting", April 1997, Proc. of the Empirical Software Engineering, Nara, Japan, 2002.  W. F. Tichy, "Should Computer Scientists Experiment More?," IEEE Computer, vol. 31, num. 5, pp. 32-40, 1998.  C. Wohlin, P. Runeson, M. Host, M. C. Ohlsson, B. Regnell, and A. Wesslen, Experimentation in Software Engineering: An Introduction: Kluwer Academic Publishers. 2000.  M. V. a. Zelkowitz and D. R. Wallace, " Experimental Models for Validating Technology," IEEE Computer, vol. 31, num. 5, pp. 23-31, 1998.
Pages to are hidden for
"10.1.1.131.2033"Please download to view full document