Proposal for Master Thesis in Software Engineering

Based on Thesis Proposal Template version 3.1 Proposal for Master Thesis in Software Engineering Base information Student 1 Name, email and P.Nr.: Jan Schulte, jasd08@student.bth.se,831214-P834 Student 2 Name, email and P.Nr.: Philip Preissing, philippreissing@googlemail.com, external (TU München, Germany) Title (preliminary): A Software V&V Management Framework for the Space Industry Link to project home page (if any): Academic Advisor: Dr. Robert Feldt (BTH), Dr. Sira Vegas (UPM) Industry Advisor: Annalena Johansson (RUAG Aerospace Sweden AB) Faculty Reviewer: Anonymous Start- and end-date: 2009/02/01 to 2009/08/01 Thesis type (research / industrial, theoretical / empirical): industrial, theoretical and empirical Student 1 suitability Software engineering course credits completed at BTH (total): 30 ECTS credit points Software engineering courses completed relevant for thesis work: Modelos y Metodos para la Evaluacion y Mejora de Procesos Software process improvement) Caso Practico de Calidad de Software (Software quality) Metricas de Software (Software metrics) Projekt Management Student 2 suitability Software engineering courses completed relevant for thesis work: Projektmanagement und Qualitätssicherung (Project management and quality assurrance) Projekt Management Advanced seminar course software quality Background Software for space applications has special requirements in terms of reliability and dependability. Therefore, existing software engineering methods are not sufficient. The European Cooperation for Space Standardization (ECSS) has developed a set of standards to cope with these issues. As any other industry, the space industry is faced with a pressure to deliver higher quality software fast and with less effort and thus cost. In a research project together with RUAG Aerospace Sweden AB and Swedish Space Corporation (SSC), BTH is trying to optimize Page 1 Based on Thesis Proposal Template version 3.1 Verification and Validation activities (VAs) wrt. effectiveness and efficiency. The companies feel that they find the same defects with multiple VAs and therefore see a chance to reduce this redundancy without negatively impacting quality. In the first part of the collaboration with RUAG and SSC, BTH has focused on analyzing the current situation of both companies. They examined the companies’ way of work and the challenges they face regarding verification and validation [8] [9]. A master thesis [10], conducted within this project, found three main problems: faults ‘slip’ through between development phases, inappropriate selection of VAs, and ECSS standards. Instead of focussing on a single verification and validation activity, research [1] [2] and industry [3] shows that combining different VAs can be more efficient in finding defects. To overcome the constant market pressure thats omnipresent in industry, further investigation is necessary to guide industry on how to select VAs to maximise the defect detection efficiency while minimising the effort spent. Some initial work that has been done towards the creation of a comparison framework is presented in the following. Wojcicki and Strooper present in [1] an iterative selection strategy (ISS) for verification- & validation- activities. The VAs are hereby firstly selected by their efficiency in order to maximise completeness, and secondly by the effort, i.e. the cost, they require. Based on the data collected while applying the technique, the selection is refined iteratively. In [5] the concept of Fault-Slip-Through (FST) is presented. In this approach, the faults found are categorized according to which phase they belong to, i.e. in which VA they should have been found. Based on these findings and the effort of the VA, the improvement potential for each activity is calculated. Wagner [3] proposes a more analytical approach. The defect detection techniques are hereby compared using an economical metric, namely the return on investment (ROI). The model also considers the effect of combining different defect detection techniques. A framework for the comparison of testing activities and formal verification is presented in [7]. However, this approach focuses more on studying the synergy and relationship of these two activities and doesn’t give any advice on how this combination can be optimized. There exists an IEEE Standard 1061 [11] related to software metrics and measurement. It defines a methodology for establishing quality requirements and finding, measuring and validating appropriate software product and process metrics. This can help in judging the effort and defect detection capability of VAs. This thesis targets the first two of the previously mentioned problems by combining measurements and a selection strategy for VAs in a coherent framework. Today, no such framework exists. It is neither known which measurements and VAs can be used for such a framework, nor how iterations can be used to gradually improve real-world software development processes. Because the framework is going to be used in an industry setting, it has to be as lightweight as possible and should only require a minimal set of changes to the existing processes. Aims and objectives The goal with this project is to create a coherent framework for the management and optimization of software verification and validation activities. Page 2 Based on Thesis Proposal Template version 3.1 Adapt FST and ISS to the development processes at RUAG Aerospace Sweden AB Creation of a framework that enables the comparison of the effectiveness of different VAs in eliminating different types of software defects and the comparison of the efficiency of different Vas. Validation of the framework at RUAG Aerospace Sweden AB Description of a minimal set of changes that RUAG needs to make to adapt to the framework A consideration of how the framework could be applied to the more agile development processes used at Swedish Space Corporation (SSC). Research questions 1. How can Fault-Slip-Through(FST) and Iterative Selection Strategy(ISS) be combined into a coherent framework? 2. Which alternatives exist to the aforementioned models? 3. Which measurements need to be done to support this framework? 4. How can the framework support parameterization to include existing measures and thus, minimize the measuring effort to be real-world applicable? 5. How can the framework be adapted to a real-world industry setting by only doing minimal changes to the company’s processes? 6. How does the framework perform in a real-world industry setting? 7. How can the framework be adapted to a more agile process? Expected outcomes • • • • • A description of the framework A process guideline for RUAG describing how the framework can be applied to their processes. Validation results gained by interviews and, if possible, by the data measured in a real project A list of recommendations for improving the RUAG development process based on the application of the framework A description on the suitability of applying the framework in a more agile environment Research Methodology First, the expectations and goals of RUAG are collected based on interviews. This provides hints on the important aspects to address in the framework. A literature survey is carried out to reveal the current state-of-the-art including which alternative models exist for the selection of the VAs and the measurement of these. The process documents of RUAG will be analysed to gain an in-depth knowledge of their software development processes. This includes roles, development phases, VAs used in each phase and metrics collected. Furthermore, the existing data and metrics collected for the different VAs will be investigated and evaluated with respect to their suitability for usage in the framework. Since for FST the test strategy has to be known in advance, it is extracted out of RUAG documents and data collected; and then further refined through interviews with RUAG responsibles. Page 3 Based on Thesis Proposal Template version 3.1 To allow a lightweight application in real industry settings, the framework is parameterized with different measurement levels, e.g. reaching from fully automated to manual data entry. These levels will be investigated under consideration of previous work done at Ericsson. Additionally, the results of the master thesis [8] serve as input for providing preliminary data of the efficiency and effectiveness of the VAs in RUAG. Based on these inputs (the master thesis, the state-of-the-art and alternative models, RUAG processes, the available metrics and data, the expectations and goals of RUAG, the test strategy and the measurement levels), a coherent framework will be developed. It will be documented in a written framework description. It is then determined whether the existing data is sufficient to support the application of the framework. According to the measurement levels, a minimum set of changes to the RUAG processes is developed. Those boil down to a process guideline describing the possible application of the framework at RUAG. The framework will be analysed statically by reviews or if possible through to the application of the framework in a real project and analysing the data obtained after a certain period of time. Based on the feedback gained, the framework will be revised iteratively. Based on the framework and the evaluation a guideline is established giving improvement recommendations to RUAG. Those are evaluated through interviews with RUAG responsibles. In cooperation with SSC it is evaluated how the framework can be adapted to a more agile process using interviews or workshops. A description concerning the suitability is developed including possible necessary adaptations. Risks Threat Insufficient data or insufficient support from RUAG to measure meaningful/significant data Dynamic validation of the framework not possible due to lack of available projects at that time Lack of time Possible solution Data gained from interviews with RUAG in the Master thesis [Towards Optimizing Verification and Validation Activities in Space Industry] will serve as input A static evaluation through reviews will be conducted Probability Medium/ High High Prioritization of the research questions, e.g. by Low limiting the effort spent on less important tasks (in discussion with supervisors and RUAG) Adaptability of the Being focussed and effective in the Medium/ framework depend on the interactions with RUAG High resources available at RUAG Time plan Scheduled Milestones and Meetings: 20090126: Start writing the proposal 20090129: First draft of proposal to supervisor Page 4 Based on Thesis Proposal Template version 3.1 20090218: Final draft of proposal to supervisor 20090222: End of literature review 20090216: Start collecting expectations and goals of RUAG 20090222: End collecting 20090216: Start of analysis of the software development processes at RUAG 20090315: End of analysis 20090216: Start data analysis 20090315: End data analysis 20090316: Start definition of test strategy and measurement classification 20090329: End of definition and classification 20090330: Start working on the framework 20090412: Finish framework and documentation so that it is ready to be used in a project 20090413: Start evaluation of the framework 20090517: Evaluation finished 20090518: Start working on improvement proposals 20090511: Preliminary results available for presentation at DASIA 20090531: Improvement proposals finished 20090601: Start of the evaluation of the framework at SSC 20090614: Evaluation of the framework at SSC finished 20090615: Evaluation of the recommended improvements 20090628: Evaluation finished 20090705: Supervisor tells examiner we are ok for presentation 20090722: Updated final draft sent to opponents 20090804: Thesis Presentation 20090815: Final thesis updated, approved and sent to examiner References [1] R. Feldt, E. Ahmad, B. Raza, E. Hult, and T. Nordebäck, "Evolving the ECSS standards and their Use: Experience based on Industrial Case Studies," Data Systems in Aerospace, 2009, in submission. [2] B. Raza, E. Ahmad, R. Feldt, and T. Nordebäck, "ECSS Standard Compliant Agile Development for Dependable Space Software - an Industrial Case Study," 2008, in submission. [3] E. Ahmad and B. Raza, Towards Optimizing Verification and Validation Activities in Space Industry. Ronneby: Blekinge Tekniska Högskola, 2009, Master thesis. [4] B. Littlewood, P. T. Popov, L. Strigini, and N. Shryane, "Modeling the effects of combining diverse software fault detection techniques," IEEE Transactions on Software Engineering, vol. 26, no. 12, pp. 1157-1167, 2000. [5] B. Kitchenham and S. Linkman, "Validation, verification, and testing: diversity rules," IEEE Software, vol. 15, no. 4, pp. 46-49, 1998. [6] N. Kikuchi and T. Kikuno, "Improving the testing process by program static analysis," in Software Engineering Conference, 2001. APSEC 2001. Eighth Asia-Pacific, 2001, pp. 195-201. [7] M. A. Wojcicki and P. Strooper, "An Iterative Empirical Strategy for the Systematic Selection of a Combinationof Verification and Validation Technologies," in Fifth Page 5 Based on Thesis Proposal Template version 3.1 International Workshop on Software Quality, 2007. [8] L.-O. Damm, Early and Cost-Effective Software Fault Detection. Ronneby: Blekinge Tekniska Högskola, 2007, Ph.D. thesis. [9] S. Wagner and T. Seifert, "Software quality economics for defect-detection techniques using failure prediction," in 3-WoSQ: Proceedings of the third workshop on Software quality, New York, NY, USA, 2005, pp. 1-6. [10] J. S. Bradbury, J. R. Cordy, and J. Dingel, "An empirical framework for comparing effectiveness of testing and property-based formal analysis," SIGSOFT Softw. Eng. Notes, vol. 31, no. 1, pp. 2-5, 2006. [11] IEEE, IEEE Std 1061-1998: Standard for a Software Quality Metrics Methodology. 1998. Page 6 Based on Thesis Proposal Template version 3.1 Appendix A – Research Methodology Literature survey RUAG process descriptions Interviews Data Ericsson Interviews Master thesis Consideration of alternatives Process analysis Data analysis RQ2 Collect expectations and goals of RUAG Develop test strategy Measurement classification RQ4 Framework development Framework description RQ1&3 Develop miminum set of changes Process guideline RQ5 Interviews Evaluation Validation results RQ6 Corrections necessary? Project Yes No Improvement recommendati ons List of recommendations Evaluation of improvement recommendati ons Consider applicability to agile processes at SSC Agile description RQ7 Page 7 Based on Thesis Proposal Template version 3.1 Appendix B – Time plan Page 8 Based on Thesis Proposal Template version 3.1 Page 9 Based on Thesis Proposal Template version 3.1 Page 10 Based on Thesis Proposal Template version 3.1 Page 11 Based on Thesis Proposal Template version 3.1 Page 12