ADAPTATION OF ATAM TO SOFTWARE ARCHITECTURE PRACTICES OF SMALL COMPANIES

ADAPTATION OF ARCHITECTURE TRADEOFF ANALYSIS METHODSM TO SOFTWARE ARCHITECTURE PRACTICES OF SMALL & MEDIUM SIZED SOFTWARE COMPANIES by S.M. Saiful Islam A Project Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in Software Engineering INDEPENDENT UNIVERSITY, BANGLADESH May 2009 APPROVAL ADAPTATION OF ARCHITECTURE TRADEOFF ANALYSIS METHODSM TO SOFTWARE ARCHITECTURE PRACTICES OF SMALL & MEDIUM SIZED SOFTWARE COMPANIES by S.M. Saiful Islam ID: 0712004 has been approved May 2009   APPROVED: Dr. Md. Rokonuzzaman……………………………………………………. ,Chairperson Prof. M. Abdus Sobhan……………………………………………………. Dr. Ali Shihab Sabbir……………………………………………………… ,Member ,Member Supervisory Committee     ACCEPTED:     ____________________________________________ Director, School of Engineering & Computer Science ABSTRACT The building block of any kind of software system is its architecture. The architecture determines the degree of success of a system. It is thus very important to find appropriate means of validating any software architecture to ensure the success of the overall system. Evaluation is a critical analytical process in all disciplines and fields and therefore also in software engineering. This report attempts to understand the practices of Architecture Tradeoff Analysis MethodSM (ATAM), developed by the Software Engineering Institute (SEI), as a means of architecture validation of local software companies and propose a modified ATAM which best fits for their needs. For conducting this study, data for five typical software companies were collected by applying market research approach from respective software architects. The local software companies are following different procedures and their own standards for representing and evaluating their softwares’ architecture. But usually they don’t follow any global standards to represent and evaluate their architecture or can tradeoff their architectural decisions. This report contains the information about the ATAM practice level of small and medium sized software companies of Bangladesh. A gap analysis has been conducted between the industry practices and theoretical knowledge of ATAM. Based on the findings of this investigation an adapted ATAM has been proposed describing the relevance of the ATAM framework to the local industry along with relevant recommendations, which is likely to be more useful and understandable by the small and medium sized software companies. The results of the report facilitate the identification of each evaluation component and stress some key aspects, such as the relevant role of stakeholders and the significance of attribute-based architectural iii styles in an ATAM evaluation. This report helps small and medium size software companies to practice the ATAM for the evaluation and validation of their architectures and improve their delivery capacity in terms of quality, time and cost. iv ACKNOWLEDGMENTS Apart from the efforts of me, the success of this project depends largely on the encouragement and guidelines of many others. I take this opportunity to express my gratitude to the people who have been involved in the successful completion of this project. First of all, I would like to show my greatest appreciation to my honorable supervisor Dr. Md. Rokonuzzaman for his magnificent support and help, who has meticulously reviewed this paper and helped me to bring it in its final shape. I feel motivated and encouraged every time I attend his meeting. Without his encouragement and guidance this project would not have materialized. I would like to express my deep appreciation to my teammate Mr. Mohammad Forhad Hossain and many of my classmates including Mr. Aref Samad, Mr. Abdur Rashid, Mr. Ariful Haque Bhuiya, Mr. Kiriti Prasad Chowdhury and Mr. Siddharto Kirtonia for allowing me to share their knowledge and ideas proactively that was very vital for the success of this project. I am very grateful for their continuous support and help. I would like to thank Mr. Ali Ishtiaq, CEO, PyxisNet Ltd., for allowing me to enroll in this program and for sharing his knowledge, and providing his continuous support and encouragement to make this study effective. In this connection, I would also like to thank many of my colleagues of PyxisNet Ltd. who have extended their hands to make my work a success. v I would like to thank my ex-colleagues and respected people from different software companies for sparing their time and providing me Architecture practice related information during this project period. Without their cooperation and continuous support this project would not have been possible. Finally, I am very grateful to all of my family members, specially my wife and little son, who have been deprived of my association for long time in the process of preparing this report. vi TABLE OF CONTENTS Page  LIST OF TABLES .....................................................................................................................  i  x LIST OF FIGURES ................................................................................................................. xiv  CHAPTER ONE ........................................................................................................................ 1  INTRODUCTION ..................................................................................................... 1  1.1  Background ..................................................................................................... 1  1.2  Objectives ........................................................................................................ 5  CHAPTER TWO ....................................................................................................................... 7  PROBLEM STATEMENT ........................................................................................ 7  2.1 Introduction .......................................................................................................... 7  2.2 Characteristics of the Local Practices of Architecture ......................................... 7  2.2 Typical challenges of the Architectural practices ................................................ 8  2.3 Role of ATAM for improving Architectural Practices ........................................ 9  CHAPTER THREE ................................................................................................................. 10  METHODOLOGY .................................................................................................. 10  3.1 Literature Review and Defining Scope of Research .......................................... 10  3.2 Key Informant Interview.................................................................................... 10  3.2.1 Interview Questions .................................................................................... 11  3.2.2 Selecting Key Informants ........................................................................... 11  3.2.3 Interview Technique ................................................................................... 12  3.2.4 Recording Interview Responses .................................................................. 12  3.2.5 Organizing Key Informant Interview Information ..................................... 12  3.3 Questionnaire Survey ......................................................................................... 13  3.3.1 High Level Questionnaires ......................................................................... 14  3.3.3 Detailed Level Questionnaires .................................................................... 20  3.3.4 Quantification of Feedback ......................................................................... 31  3.4 Limitations ......................................................................................................... 31  CHAPTER FOUR ................................................................................................................... 32  LITERATURE REVIEW ........................................................................................ 32  4.1 Introduction ........................................................................................................ 32  4.2 Background ........................................................................................................ 34  4.3 History................................................................................................................ 37  4.4 Architectural Views ........................................................................................... 39  4.5 Architectural Styles ............................................................................................ 40  4.5.1 Blackboard .................................................................................................. 40  4.5.2 Client-server ................................................................................................ 40  4.5.3 Database-centric architecture ...................................................................... 41  4.5.4 Distributed Computing................................................................................ 42  4.5.5 Event Driven Architecture .......................................................................... 43  4.5.6 Service-oriented Architecture ..................................................................... 44  4.5.7 Implicit Invocation ...................................................................................... 45  4.5.8 Monolithic Application ............................................................................... 45  vii 4.5.9 Peer-to-peer ................................................................................................. 46  4.5.10 Pipes and Filters ........................................................................................ 47  4.5.11 Plugin ........................................................................................................ 47  4.6 Architecture Evaluation Methods ...................................................................... 48  4.7 Architecture Tradeoff Analysis MethodSM ........................................................ 50  4.7.1 Trade off Analysis....................................................................................... 50  4.7.2 Description of ATAM ................................................................................. 51  4.7.2.1 Presentation ...................................................................................................... 53  4.7.2.2 Investigation and Analysis ............................................................................... 53  4.7.2.3 Testing.............................................................................................................. 54  4.7.2.4 Reporting ......................................................................................................... 55  . CHAPTER FIVE ..................................................................................................................... 56  DATA ANALYSES AND RESULTS..................................................................... 56  5.1 Introduction ........................................................................................................ 56  5.2 ATAM Step by Step Data Analysis ................................................................... 57  5.2.1 Phase 1 - Presentation ................................................................................. 57  Step 1: Present the ATAM .......................................................................................... 57  Step 2: Present the Business Driver ............................................................................ 59  Step 3: Present the Architecture to be evaluated ......................................................... 62  5.2.2 Phase 2 - Investigation and Analysis .......................................................... 66  Step 4: Identify the Architectural Approaches ............................................................ 66  Step 5: Generate the Quality Attribute Utility Tree. ................................................... 69  Step 6: Analyze the Architectural Approaches ........................................................... 74  Step 6.1: Investigation of Architectural Approach ..................................................... 76  Step 6.2: Creation of Analysis Questions ................................................................... 80  Step 6.3: Answers to the Analysis Questions ............................................................. 85  Step 6.4: Find the risks, non-risks, sensitivity points, trade-off points. ...................... 87  5.2.3 Phase 3 – Testing ........................................................................................ 90  Step 7: Brainstorm and Prioritize Scenarios ............................................................... 90  Step 8: Analyze the Architectural Approaches ........................................................... 94  5.2.4 Phase 4 – Report the ATAM....................................................................... 97  Step 9: Report the ATAM ........................................................................................... 97  5.3 Analysis Result ................................................................................................ 101  5.3.1 Combined Practice Level of the Companies ............................................. 101  5.3.2 Combined Analysis Result ........................................................................ 102  viii CHAPTER SIX ...................................................................................................................... 109  ATAM ADAPTATION ......................................................................................... 109  6.1 Introduction ...................................................................................................... 109  6.2 Adapted ATAM ............................................................................................... 109  6.2.1 Phase 1 - Presentation ............................................................................... 109  6.2.1.1 Present the ATAM ......................................................................................... 110  6.2.1.2 Present the Business Drivers .......................................................................... 111  6.2.1.3 Present the Architecture to be evaluated ........................................................ 112  6.2.2 Phase 2 - Investigations and Analysis....................................................... 113  6.2.2.1 Identify the Architectural Approach .............................................................. 114  6.2.2.2 Generate the Quality Attribute Utility Tree ................................................... 114  6.2.2.3 Analyze the Architectural Approach  ............................................................. 115  . 6.2.2.4 Investigation of Architectural Approach  ....................................................... 116  . 6.2.2.5 Creation of Analysis Questions  ..................................................................... 117  . 6.2.2.6 Answers to the Analysis Questions ................................................................ 117  6.2.2.7 Find the Risk, Non-Risks, Sensitivity and Tradeoff Points ........................... 118  6.2.3 Phase 3 - Testing ....................................................................................... 119  6.2.3.1 Brainstorm and Prioritize Scenarios .............................................................. 119  6.2.3.1 Analyze the Architectural Approaches .......................................................... 120  6.2.4 Phase 4 - Report the ATAM ..................................................................... 122  6.2.4.1 Report the ATAM .......................................................................................... 122  6.3 Work Items of Adapted ATAM ....................................................................... 123  CHAPTER SEVEN ............................................................................................................... 125  SUMMARY AND PROPOSITION FOR FUTURE WORK ................................ 125  7.1 Progress Made .................................................................................................. 125  7.2 Limitation of the Progress ................................................................................ 126  7.3 Recommendation for Future Work .................................................................. 127  REFERENCES ...................................................................................................................... 128  ACRONYMS  ........................................................................................................................ 131  . APPENDIX ........................................................................................................................... 132  A B C D E High Level Feedback of Company - A ............................................................. 132  High Level Feedback of Company - B.............................................................. 138  High Level Feedback of Company - C.............................................................. 143  High Level Feedback of Company - D ............................................................. 149  High Level Feedback of Company - E .............................................................. 154  ix F Detail Level Feedback of Company - A ............................................................ 159  G Detail Level Feedback of Company - B............................................................ 169  H Detail Level Feedback of Company - C............................................................ 182  I Detail Level Feedback of Company - D ............................................................. 194  J Detail Level Feedback of Company - E ............................................................. 210  x LIST OF TABLES Table Page Table 1 : High Level Feedback on Step 1 .................................................................................. 57  Table 2 : Detail Level Feedback on Step 1  ............................................................................... 57  . Table 3 : Quantitative Status of ATAM Step 1(High Level) ...................................................... 58  Table 4 : Quantitative Status of ATAM Step 1(Detail Level) .................................................... 58  Table 5 : High Level Feedback on Step 2 .................................................................................. 59  Table 6 : Detail Level Feedback on Step 2  ............................................................................... 60  . Table 7 : Quantitative Status of ATAM Step 2 (High Level) ..................................................... 61  Table 8 : Quantitative Status of ATAM Step 2 (Detail Level) ................................................... 61  Table 9: High Level Feedback on Step 3  .................................................................................. 62  . Table 10 : Detail Level Feedback on Step 3.............................................................................. 63  Table 11 : High Level Quantitative Feedback on Step 3 .......................................................... 65  Table 12 : Detail Level Quantitative Feedback on Step 3 ........................................................ 65  Table 13 : High Level feedback on Step 4 ................................................................................ 66  Table 14 : Detail feedback on Step 4 ....................................................................................... 67  Table 15 : High Level Quantitative feedback on Step 4 ........................................................... 67  Table 16 : Detail Quantitative feedback on Step 4 .................................................................. 68  Table 17 : High Level feedback on Step 5 ................................................................................ 69  Table 18 : Detail feedback on Step 5 ....................................................................................... 70  Table 19 : High Level Quantitative feedback on Step 5 ........................................................... 72  Table 20 : Detail Quantitative feedback on Step 5 .................................................................. 73  Table 21 : High Level feedback on Step 6 ................................................................................ 74  Table 22 : Detail feedback on Step 6 ....................................................................................... 74  Table 23 : High Level Quantitative feedback on Step 6 ........................................................... 75  Table 24 : Detail Quantitative feedback on Step 6 .................................................................. 75  Table 25 : High Level feedback on Step 6.1 ............................................................................. 76  xi Table 26 : Detail feedback on Step 6.1 .................................................................................... 77  Table 27 : High Level feedback on Step 6.1 ............................................................................. 78  Table 28 : Detail feedback on Step 6.1 .................................................................................... 79  Table 29 : High Level feedback on Step 6.2 ............................................................................. 80  Table 30 : Detail feedback on Step 6.2 .................................................................................... 81  Table 31 : High Level Quantitative feedback on Step 6.2 ........................................................ 83  Table 32 : Detail Quantitative feedback on Step 6.2 ............................................................... 83  Table 33: High Level feedback on Step 6.3 .............................................................................. 85  Table 34 : Detail feedback on Step 6.3 .................................................................................... 85  Table 35 : High Level Quantitative feedback on Step 6.3 ........................................................ 86  Table 36 : Detail Quantitative feedback on Step 6.3 ............................................................... 86  Table 37 : High Level feedback on Step 6.4 ............................................................................. 87  Table 38 : Detail feedback on Step 6.4 .................................................................................... 87  Table 39 : High Level Quantitative feedback on Step 6.4 ........................................................ 89  Table 40 : Detail Quantitative feedback on Step 6.4 ............................................................... 89  Table 41 : High Level feedback on Step 7 ................................................................................ 90  Table 42 : Detail feedback on Step 7 ....................................................................................... 91  Table 43 : High Level Quantitative feedback on Step 7 ........................................................... 92  Table 44 : Detail Quantitative feedback on Step 7 .................................................................. 93  Table 45 : High Level feedback on Step 8 ................................................................................ 94  Table 46 : Detail feedback on Step 8 ....................................................................................... 95  Table 47 : High Level Quantitative feedback on Step 8 ........................................................... 96  Table 48 : Detail Quantitative feedback on Step 8 .................................................................. 96  Table 49 : High Level feedback on Step 9 ................................................................................ 97  Table 50 : Detail feedback on Step 9 ....................................................................................... 98  Table 51 : High Level Quantitative feedback on Step 9 ........................................................... 99  Table 52 : Detail Quantitative feedback on Step 9 ................................................................ 100  xii Table 53 : High Level Combined Status of ATAM Practice Level of 5 Companies ................. 101  Table 54 : Combined detailed level status of ATAM practice of 5 companies ...................... 102  Table 55 : Combined Status of ATAM Practice Level (Points earned) ................................... 104  Table 56 : Comparative Status of ATAM Practice Level (% of available points) .................... 105  xiii LIST OF FIGURES Figure Page Figure 1 : A simple pictorial showing of the main steps and sub‐steps of ATAM .................... 52  Figure 2 : Practice Level Status of ATAM Step 1 "Present the ATAM"..................................... 59  Figure 3 : Practice Level Status of ATAM step 2 “Present the business driver” ...................... 62  Figure 4 : Practice Level Status of ATAM step 3 "Present the Architecture to be evaluated" 66  Figure 5 : Practice Level Status of ATAM Step 4 "Identify the architectural approaches" ...... 69  Figure 6 : Practice Level Status of ATAM Step 5 "Generate Quality Attributes" ..................... 74  Figure 7 : Practice Level Status of ATAM Step 6 "Analyze the architectural approaches" ...... 76  Figure 8 : Practice Level Status of ATAM Step 6.1 “Investigation on architectural approach”  0  8 Figure 9 : Practice Level Status of ATAM Step 6.2 "Creation of analysis question" ................ 85  Figure 10 : Practice Level Status of ATAM Step 6.3 “Answer to the analysis questions” ........ 87  Figure 11 : Practice Level Status of ATAM Step 6.4 "Find the risks, non‐risks, sensitivity  points, tradeoff points"  ........................................................................................................... 90  . Figure 12 : Practice Level Status of ATAM Step “Brainstorm and prioritize scenarios” .......... 94  Figure 13 : Practice Level Status of ATAM Step 8 "Analyze the architectural approaches" .... 97  Figure 14 : Practice Level Status of ATAM Step 9 "Report the ATAM" .................................. 101  Figure 15 : Combined ATAM Practice Level Status of Company A ........................................ 106  Figure 16 : Combined ATAM Practice Level Status of Company B ........................................ 106  Figure 17 : Combined ATAM Practice Level Status of Company C ........................................ 107  Figure 18 : Combined ATAM Practice Level Status of Company D ........................................ 107  Figure 19 : Combined ATAM Practice Level Status of Company E ......................................... 108  Figure 20 : Average Practice Level Status of the Companies ................................................. 108  xiv CHAPTER ONE INTRODUCTION 1.1 Background The architecture of a system refers to the components of the system, their organization and the interactions between these components. It provides a meaningful depiction of the system. “Architectures allow or preclude nearly all of the system’s quality attributes” [1]. Examples of such quality attributes include modifiability, performance, reliability, and more. The above mentioned definition leads us to mention this truth about software architecture evaluation: “If architectural decisions determine a system’s quality attributes, then it is possible to evaluate architectural decisions with respect to their impact on those attributes” [1]. The quality of a software-intensive system depends heavily on the system’s software architecture. When used appropriately, software architecture evaluations can have a favorable effect on a delivered system. Software analysis and evaluation has become a well-established practice inside the architecting community of the software systems. The development effort, the time and costs of complex systems are considerably high. In order to assess system’s quality against the requirements of its customers, the architects and the developers need methods and tools to support them during the evaluation process [17]. There are many software architecture techniques being used nowadays, but this project focuses on the Architecture Tradeoff Analysis MethodSM (ATAM) only. The objective of this project is to determine the practice level of this evaluation framework in local software companies and proposing the locally adapted ATAM. 2 The software architecture of a program or computing system is the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationships between them. The term also refers to documentation of a system's software architecture. Documenting software architecture facilitates communication between stakeholders, documents early decisions about high-level design, and allows reuse of design components and patterns between projects. Software architecture provides an overview of the composition and functionality of the given software system. Just like a structural architect, a software architect needs to analyze the requirements, determine components that should be used to build the system, and support the project by guiding and solving problems all along the execution cycle. Understanding of the domain provides the capability to analyze the requirements and envisioning a solution. Exposure to various tools and technologies imparts the ability to choose among the various solutions available, and anticipate and solve technical problems. The right software architecture is essential for a software-intensive system to meet its functional requirements as well as its quality requirements that govern real-time performance, reliability, maintainability, and a host of other quality attributes. Conducting an architecture evaluation to determine the software architecture’s fitness is one of the most powerful, technical risk mitigation strategies. Bangladesh is a new country for the IT business. Several software companies are doing the IT business in local market as well as in global market. Some of them work on product oriented and some of them work on service oriented fashion. Also Bangladesh Govt. automating their work processes in several sectors. Medium to 3 large scaled software products are being developed in our country. Some of them are being used in government sectors and some of them are being developed for offshore clients. In the outsourcing market segment Bangladesh has a great opportunity to make business. Lots of new and old software companies are doing the outsourcing business. All of them are designing software architecture for different types of software project. But few of them follow any standard process to trade-off their architectures, which helps everyone to understand the strength and weakness of their architectural designs. These are also creating problems for the companies to communicate with the clients and clients are not confident enough to make contract with these companies. As software architecture is a key business asset for an organization, architectural analysis must also be a key practice for that organization. Architectures are complex and involve many design tradeoffs. Without undertaking a formal analysis process, the organization cannot ensure that the architectural decisions made—particularly those which affect the achievement of quality attribute such as performance, availability, security, and modifiability— are advisable ones that appropriately mitigate risks. For evaluating the architecture, Software Engineering Institute of Carnegie Mellon University (SEI) has developed a framework named Architecture Tradeoff Analysis MethodSM (ATAM). It is evident that Architecture Tradeoff Analysis MethodSM (ATAM) helps everyone to overcome the problem of tradingoff their architecture. ATAM is a risk-mitigation process used early in the software development life cycle [1]. ATAM is most beneficial when done early in the software development life-cycle, when the cost of 4 changing architectures is minimal [1]. Proven benefit of ATAM is clarified quality attribute requirements, improved architecture documentation, documented basis for architectural decisions, identified risks early in the life-cycle and increased communication among stakeholders [3]. Also this helps us to identify the strength and weakness of architecture. Because of these benefits, ATAM has been considered as the risk mitigation process which helps to improve performance and decision making for designing architecture. The ATAM process consists of gathering stakeholders together to analyze business drivers and from these drivers extract quality attributes that are used to create scenarios. These scenarios are then used in conjunction with architectural approaches and architectural decisions to create an analysis of trade-offs, sensitivity points, and risks (or non-risks). This analysis can be converted to risk themes and their impacts whereupon the process can be repeated [1]. This helps in developing any sort of software project in a sound manner for the small and medium companies in this industry. However, the generic process of using ATAM might cause unacceptable and unnecessary work for designing the trade-off mechanism. A good amount of literatures and resources are available on providing the guideline for developing well structured ATAM. But these guidelines or ATAM applying process need to be tailored according to existing practices. Moreover, available literatures do not provide indication on how to adapt ATAM considering the existing practices. Many segments of ATAM are applied but companies don’t know how to reorganize them to fit with the ATAM. An ATAM adaptation process considering the existing practices is 5 required to practice the ATAM in a cost effective manner which will be able to serve its intended purpose. The research will be identifying typical nature of practice level of architecture tradeoff process in small and medium software companies in Bangladesh. Challenges in defining software architecture in a structured manner and adaptation of ATAM will be identified. Based on such findings, a recommended ATAM adaptation process for their best benefit will be proposed considering the existing practice level. 1.2 Objectives The research objectives are: 1. To identify the architecture knowledge level of practitioners and nature of architecture design process in practice in small and medium sized software companies of Bangladesh. 2. To identify the typical nature of practice level of architecture evaluation and architecture tradeoff process in small and medium size software companies of Bangladesh. 3. To conduct high level and detail level investigation on the architecture evaluation process in practice following the ATAM framework in small and medium size software companies of Bangladesh. 4. To analyze the gap of practice with best practices and standards along with their implications to the local software companies and all the related stakeholders, if any. 5. To identify the relevance and scope of practices of ATAM as technical risk mitigation strategy in small and medium size companies of Bangladesh. 6 6. To propose an adapted ATAM fusing the knowledge and concern of local practitioners, global standards and best practices, instead of full-blown ATAM that best fits the capacity, needs, and standards of the small and medium size software companies in Bangladesh. CHAPTER TWO PROBLEM STATEMENT 2.1 Introduction To achieve the major objectives (stated in chapter 1) of this research work – studying the nature of the existing architectural designing process practices and identifying the gap between the academic knowledge and industry practices. The software architects of several well know Bangladeshi Software firms were considered as key informant and interviewed for proper analysis with efficiency. Architectural tradeoff analysis practices for different companies identified at high level and detail level from the key informant interview. In addition implications of not practicing ATAM are identified. Work items for practicing ATAM are also identified. Finally, how ATAM can be practiced corresponding to existing practice are identified, which is the core part of the research. 2.2 Characteristics of the Local Practices of Architecture Most of the local software companies only provide small and standard size solution and where they do not concern about architectural design problems and solutions. They usually follow the contemporary and commonly available solutions blindly and do not take proper concentration to the business scenario for which the application is likely to be built. In most of the cases very little attention is given to the architectural design. Therefore, most of the time these software companies suffer in the later phases of the software development life cycle. The problem turned into very acute in the maintenance phase in project life cycle and leads to heavy rework. Therefore, 8 restructuring of the project prevents its ability to accommodate with the new functionality or changes to the current functionality. Sometimes it costs much higher then reengineers the entire project, and in case of large scale projects it goes worse. The importance, implications and knowledge about the architecture and its evaluation process are very limited. In most of the cases, the architecture is defined by the senior person of the project following the readily available architectures. But usually there is no awareness or feelings about the need of the right architecture design, evaluation and validation. The involvement of the different stakeholders for providing their inputs in design and evaluation process is not very common practice. In turn the quality of the project is seriously hampered which sometimes causes major failure of the project. Therefore, this is a big concern that it should be taken into considerations. 2.2 Typical challenges of the Architectural practices The applied knowledge of the local practitioners is not up to the mark. Some of them have theoretical knowledge which is not good enough for architecting the business scenario based projects. Some of them are using their tacit knowledge and blind followers of their senior colleagues. But, this practice lacks seriously the standards of the architectural practices. In both the case, their architecture suffers the quality. Therefore, a good guidelines is required which can be practiced easily and will provide the maximum benefit to the stakeholders. There are a lot of materials available in different sources that can be good reference materials. But, this is not very easy to choose the right materials and practice that as a bible. Besides these, the companies have many constraints like lack of skilled resources, inadequate time to conceive the right architecture, mobilize the budget for spending enough time for this 9 purpose, and finally making available all the relevant stakeholders. Therefore, some easy and suitable framework or guidelines are required that can help in this regard. 2.3 Role of ATAM for improving Architectural Practices The ATAM is the best and proven framework among all available frameworks for ensuring the right architectures for the project. It has the specific guidelines and procedures to follow which ultimately gives the rightmost architecture for the project. Therefore, the ATAM can play a very vital role for improving the local architectural practices. But, the ATAM is a vast framework of architecture analysis which is usually considered as unaffordable for the local small and medium sized software companies. Therefore, it requires some adjustments and guidelines for the adaptation to the local practices. In that case, it would be a great help to the practitioners to ensure the right architectural practices. CHAPTER THREE METHODOLOGY 3.1 Literature Review and Defining Scope of Research Study has been conducted on a number of research papers and available standards to develop a clear concept of ATAM, its role as a risk-mitigation process used early in the software development life cycle. The literature review concentrated on the following definitions necessary for the research: • • • • Architectural Concept Component-Based Software Architecture Architecture Evaluation Process Architecture Tradeoff Analysis MethodSM. The main focus of the literature review was: • • Study available architecture evaluation methods, and Applying ATAM framework for Architecture evaluation. Literature review on the above mentioned terms and issues showed that there is a need for defining a process of adapting ATAM for the small and medium companies in Bangladesh. Therefore, the research proposed an adapted ATAM considering the existing practice of architectural design. 3.2 Key Informant Interview A key informant interview has been performed with some of reputed software architects from different software companies, who have years of experience in 11 developing and designing contract projects. The goal was to identify typical nature of architectural practices of contract software development projects from the software architects who have firsthand knowledge and experience with such projects. These software architects, with their particular knowledge and understanding have provided insight on the characteristics and challenges of the project to design the architecture in a cost effective manner and standard process. They have also provided recommendations for different design issues. The typical architecture tradeoff analysis method practices identified from the interviews have been incorporated in the adapted ATAM. 3.2.1 Interview Questions A set of open-ended questions have been used to guide the interview process. But most of the questions have been followed by several supplementary questions that have been deemed necessary for further clarification of interviewees’ response. The interview questions have been designed to get preliminary and in-depth information about the knowledge and practice level of software architecture, the knowledge and practice level of architecture evaluation, the knowledge and practice level of ATAM, its limitation and relevance to the industry practices. The interview also has been intended to get their suggestions and recommendations for applying the ATAM to the small and medium size software companies in Bangladesh. 3.2.2 Selecting Key Informants For selecting the Key informants, several criteria have been considered. These include software engineering knowledge, track record, working experience, serving 12 companies and reputation in the industry. The potential list of key informants included technical project manager, software architect, system analysts and senior software engineers. At least five years working experience has be taken as the major qualifying criterion for the Key informant. 3.2.3 Interview Technique Face-to-Face interview technique has been used to obtain information from each of the key informant. This technique has been selected as because it provides the greater opportunity to exchange the information and idea more details, which is required for getting a clear picture of their knowledge and practices. Several times of meetings were arranged with the interviewees based on their convenient time and place for the interviews. The duration of these interviews were 30-45 minutes for high level discussion and 1 to 2 hours for detailed level discussion. 3.2.4 Recording Interview Responses The interview responses have been recorded by taking notes for each of the questions on the paper. Immediately after each interview, some time has been taken to review the notes and make its electronic copy in Laptop. The notes have been also shared with the interviewees for their clarification and confirmation of understanding through Email. 3.2.5 Organizing Key Informant Interview Information The information identified for each of the questions of key informant interview has been compiled and organized as tabular format and represented in this chapter. 13 3.3 Questionnaire Survey Two separate questionnaire survey has been performed for identifying the high level usage of ATAM and detailed level usage of ATAM. This questionnaire defines the way of identifying the process of mapping the industry practices and the academic knowledge. Using this methodology we can determine the gap of practices between the academic knowledge or standards and industry practices. We can distinctly identify where we need to update or where we can work for further improvement. There are also some practices in industry which are deemed important but not available in academic knowledge or in standards, has been also identified. For collecting the data about the practice level of ATAM, two sets of questionnaire have been created following ATAM steps. First set of questions focuses on high level use of ATAM and the other focuses on detail level practice and knowledge area. There are four main segments of ATAM - Presentation, Investigation & Analysis, Testing and Reporting. For each segment, there are one or more steps. For each step a number of questions have been asked. For each step, detail description and information has been provided in the questionnaire, so that the architect can have a clear understanding about the activities performed in the step and can provide more justified answer relating with the industry practice. 14 3.3.1 High Level Questionnaires Following is the high level questionnaire. This high level questionnaire helps us to trace the overall practice and knowledge level of ATAM to the practitioners. SL No. ATAM Step Description Question Answer Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. 1 Step 1: Present the This step involves the description of the evaluation ATAM process of the ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. 2 Step 2: Present the In this step, the business goals of the architectural drivers business drivers of the system are mentioned. This step focuses on the business perspective of the system. It provides additional information on the functions of the system, the major stakeholders, the business goals and other constraints of the system. 3 Step 3: Present the In this step, the architecture team describes the architecture to be architecture to be evaluated. It focuses on the architecture, evaluated the time availability for the evaluation, and the quality requirements of the architecture under investigation. The Do you practice architecture evaluation process? If yes do you have any process presentation meeting? If yes who are the participants of the meeting. How the participants are chosen? Do you consider business goal or business drivers in architecture meeting? Do you consider the constraints of the business? Do you involve all the relevant stake holders like end-user, architect, developers, etc? Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? 15 SL No. ATAM Step Description architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that represent the desired qualities of a system. Examples of these attributes include performance, reliability etc. Question Does the presentation include architecture diagram in sufficiently detail? Does the presentation address technical constraints? Does the presentation address quality attributes requirement? Does the presentation address quality attribute requirements like performance, reliability, etc.? Answer Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. 4 Step 4: Identify the This step involves identifying the critical architectural Does the architectural team explain the architectural approaches that provide an understanding of the critical flow of control of the architecture? approaches requirements of the system. In this step, the architectural Does the architectural approach address the team explains the flow of control of the architecture and critical requirements? provides a proper explanation of how and whether the Does the architectural team provide critical goal is met. explanation how the critical goal is met? 5 Step 5: Generate the In this phase of the evaluation, the system’s most Quality Attribute important quality attribute goals are identified, prioritized Utility tree. and refined. This step is crucial because it focuses the attention of all the stakeholders and the evaluators on the different aspects of the architecture that are critical to the success of the system. This is achieved through the construction of a utility tree. Are the important quality attributes goals clearly identified? Is the utility tree made? Does the utility tree specify the system goals clearly? Are the scenarios are clearly described? Are the quality attributes are prioritized? Are all the stake holders scenarios The utility tree provides a way of making the system goals considered? more specific and more concrete. It also provides a Are all the quality attributes like the 16 SL No. ATAM Step Description Question security, performance, availability, reliability, modifiability, portability, functionality, variability, conceptual integrity, etc. quality attribute considered? If utility tree is not practiced, is there other way in practice? Answer 6 comparison of the importance of quality attribute goals relative to each other. These requirements are called scenarios. A scenario is a statement that illustrates the interaction between a stakeholder and the system. These scenarios are the quality goals against which the architecture will be judged. Scenario Generation Scenario generation is an important preceding step to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the utility tree. The utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). Step 6: Analyze the This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated Architectural in the previous step is analyzed, and a thorough Approaches investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade- How investigation done to define architectural approach? How risk and non risks are identified? 17 SL No. ATAM Step Description Question Answer 6.1 off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, trade-off points. 6.1 Investigation of After identifying these quality attributes that are important architectural to the goal of the system, the architectures are determined approach how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. 6.2 6.2 Creation of The next stage in this step involves in gathering analysis analysis questions questions generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Does the architectural approach support the quality attribute? How does architectural approach support variability? How does architectural approach support modifiability? How does architectural approach support reliability? How does architectural approach support conceptual integrity? How does architectural approach support functionality? Do you create analysis questions? Can the components of the architecture reused for future projects? Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Can any new application-specific functionality be added to the architecture? 18 SL No. ATAM Step Description Question Can the system be modified in short time and cost-effective manner? Do the components interact properly? Are the answer and explanations to the questions analyzed? Are the risks and non-risks identified properly? Are the sensitivity points identified properly? Are the trades off points identified properly? Answer 6.3 6.4 6.3 Answers to the analysis questions 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. The final stage involved in this step is to identify the risk, non-risk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Phase 3 - Testing 7 Step 7: Brainstorm This is the first step in the Testing Phase of the ATAM. and Prioritize The former represents the interests of the stakeholders and Scenarios is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. Does all the stake holders involve in brainstorm? Are all scenarios collected after brainstorm? Are use case scenarios considered? Are growth scenarios considered? Are exploratory scenarios considered? Are the scenarios are prioritized? Are the voting process used for prioritizing scenarios? 19 SL No. ATAM Step Description Question Answer 8 · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. Step 8: Analyze the This is the last step in the ’Testing’ phase. In this step the architectural highly voted quality attributes from the previous step are Approaches analyzed. The architectural approach that deals with these quality attributes are found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, nonrisks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Step 9: Report the This is the final phase of the ATAM evaluation, where all ATAM the information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches Is the new quality attributes considered found in the brainstorm session? Are the new scenarios considered? Are the new risks, non-risks, sensitivity points and trade off points included? Phase 4 - Report the ATAM 9 Is the utility tree prepared and reported? Are the generated scenarios reported? Are the analysis questions reported? Are the risks and non-risks reported? Is the architectural approach identified, finalized and reported? 20 3.3.3 Detailed Level Questionnaires Following is the detail level questionnaire. This detail level questionnaire helps us to trace the detail practice and knowledge level of ATAM to the practitioners. SL No ATAM Step & Description Question Answer Supplementary Question Supplementary Answer Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. Do you practice Yes 1 Step 1: Present the ATAM architecture evaluation This step involves the description of the process? evaluation process of the ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the No evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. 2 Step 2: Present the business drivers In this step, the business goals of the architectural drivers of the system are mentioned. This step focuses on the business perspective of the system. It Do you consider Yes business goal or No business drivers in architecture meeting? Do you consider the Yes constraints of the 1.1 Why do you practice architecture evaluation process? 1.2 Do you have any Architecture process presentation meeting? 1.3 If yes who are the participants of the meeting? 1.4 How the participants are chosen? 1.5 What are the problems you are facing for not practicing architecture evaluation process? 1.1. What goals are considered? 1.2. Do you have any implications in the long run for not considering business goals? 2.1. What constraints are considered? 21 SL No ATAM Step & Description Question Answer Supplementary Question 2.2. Do you have any implications in the long run for not considering the constraints? 3. 1. Who are the relevant stakeholders? 3.2. Do you have any implications in the long run for not considering all the relevant stakeholders? 1.1. Do you see any implications for not defining architecture beforehand? Supplementary Answer No provides additional information on the business? functions of the system, the major stakeholders, the business goals and other constraints of the system. Do you involve all the Yes relevant stake holders of the project? No 3 Step 3: Present the architecture to be Does the architecture team define any evaluated architecture before In this step, the architecture team describes evaluation? the architecture to be evaluated. It focuses on the architecture, the time availability for Does the presentation the evaluation, and the quality requirements address architectural of the architecture under investigation. The approach? architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the Does the presentation technical constraints, other systems that include architecture interact with the system being evaluated, diagram in sufficiently and the architectural approaches detail? implemented in order to meet the quality Does the presentation attributes. The quality attributes of a system address technical are the issues that represent the desired constraints? qualities of a system. Examples of these attributes include performance, reliability etc. Does the presentation Yes No Yes 2.1. What are the approaches considered? architectural No 2.2. Do you have any implications for not considering the architectural approaches? 3.1. Do you have any implications for not including architecture diagram in detail? 4.1. What are the technical constraints are considered? 4.2. Do you see any implications for not addressing technical constraints? 5.1. What are the quality attributes are Yes No Yes No Yes 22 SL No ATAM Step & Description Question address attributes requirement? quality Answer Supplementary Question considered? Supplementary Answer No 5.2 Do you have any implications in the long run for not considering the quality attributes requirement? Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. 4 Step 4: Identify the architectural Does the architectural Yes team explain the flow approaches This step involves identifying the critical of control of the No 1.1 Do you see any implications for architectural approaches that provide an architecture? not explaining the flow of control? understanding of the critical requirements of the system. In this step, the architectural team explains the flow of control of the 2.1. What are the critical requirements architecture and provides a proper Does the architectural Yes approach address the are considered? explanation of how and whether the critical critical requirements? 2.2. Does the architectural team goal is met. provide explanation how the critical goal is met? No 2.3 Do you have any implications in the long run for not considering the critical requirements? 1.1 What are the important qualities 5 Step 5: Generate the Quality Attribute Are the important Yes quality attributes goals attributes goals? Utility tree. clearly identified? No 1.2 What are the implications for not In this phase of the evaluation, the system’s identifying these? most important quality attribute goals are Is the utility tree Yes 2.1 What are the advantages for this? identified, prioritized and refined. This step made? No 2.2 What are the implications of not 23 SL No ATAM Step & Description is crucial because it focuses the attention of all the stakeholders and the evaluators on the different aspects of the architecture that are critical to the success of the system. This is achieved through the construction of a utility tree. Question Answer Supplementary Question practicing this? 3.1 What are the system goals considered for the utility tree? 3.2 What are the implications of not practicing this? 4.1 What are the scenarios to be considered? 4.2 What are the implications of not practicing this? 5.1 What are the criteria to prioritize quality attributes? 5.2 What are the implications for not prioritizing quality attributes? 6.1 What are the common scenarios considered? 6.2 What are the implications for not considering all the scenarios? 7.1 What are the attributes considered? 7.2 What are the implications for missing attributes? 7.3 What are the implications for not considering the quality attributes? Supplementary Answer Does the utility tree Yes specify the system goals clearly? No Are the scenarios are Yes clearly described? The utility tree provides a way of making No the system goals more specific and more concrete. It also provides a comparison of Are the quality Yes the importance of quality attribute goals attributes are relative to each other. prioritized? No These requirements are called scenarios. A scenario is a statement that illustrates the Are all the stake Yes interaction between a stakeholder and the holders scenarios system. These scenarios are the quality considered? No goals against which the architecture will be judged. Are the quality Yes attributes like the Scenario Generation security, performance, Scenario generation is an important availability, reliability, No preceding step to the utility tree creation. modifiability, portability, Utility tree generation functionality, The utility tree contains ’utility’ as the root variability, conceptual node, with the quality attributes forming integrity, etc. quality the secondary level of the utility tree. attribute considered? Under each of the quality attributes, If utility tree is not Yes specific quality attribute refinements are practiced, is there included that provide a more precise other way in practice? 8.1 Give the descriptions of the practices. 24 SL No ATAM Step & Description Question Answer No Supplementary Question 8.2 What are the implications for practicing nothing like utility tree? Supplementary Answer 6 description of the scenarios. The latter form the leaf nodes in the utility tree. The utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). Step 6: Analyze the Architectural 1. Do you analyze architectural Approaches approaches This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output 2. Are risk and non of the utility tree generated in the previous risks identified? step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions Yes No Yes No 1.1 How investigation done to define architectural approach? 1.2 What are the implications of not doing so? 2.1 How risk and non risks are identified? 2.2 What are the implications of not identifying risks and non-risks? 25 SL No ATAM Step & Description - Find the risks, non-risks, sensitivity points, trade-off points. 6.1 Investigation of architectural approach After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Question Answer Supplementary Question Supplementary Answer 6.1 Does the architectural Yes approach support the quality attribute? No Does architectural Yes approach support variability? No How does Yes architectural approach support modifiability? No Does architectural Yes approach support reliability? No Does architectural Yes approach support conceptual integrity? No Does architectural Yes approach support functionality? No 6.2 6.2 Creation of analysis questions Do you create analysis Yes The next stage in this step involves in questions? gathering analysis questions generated from the high prioritized scenarios already 1.1 What are the quality attributes considered? 1.2 What are the implications of not practicing this? 2.1 What are the processes to identify the variability? 2.2 What are the implications of not practicing this? 3.1 What are the processes to identify the modifiability? 3.2 What are the implications of not practicing this? 4.1 What are the processes to identify the reliability? 4.2 What are the implications of not practicing this? 5.1 What are the processes to identify the conceptual integrity? 5.2 What are the implications of not practicing this? 6.1 What are the processes to identify the functionality of project? 6.2 What are the implications of not practicing this? 1.1 What are the steps followed to create analysis questions? 1.2. What are covered in analysis questions? 26 SL No ATAM Step & Description discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Question Answer No Supplementary Question 1.3 What are the implications of not practicing this? 2.1 What are the design level considerations for component reusability? 2.2 What are the implications of not practicing this? 3.1 What are the steps considered for the framework expansion? 3.2 What are the implications of not practicing this? 4.1 How the validation considered? 4.2 What are the implications of not practicing this? 5.1 What are the steps followed to identify the architectural behavior? 5.2 What are the implications of not practicing this? 6.1 How the decision taken at the time of design to add application specific functionality? 6.2 What are the implications of not practicing this? 7.1 How the time and cost calculated for change management? 7.2 What are the implications of not practicing this? 8.1 How the component interaction Supplementary Answer Can the components Yes of the architecture reused for future projects? No Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Yes No Yes No Yes No Can any new Yes application-specific functionality be added to the architecture? No Can the system be Yes modified in short time and cost-effective No manner? Do the components Yes 27 SL No ATAM Step & Description Question interact properly? Answer Supplementary Question tested? 8.2 What are the implications of not practicing this? 1.1 How the answer and explanations to the questions analyzed? 1.2 What are the implications of not practicing this? 1.1 What are the processes to identify the risk and non risks? 1.2. What are the common risks and non-risks? 1.3 What are the implications of not practicing this? 2.1 What are the processes to identify the sensitivity points? 2.2 What are the common sensitivity points? 2.3 What are the implications of not practicing this? 3.1 What are the processes to identify the trade off points? 3.2 What are the common trade-off points? 3.3 What are the implications of not practicing this? Supplementary Answer No 6.3 6.3 Answers to the analysis questions The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The final stage involved in this step is to identify the risk, non-risk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. Are the A Sensitivity point is a property of one or points more components, that is critical for the properly? achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Are the answer and Yes explanations to the questions analyzed? No Are the risks and non- Yes risks identified properly? No Are the points properly? sensitivity Yes identified 6.4 No trade off Yes identified No Phase 3 – Testing 7 Step 7: Brainstorm and Prioritize Scenarios Does all the stake Yes This is the first step in the Testing Phase of holders involve in 1.1 What are the criteria to select stakeholders? 28 SL No ATAM Step & Description the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. Question brainstorm? Answer No Supplementary Question 1.2 What are the implications of not practicing this? 2.1 Give some example of Scenarios? 2.2 What are the implications of not practicing this? 3.1 What are the cases of scenarios considered? 3.2 What are the implications of not practicing this? 4.1 What are the steps to consider growth scenarios? 4.2 What are the implications of not practicing this? 5.1 What are the steps to consider exploratory scenarios? 5.2 What are the implications of not practicing this? 6.1 How the scenarios are prioritized? 6.2 What are the implications of not practicing this? 7.1 What is the voting process for prioritizing scenarios? 7.2 What are the implications of not using voting process? 1.1 How do you do that? 1.2 What are the implications of not practicing this? 2.1 What are the processes to consider Supplementary Answer Are all scenarios Yes collected after brainstorm? No Are use case scenarios Yes considered? No Are growth scenarios Yes considered? No Are exploratory Yes scenarios considered? No Are the scenarios are Yes prioritized? No Are the voting process Yes used for prioritizing scenarios? No 8 Step 8: Analyze the architectural Approaches This is the last step in the ’Testing’ phase. In this step the highly voted quality attributes from the previous step are Are the new quality Yes attributes considered No found in the brainstorm session? Are the new scenarios Yes 29 SL No ATAM Step & Description analyzed. The architectural approach that deals with these quality attributes are found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Question considered? Answer Supplementary Question new scenarios? 2.2 What are the implications of not practicing this? 3.1 What are the processes to consider risk and non risk items? 3.2 What are the processes to consider sensitivity points? 3.3 What are the processes to consider trade off points? 3.4 What are the implications of not practicing this? Supplementary Answer No Are the new risks, Yes non-risks, sensitivity points and trade off points included? No Phase 4 - Report the ATAM 9 Step 9: This is the final phase of the Report the ATAM evaluation, where all ATAM the information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and Is the utility tree Yes prepared and reported? No Are the generated Yes scenarios reported? No Are the analysis Yes questions reported? No Are the risks and non- Yes risks reported? No Is the architectural Yes 1.1 What are they? 1.2 What are the implications of not practicing this? 2.1 What are they? 2.2 What are the implications of not practicing this? 3.1 What are they? 3.2 What are the implications of not practicing this? 4.1 What are they? 4.2 What are the implications of not practicing this? 5.1 What are the common approaches 30 SL No ATAM Step & Description Question Answer Supplementary Question identified? 5.2 What are the implications of not practicing this? Supplementary Answer non-risks approach identified, - the identified architectural finalized and reported? No approaches 31 3.3.4 Quantification of Feedback The narrative answers of the participants are converted to quantitative feedback following some procedures based on the merits of answers described in the section 5.1. 3.4 Limitations • The ATAM is very vast framework and the number of questions is many. Therefore, the key informant has to provide long amount of time which is difficult to manage. • The questions are not very straight forward as concept of architecture evaluation among the local practitioners in small and medium sized companies is not very common. Therefore, detail description and background information has been provided. This has also made the questionnaire bulky and difficult to deal with. • The answers are mostly recorded or written by the interviewer as the interviewees did not agree to do that due to bulkiness of the questions. Therefore, there is chance of some degree of information loss or alteration though these are tried to verify by the respective informants. • All the firsthand inputs have been taken as qualitative manner and later converted them to quantitative points. Therefore, there is a chance of some degree of errors. • Due to the bulkiness of the framework, the analysis of data from different perspective would be limited and hence the findings might not be the 100% true representation of the reality. CHAPTER FOUR LITERATURE REVIEW 4.1 Introduction Software architecture is the blueprint of a software system. Its description provides an abstract representation of components and their interactions of a software system by means of Architecture Description Languages (ADLs). This technique is called Component-Based Software Architecture (CBSA). CBSA helps software architects to abstract the details of implementation and facilitates the manipulation and the reuse of components. Software architecture provides an overview of the composition and functionality of the given software system. Just like a structural architect, a software architect needs to analyze the requirements, determine components that should be used to build the system, and support the project by guiding and solving problems all along the execution cycle. Understanding of the domain provides the capability to analyze the requirements and envisioning a solution. Exposure to various tools and technologies imparts the ability to choose among the various solutions available, and anticipate and solve technical problems. In short we can say about Software Architecture: Software architecture represents the structure of the software. This includes the structural arrangements of software components, and various static and dynamic interrelationships between these components. 33 Software architecture is expressed using certain views, each of which serves a specific purpose. Each view is a specific abstraction of the architecture, for a specific purpose. Software architecture includes the principles behind design and evolution of the software. The essential characteristics of architecture are as follows: Software architecture should represent a high-level view of the system revealing the structure, but hiding all implementation details. Specifically, it should reveal attributes such as responsibilities (of the constituents of the architecture), distribution, and deployment. Architecture should realize all the use case scenarios. While the use case model serves to record the functional requirements as seen by various actors, the architecture should enable the stake holders of the software to walk through the scenarios of each use case. This guarantees that the structure as represented by the architecture meets the functional requirements. It should present other systemic views to all the stake holders of the software. Examples are - a component view for the development team, a network-centric deployment view for the network and hardware team, and a distributioncentric deployment view for the installation team, etc. Some of the common representations of software architecture are as follows. High-level design: A simplistic approach is to represent the architecture as a concise view of a high-level design of the software. However, design is an implementer’s view of the software - a view that reveals how to arrive at the 34 structure of the software. So a high-level design does not necessarily represent the architecture of the software. Deployment: This is the most common form of representations of architecture. In this view, the software is described in terms of how it is deployed across various platforms, and how these parts communicate with each other. Note that deployment view is only one of the possible views of architecture, and does not necessarily reveal the structure of the software. Also note that during the life-cycle of a software, the deployment could change with no or minimal changes to the structure of the software. This is one of the reasons that is driving distributed component based technologies. Generic Technology Architectures: In this form, software architecture is represented as a two-tier, three-tier or a multi-tier system. Note that architectures such as these, distributed (and layered) architectures such as COM or CORBA, or component based architectures such as MTS or EJB are generic and do not address the needs of the domain in which the software is to operate and evolve. However, these technology architectures provide the basis for developing domain specific application architectures. 4.2 Background Although the term “software architecture” is relatively new to the industry, the fundamental principles of the field have been applied sporadically by software engineering pioneers since mid 1980s. Early attempts to capture and explain software architecture of a system were imprecise and disorganized - often characterized by a set of box-and-line diagrams. During the 1990’s there was a concentrated effort to define and codify fundamental aspects of the discipline. Initial sets of design patterns, 35 styles, best practices, description languages, and formal logic were developed during that time. The software architecture discipline is centered on the idea of reducing complexity through abstraction and separation of concerns. To date there is still no agreement on the precise definition of the term “software architecture”. Because software architecture is a major determinant of software quality, it follows that software architecture is critical to the quality of any software-intensive system. For a Department of Defense (DoD) acquisition organization, the ability to evaluate software architectures before they are realized in finished systems can substantially reduce the risk that the delivered systems will not meet their quality goals. Over the past several years, the Software Engineering Institute (SEISM) has developed the Architecture Tradeoff Analysis MethodSM (ATAM) and validated its usefulness in practice [10]. This method not only permits evaluation of specific architectural quality attributes (e.g., modifiability, performance, security, and reliability) but also engineering tradeoffs to be made among possibly conflicting quality goals [7]. The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them [8]. The software architecture for a system represents the earliest software design decisions. As such, they are the most critical things to get right and the most difficult things to change downstream in the development life cycle. Even more important, the software architecture is the key to software quality; it permits or precludes the system’s ability to meet functional and quality attribute goals and requirements such as reliability, modifiability, security, real-time performance, and interoperability. The 36 right software architecture can pave the way for successful system development, while the wrong architecture results in a system that fails to meet critical requirements and incurs high maintenance costs. [7] Modern treatment of software architecture takes advantage of the fact that there are many relevant views of a software architecture. A view is a representation of some of the system’s elements and the relationships associated with them. Views help us separate concerns and achieve intellectual control over an abstract concept. Different views also speak to different stakeholders—those who have a vested interest in the architecture. Which ones are relevant depends on the stakeholders and the system properties that interest them. If we consider the analogy of a building’s architecture, various stakeholders (such as the construction engineer, plumber, and electrician) all have an interest in how the building is to be constructed. Although they are each interested in different elements and relationships, each of their views is valid—each one represents a structure that maps to one of the building’s construction goals. A suite of views is necessary to engineer the architecture of the building fully and to represent that architecture to stakeholders. [7] Similarly, a software architecture has a variety of stakeholders, including possibly the development organization, end user, system maintainer, operator, and acquisition organization. Each stakeholder has a vested interest in different system properties and goals that are represented by different structural views of the system. These different properties and goals, and their corresponding architectural views are important to engineer, understand, and analyze. They all provide the basis for reasoning about the appropriateness and quality of the architecture. Some experts prescribe using a fixed 37 set of views. Rational’s Unified Process (RUP), for example, relies on Kruchten’s “4+1 view” approach to software architecture. A current and more healthy trend, however, is to recognize that architects should choose a set of views based on the needed engineering leverage that each view provides and the stakeholder interests that each one serves. This trend is exemplified by the recent American National Standards Institute/Institute of Electrical and Electronics Engineers (ANSI/IEEE) recommended practice for architectural documentation on software-intensive systems [13] and the “views and beyond” approach to architecture documentation from the SEI [11]. 4.3 History The origin of software architecture as a concept was first identified in the research work of Edsger Dijkstra in 1968 and David Parnas in the early 1970s. The scientists emphasized that the structure of a software system matters and getting the structure right is critical. The study of the field increased in popularity since the early 1990s with research work concentrating on architectural styles (patterns), architecture description languages, architecture documentation, and formal methods. Research institutions have played a prominent role in furthering software architecture as a discipline. Mary Shaw and David Garlan of Carnegie Mellon wrote a book titled Software Architecture: Perspectives on an Emerging Discipline in 1996, which brought forward the concepts in Software Architecture, such as components, connectors, styles and so on. The University of California, Irvine's Institute for Software Research's efforts in software architecture research is directed primarily in architectural styles, architecture description languages, and dynamic architectures. Kruchten, Booch, Bittner, and Reitman on Architecture 38 Philippe Kruchten, Grady Booch, Kurt Bittner, and Rich Reitman derived and refined a definition of architecture based on work by Mary Shaw and David Garlan (Shaw and Garlan 1996). Their definition is: “Software architecture encompasses the set of significant decisions about the organization of a software system including: Selection of the structural elements and their interfaces by which the system is composed. Behavior as specified in collaboration among those elements. Composition of these structural and behavioral elements into larger subsystems. Architectural style that guides this organization. Software architecture also involves functionality, usability, resilience, performance, reuse, comprehensibility, economic and technology constraints, tradeoffs and aesthetic concerns.” [7] Fowler on Architecture In Patterns of Enterprise Application Architecture, Martin Fowler outlines some common recurring themes when explaining architecture [7]: • • • • • The highest-level breakdown of a system into its parts. The decisions that are hard to change. There are multiple architectures in a system. What is architecturally significant can change over a system’s lifetime. In the end, architecture boils down to whatever the important stuff is. Bass, Clements, and Kazman on Architecture In Software Architecture in Practice (2nd edition), Bass, Clements, and Kazman define architecture as follows: “The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the 39 relationships among them. Architecture is concerned with the public side of interfaces; private details of elements—details having to do solely with internal implementation—are not architectural.” [7] 4.4 Architectural Views Software architecture is commonly organized in views, which are analogous to the different types of blueprints made in building architecture. Within the ontology established by ANSI/IEEE 1471-2000, views are instances of viewpoints, where a viewpoint exists to describe the architecture in question from the perspective of a given set of stakeholders and their concerns. Some possible views are: Functional/logic view Code view Development/structural view Concurrency/process/thread view Physical/deployment view User action/feedback view Several languages for describing software architectures have been devised, but no consensus has yet been reached on which symbol-set and view-system should be adopted. The UML was established as a standard "to model systems (and not just software)," and thus applies to views about software architecture. Others believe that effective development of software relies on understanding unique constraints of each problem, and so universal notations are doomed because each provides a notational bias that necessarily makes the notation useless or dangerous for some set of tasks. They point to the proliferation of programming languages and a succession of failed 40 attempts to impose a single 'universal language' on programmers, as proof that software thrives on diversity and not on standards. 4.5 Architectural Styles There are many common ways of designing computer software modules and their communications. Some of them are described in the following sections. 4.5.1 Blackboard A blackboard system enables this flexible brainstorming style of interaction between diverse software specialists. Each of these specialists scans the changes to the blackboard, and posts an updated partial solution based on the state of the blackboard whenever its own internal conditions for doing so are met. These partial solutions cause other knowledge sources to update their portions of the solution on the blackboard until eventually an answer is found. In this fashion, the specialists work together to solve the problem. 4.5.2 Client-server Client-server is a computing architecture which separates a client from a server, and is almost always implemented over a computer network. Each client or server connected to a network can also be referred to as a node. The most basic type of client-server architecture employs only two types of nodes: clients and servers. This type of architecture is sometimes referred to as two-tier. It allows devices to share files and resources. Each instance of the client software can send data requests to one or more 41 connected servers. In turn, the servers can accept these requests, process them, and return the requested information to the client. Although this concept can be applied for a variety of reasons to many different kinds of applications, the architecture remains fundamentally the same. These days, clients are most often web browsers, although that has not always been the case. Servers typically include web servers, database servers and mail servers. Online gaming is usually client-server too. The interaction between client and server is often described using sequence diagrams. Sequence diagrams are standardized in the Unified Modeling Language. 4.5.3 Database-centric architecture Database-centric architecture or data-centric architecture has several distinct meanings, generally relating to software architectures in which databases play a crucial role. Often this description is meant to contrast the design to an alternative approach. For example, the characterization of an architecture as "database-centric" may mean any combination of the following: using a standard, general-purpose relational database management system, as opposed to customized in-memory or filebased data structures and access methods. With the evolution of sophisticated DBMS software, much of which is either free or included with the operating system, application developers have become increasingly reliant on standard database tools, especially for the sake of rapid application development. using dynamic, table-driven logic, as opposed to logic embodied in previously compiled programs. The use of table-driven logic, i.e. behavior that is heavily dictated by the contents of a database, allows programs to be simpler and more flexible. This capability is a central feature of dynamic programming languages. 42 Using stored procedures that run on database servers, as opposed to greater reliance on logic running in middle-tier application servers in a multi-tier architecture. The extent to which business logic should be placed at the back-end versus another tier is a subject of ongoing debate. For example, Toon Koppelaars presents a detailed analysis of alternative Oracle-based architectures that vary in the placement of business logic, concluding that a database-centric approach has practical advantages from the standpoint of ease of development and maintainability. Using a shared database as the basis for communicating between parallel processes in distributed computing applications, as opposed to direct inter-process communication via message passing functions and message-oriented middleware. A potential benefit of database-centric architecture in distributed applications is that it simplifies the design by utilizing DBMS-provided transaction processing and indexing to achieve a high degree of reliability, performance, and capacity. For example, Base One describes a database-centric distributed computing architecture for grid and cluster computing, and explains how this design provides enhanced security, fault-tolerance, and scalability. 4.5.4 Distributed Computing Distributed computing is a method of computer processing in which different parts of a program are run simultaneously on two or more computers that are communicating with each other over a network. Distributed computing is a type of segmented or parallel computing, but the latter term is most commonly used to refer to processing in which different parts of a program run simultaneously on two or more processors that are part of the same computer. While both types of processing require that a 43 program be segmented—divided into sections that can run simultaneously, distributed computing also requires that the division of the program take into account the different environments on which the different sections of the program will be running. For example, two computers are likely to have different file systems and different hardware components. An example of distributed computing is BOINC, a framework in which large problems can be divided into many small problems which are distributed to many computers. Later, the small results are reassembled into a larger solution. Distributed computing is a natural result of using networks to enable computers to communicate efficiently. But distributed computing is distinct from computer networking or fragmented computing. The latter refers to two or more computers interacting with each other, but not, typically, sharing the processing of a single program. The World Wide Web is an example of a network, but not an example of distributed computing. There are numerous technologies and standards used to construct distributed computations, including some which are specially designed and optimized for that purpose, such as Remote Procedure Calls (RPC) or Remote Method Invocation (RMI) or .NET Remoting. 4.5.5 Event Driven Architecture Event-driven architecture (EDA) is a software architecture pattern promoting the production, detection, consumption of, and reaction to events. An event can be defined as "a significant change in state". For example, when a consumer purchases a car, the car's state changes from "for sale" to "sold". A car dealer's system architecture may treat this state change as an event to be detected, produced, published and consumed by various applications within the architecture. This architectural pattern may be applied by the design and implementation of applications and systems which 44 transmit events among loosely coupled software components and services. An eventdriven system typically consists of event consumers and event producers. Event consumers subscribe to an intermediary event manager, and event producers publish to this manager. When the event manager receives an event from a producer, the manager forwards the event to the consumer. If the consumer is unavailable, the manager can store the event and try to forward it later. This method of event transmission is referred to in message-based systems as "store and forward". Building applications and systems around an event-driven architecture allows these applications and systems to be constructed in a manner that facilitates more responsiveness, because event-driven systems are, by design, more normalized to unpredictable and asynchronous environments. Event-driven architecture complements service-oriented architecture (SOA) because services can be started by triggers such as events. Computing machinery and sensing devices can detect state changes of objects or conditions and create events which can then be processed by a service or system. Event triggers are conditions that result in the creation of an event. 4.5.6 Service-oriented Architecture Service Oriented Architecture (SOA) is a computer systems architectural style for creating and using business processes, packaged as services, throughout their lifecycle. SOA also defines and provisions the IT infrastructure to allow different applications to exchange data and participate in business processes. These functions are loosely coupled with the operating systems and programming languages underlying the applications. 45 SOA separates functions into distinct units (services), which can be distributed over a network and can be combined and reused to create business applications. These services communicate with each other by passing data from one service to another, or by coordinating an activity between two or more services. Companies have long sought to integrate existing systems in order to implement information technology (IT) support for business processes that cover all present and prospective systems requirements needed to run the business end-to-end. A variety of designs can be used to this end, ranging from rigid point-to-point electronic data interchange (EDI) interactions to Web auctions. By updating older technologies, such as Internetenabling EDI-based systems, companies can make their IT systems available to internal or external customers; but the resulting systems have not proven to be flexible enough to meet business demands. 4.5.7 Implicit Invocation The idea behind implicit invocation is that instead of invoking a procedure directly, a component can announce (or broadcast) one or more events. Other components in the system can register an interest in an event by associating a procedure with the event. When the event is announced the system itself invokes all of the procedures that have been registered for the event. Thus an event announcement implicitly causes the invocation of procedures in other modules." 4.5.8 Monolithic Application In software engineering, a monolithic application describes a single-tiered software application in which the user interface and data access code are combined into a 46 single program from a single platform. A monolithic application is self contained, and independent from other computing applications. The design philosophy is that the application is responsible not just for a particular task, but can perform every step needed to complete a particular function. Today, some personal finance applications are monolithic in the sense that they help the user carry out a complete task, end to end, and are "private data silos" rather than parts of a larger system of applications that work together. Word processors are examples of monolithic architectural style. 4.5.9 Peer-to-peer A peer-to-peer (or "P2P", or, rarely, "PtP") computer network uses diverse connectivity between participants in a network and the cumulative bandwidth of network participants rather than conventional centralized resources where a relatively low number of servers provide the core value to a service or application. Peer-to-peer networks are typically used for connecting nodes via largely ad hoc connections. Such networks are useful for many purposes. Sharing content files containing audio, video, data or anything in digital format is very common, and realtime data, such as telephony traffic, is also passed using P2P technology. A pure peer-to-peer network does not have the notion of clients or servers, but only equal peer nodes that simultaneously function as both "clients" and "servers" to the other nodes on the network. This model of network arrangement differs from the client-server model where communication is usually to and from a central server. A typical example for a non peer-to-peer file transfer is an FTP server where the client and server programs are quite distinct, and the clients initiate the download/uploads and the servers react to and satisfy these requests. 47 4.5.10 Pipes and Filters In software engineering, a pipeline consists of a chain of processing elements (processes, threads, coroutines, etc.), arranged so that the output of each element is the input of the next. Usually some amount of buffering is provided between consecutive elements. The information that flows in these pipelines is often a stream of records, bytes or bits. The concept is also called the pipes and filters design pattern. It was named by analogy to a physical pipeline. 4.5.11 Plugin A plugin (plug-in, addin, add-in, addon, or add-on) is a computer program that interacts with a host application (a web browser or an email client, for example) to provide a certain, usually very specific, function "on demand". Applications support plugins for many reasons. Some of the main reasons include: enabling third-party developers to create capabilities to extend an application, to support features yet unforeseen, reducing the size of an application, and separating source code from an application because of incompatible software licenses. Some existing applications and their plugins are discussed below. Email clients use plugins to decrypt and encrypt email (Pretty Good Privacy), Graphics software use plugins to support file formats and process images (Adobe Photoshop), Media players use plugins to support file formats and apply filters (foobar2000, GStreamer, Quintessential, VST, Winamp, XMMS) , Packet sniffers use plugins to decode packet formats (OmniPeek) , Remote sensing applications use plugins to process data from different sensor types (Opticks) , Software development 48 environments use plugins to support programming languages (Eclipse, jEdit, MonoDevelop) , Web Browsers use plugins to play video and presentation formats (Flash, QuickTime, Microsoft Silverlight) , Some digital mixing consoles allow plugins to extend features such as reverberation effects, equalization and compression. 4.6 Architecture Evaluation Methods Software Architecture (SA) has been attracting tremendous attention from researchers and practitioners since the last decade of 20th century. The increasing size, complexity, and demand for quality software systems are some of the most important issues that have increased interest in this sub-discipline of software engineering. The size of software systems has increased tenfold over the past decade or so. The problems of developing and maintaining large systems have led to a realization that high level design description can play an important role in successfully understanding and managing large and complex software systems [1, 20]. During recent years, researchers and practitioners have also recognized that quality attributes (such as maintainability, reliability, usability, performance, flexibility etc.) of large software systems are largely constrained by the systems’ SA [21]. Since SA plays a significant role in achieving system wide quality attributes, it is very important to evaluate a system’s architecture with regard to desired quality requirements as early as possible. The principle objective of SA evaluation is to assess the potential of the chosen architecture to deliver a system capable of fulfilling required quality requirements and to identify any potential risks [22]. Additionally, it is quicker and less expensive to detect and fix design errors during the initial stages of the software development. That is why an effective SA assessment method to analyze prospective architectures is of great business value [23]. 49 A number of methods have been developed to evaluate quality related issues at the SA level. The SA evaluation methods specifically studied in this paper are: Scenariobased Architecture Analysis Method (SAAM) [24], Architecture Tradeoff Analysis Method (ATAM) [25], Active Reviews for Intermediate Design (ARID) [26], SAAM for Evolution and Reusability (SAAMER) [27], Architecture-Level Modifiability Analysis (ALMA) [28] , Architecture-Level Prediction of Software Maintenance (ALPSM)1 [29], Scenario-Based Architecture Reengineering (SBAR)[30], SAAM for Complex Scenarios (SAAMCS) [31], and integrating SAAM in domain-Centric and Reuse-based development (ISAAMCR) [32]. These are scenario-based methods, a category of evaluation methods considered quite mature. There are also some attribute model-based methods and quantitative models for SA evaluation (for example, [3335] etc.), but, these methods are still being validated and are considered complementary techniques to scenariobased methods. There is, however, little consensus on the technical and non-technical issues that a method should fully address and which of the existing methods is most suitable for a particular issue. There is not much work done on systematic classification and comparison of the existing methods. Moreover, there is not much guidance on the desirable features of the evaluation methods and their usefulness. Two research groups have attempted to address informally some of the above mentioned issues by providing a conceptual framework for method comparison [1, 36]. Each of these efforts enhances our understanding of the available methods. 50 4.7 Architecture Tradeoff Analysis MethodSM The ATAM relies on the principle that an architecture is suitable (or not suitable) only in the context of specific quality attributes that it must impart to the system. The ATAM uses stakeholders’ perspectives to produce a collection of scenarios that define the qualities of interest for the particular system under consideration. Scenarios give specific instances of usage, performance and growth requirements, various types of failures, and various possible threats and modifications. Once the important quality attributes are identified in detail, the architectural decisions relevant to each one can be illuminated and analyzed with respect to their appropriateness. [7] The steps of the ATAM leading up to analysis are carried out in two phases. In Phase 1, the evaluation team interacts with the system’s primary decision makers: the architect(s), manager(s), and perhaps a marketing or customer representative. During Phase 2, a larger group of stakeholders is assembled, including developers, testers, maintainers, administrators, and users. The two-phase approach ensures that the analysis is based on a broad and appropriate range of perspectives [7]. 4.7.1 Trade off Analysis Trade- off analysis is a systemic approach to balancing the trade-offs between time, cost and performance. Information is required from costing, scheduling, project review reports, and the original plan. This information can be used to support the following generally accepted six step approach to trade-off analysis [4]. • Recognizing and understanding the basis for project conflicts 51 • • • • • Reviewing the project objectives Analyzing the project environment and status Identifying the alternative courses of action Analyzing and selecting the best alternative Revising the project plan Here project can be any solution or software system. Software architecture trade off analysis helps to make decision about different approach of development considering the time, cost and quality. Anyone can identify the risk and the quality attributes corresponding to the tradeoff analysis. Quantifiable characteristics of the Software Architecture can be identified. Different quality attributes like reliability, performance, security, availability, modifiability, plugability etc. can be focused and prioritize to create the architecture. 4.7.2 Description of ATAM The ATAM evaluates architectural decisions based on quality attribute requirements. The quality attributes in a system are the desired qualities for that system, such as good performance, modifiability, security, etc [8]. To better present the ATAM steps in a concise manner, refer to the figure below. 52 Figure 1 : A simple pictorial showing of the main steps and sub-steps of ATAM The ATAM is meant to be a risk detection method, a means of detecting areas of prospective risk within the architecture of a complex software exhaustive system. This has several implications: The ATAM can be done early in the software development life cycle. It can be done relatively inexpensively and quickly (because it is assessing architectural design artifacts). The ATAM will produce analyses commensurate with the level of detail of the architectural specification. Furthermore it need not produce detailed analyses of any measurable quality attribute of a system (such as latency or mean time to failure) to be successful. Instead, success is achieved by identifying trends. The ATAM is an analysis method organized around the idea that architectural styles are the main determiners of architectural quality attributes. The method focuses on the identification of business goals which lead to quality attribute goals. Based upon the 53 quality attribute goals, we use the ATAM to analyze how architectural styles aid in the achievement of these goals [8]. The steps of the method are as follows: 4.7.2.1 Presentation Present the ATAM: The method is described to the assembled stakeholders (typically customer representatives, the architect or architecture team, user representatives, maintainers, administrators, managers, testers, integrators, etc.) [8]. Present business drivers: The project manager describes what business goals are motivating the development effort and hence what will be the primary architectural drivers (e.g., high availability or time to market or high security) [8]. Present architecture: The architect will describe the proposed architecture, focusing on how it addresses the business drivers [8]. 4.7.2.2 Investigation and Analysis Identify architectural approaches: Architectural approaches are identified by the architect, but are not analyzed [8]. Generate quality attribute utility tree: 54 The quality factors that comprise system “utility” (performance, availability, security, modifiability, etc.) are elicited, specified down to the level of scenarios, annotated with stimuli and responses, and prioritized [8]. Analyze architectural approaches: Based upon the high-priority factors identified in Step 5, the architectural approaches that address those factors are elicited and analyzed (for example, an architectural approach aimed at meeting performance goals will be subjected to a performance analysis). During this step, architectural risks, sensitivity points, and tradeoff points are identified [8]. 4.7.2.3 Testing Brainstorm and prioritize scenarios: Based upon the exemplar scenarios generated in the utility tree step, a larger set of scenarios is elicited from the entire group of stakeholders. This set of scenarios is prioritized via a voting process involving the entire stakeholder group [8]. Analyze architectural approaches: This step reiterates step 6, but here the highly ranked scenarios from Step 7 are considered to be test cases for the analysis of the architectural approaches determined thus far. These test case scenarios may uncover additional architectural approaches, risks, sensitivity points, and tradeoff points which are then documented [8]. 55 4.7.2.4 Reporting Present results: Based upon the information collected in the ATAM (styles, scenarios, attributespecific questions, the utility tree, risks, sensitivity points, tradeoffs) the ATAM team presents the findings to the assembled stakeholders and potentially writes a report detailing this information along with any proposed mitigation strategies [8]. CHAPTER FIVE DATA ANALYSES AND RESULTS 5.1 Introduction The data were gathered from the companies in high level and in detail level. For analysis of the data gathered from the companies the following method was used: The analysis is conducted for each segment of ATAM. Both the high level and detail level narrative answers of the practitioners of five companies have been represented in the tabular format. Then the narrative answers were converted to quantitative points and presented in the corresponding tables. The quantitative data have been presented in graphical manner for better understanding and visibility. For each questions point range is 0 to 10. For full compliance/practice the point is 10 and for no compliance/practice the point is 0. For other cases the points will be based on the level of compliance in between 1 to 9. For the supplementary implication question, the companies which are practicing the step will get full points and the others will get fractional points based on their understanding. The entire table contains five companies’ practices to define and present the architecture for different types of project in terms of ATAM. The result is presented step by step following the ATAM framework steps. 57 5.2 ATAM Step by Step Data Analysis 5.2.1 Phase 1 - Presentation Step 1: Present the ATAM The Table 1 contains the high level answers of the surveyed companies for the Step 1 of ATAM framework. It is evident that, they are following some process in the right way and some are not. Table 1 : High Level Feedback on Step 1 Question Do you practice architecture evaluation process? Do you have any process presentation meeting? who are the participants of the meeting Company A Yes Company B Yes Company C Yes Company D Yes. Company E Yes No define process. Every one related to the development No define process. Engineers from the development and marketing. Senior engineers from the development and marketing. Yes, but no define process. Engineers from the development and management. All the engineers from the development and marketing. Yes. No defined. Development Team All the related stake holders. How the participants are chosen Every one related to the development and requirement analyzing. Everyone who is and will be related to the development of the project. Among the development team The Table 2 contains the detailed feedback of the surveyed companies for the ATAM step 1. Table 2 : Detail Level Feedback on Step 1 Question Why do you practice architecture evaluation process? Do you have any Architecture process presentation meeting? If yes who are Company A For not doing the mistake of choosing wrong architecture. Normally we do. Company B It helps us track the design decision of architecture. We collect all functionality of the project to present meeting. Company C Just to make sure that we are choosing the right architecture. Usually we have. Company D It helps us make the design decision of architecture in a standard way. We collect all features of the project. Collect expected enhancement possibilities information. Business System Company E It helps us make the design decision of architecture in a standard way. We collect all features of the project. The Project System Usually the Development 58 Question the participants of the meeting? Company A members. Company B Analyst, Architect, Project Manager, Team Lead and Developers. From the project criteria we select the most skilled people for the project from their historical work experience in the company. If any change required in the project or any enhanced requirement comes it is difficult to plug the functionality. Company C Project members. Sometimes client technical person also. Depends on the project. Usually senior and talented members are chosen. Company D Analyst, Business System Architect Team leads, developers. Company E team members. How the participants are chosen? Usually seniors are selected. Everyone who is and will be related to the development of the project. Experience of doing same sort of project. Everyone who is and will be related to the development of the project. What are the problems you are facing for not practicing architecture evaluation process? N/A N/A If any change required in the project or any enhanced requirement comes it is difficult to plug the functionality. If any change required in the project or any enhanced requirement comes it is difficult to plug the functionality. The Table 3 contains the converted high level status of the surveyed companies for the Step 1 of ATAM framework. Table 3 : Quantitative Status of ATAM Step 1(High Level) Question Do you practice architecture evaluation process? Do you have any process presentation meeting? who are the participants of the meeting How the participants are chosen Total earned point % of Total Company A 10 4 5 5 24 60% Company B 10 4 6 5 25 62.5% Company C 10 7 6 5 28 70% Company D 10 10 5 8 33 82.5% Company E 10 4 8 5 27 67.5% The Table 4 contains the converted detail level status of the surveyed companies for the Step 1 of ATAM framework. Table 4 : Quantitative Status of ATAM Step 1(Detail Level) Question Why do you practice architecture evaluation process? Do you have any Architecture process presentation meeting? Company A 7 6 Company B 8 7 Company C 7 8 Company D 7 6 Company E 8 7 59 Question If yes who are the participants of the meeting? How the participants are chosen? What are the problems you are facing for not practicing architecture evaluation process? Total earned points % of Total Company A 6 5 10 Company B 6 6 3 Company C 7 6 10 Company D 7 7 4 Company E 6 6 5 34 68% 30 60% 38 76% 31 62% 32 64% The points earned by the companies indicate that their practice level of ATAM step 1 is quiet high and the difference among the companies are not much. The Figure 2 shows the high level as well as detail level status of the ATAM step 1 to the companies. Figure 2 : Practice Level Status of ATAM Step 1 "Present the ATAM" Step 2: Present the Business Driver The Table 5 contains the high level narrative feedback of the surveyed companies for the Step 2 of ATAM framework.   Table 5 : High Level Feedback on Step 2 Question Do you consider business goal or business drivers in architecture meeting? Company A Project goals are defined. Company B Project goals are defined. Company C Project goals are defined. Company D Project goals are defined. Company E Yes, Project goals are defined. 60 Do you consider the constraints of the business? Yes we consider all the constraints. Yes. Yes we consider more or less all the constraints. Yes. But not always. Yes we consider more or less all the constraints Yes. But not always. Yes Yes Do you involve all the relevant stake holders like end-user, architect, developers, etc? Yes. But end users are participated through their project manager Yes The Table 6 contains the detail level feed of the surveyed companies for ATAM step 2 - presenting the business driver. Table 6 : Detail Level Feedback on Step 2 Question What goals are considered? Company A The business goals depend on project. The business process fitting and flexibility, least time response, customizability , etc. N/A Company B Security, Performance, Availability, Future enhancement. Company C The business goals depend on project. Like for sales application, we considered diversified sales handling, business process change accommodation, least time response to customer, etc. N/A Company D Security, Performance, Availability, Future enhancement. Company E Security, Performance, Availability, Future enhancement. Do you have any implications in the long run for not considering business goals? What constraints are considered? N/A N/A N/A The resource constraint and time constraint are usually considered. Do you have any implications in the long run for not considering the constraints? Who are the relevant stakeholders? N/A Business requirement clarification uncertainty, requirement complexity, lack of domain knowledge and technological uncertainty. N/A The resource constraint and time constraint are usually considered. N/A Business requirement clarification uncertainty, requirement complexity, lack of domain knowledge and technological uncertainty. N/A Business requirement clarification uncertainty, requirement complexity, lack of domain knowledge and technological uncertainty. N/A System Analyst, Architect, Project Manager, Team Lead and Developers. Business System Analyst, Business System Architect, Team Leads, developers. Do you have Not known. N/A We involve the N/A End users are participated through their project manager and the development team. N/A 61 Question any implications in the long run for not considering all the relevant stakeholders? Company A Company B Company C stakeholders based on their availability. These sometimes create some problems in the later part of the project, sometimes after delivery. Company D Company E Following table (Table 7) contains the high level quantitative earned points for five different companies for the ATAM Step 2 - presenting the business driver. Table 7 : Quantitative Status of ATAM Step 2 (High Level) Question Do you consider business goal or business drivers in architecture meeting? Do you consider the constraints of the business? Do you involve all the relevant stake holders like end-user, architect, developers, etc? Total earned points % of Total Company A 7 Company B 7 Company C 7 Company D 7 Company E 10 10 8 8 10 10 10 6 6 5 10 27 90% 21 70% 21 70% 22 73.33% 30 100% The Table 8 contains the detail level status of the surveyed companies for the Step 2 of ATAM framework. Table 8 : Quantitative Status of ATAM Step 2 (Detail Level) Question What goals are considered? Do you have any implications in the long run for not considering business goals? What constraints are considered? Do you have any implications in the long run for not considering the constraints? Who are the relevant stakeholders? Do you have any implications in the long run for not considering all the relevant stakeholders Total earned points % of Total Point Company A 5 10 Company B 6 10 Company C 6 10 Company D 6 10 Company E 7 10 5 10 7 10 5 10 7 10 7 10 0 0 5 10 0 5 5 10 6 10 30 50% 48 80% 36 60% 48 80% 50 83.33% 62 The Step 2 data indicates that the practice level is very high for both high level and detail level. This also indicates that the practice at higher level is higher than that of the detail level. It is also evident that the variation among the companies is also quiet high. This has been reflected in Figure 3. Figure 3 : Practice Level Status of ATAM step 2 “Present the business driver” Step 3: Present the Architecture to be evaluated The Table 9 contains the high level feedback of the surveyed companies for the Step 3 of ATAM framework. Table 9: High Level Feedback on Step 3 Question Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? Does the presentation include architecture diagram in sufficiently detail? Company A No. Company B Sometimes. Very high level. Yes. Very high level. Not always. Company C Yes, level. Yes. level. high Company D Yes. Company E Yes Yes. High Yes. Yes Yes all diagrams are included. Yes. Yes, usually. Does the presentation High level. High level. Top level and process level architecture with sufficient technical details. Yes Yes Yes 63 Question address technical constraints? Does the presentation address quality attributes requirement? Company A Company B Yes. Yes, quality attributes are defined. Yes. Some quality attributes are defined. Very limited. Yes. Company C Yes. Usually quality attributes are defined. Yes. Yes. These are discussed there. Yes. Company D Company E Does the presentation address quality attribute requirements like performance, reliability, etc.? Yes quietly. Yes Following table (Table 10) contains the detail level feedback of the surveyed companies for presenting the architecture to be evaluated step of ATAM (Step 3). Table 10 : Detail Level Feedback on Step 3 Question Do you see any implications for not defining architecture beforehand? Company A N/A Company B Problem faced on further enhancement of the project, adding pluggable features and communicating with other applications. Pipeline approach (small application), Ntier approach (mid to large application), client server approach (large application). Company C N/A Company D N/A Company E N/A What are the architectural approaches considered? We usually consider 3tiered architecture, web architecture, different patterns and frameworks, etc Do you have any implications for not considering the architectural approaches? What are the diagrams you produce? N/A N/A We usually consider n-tier web architecture, 3tiered enterprise architecture, Sometimes consider well known framework, sometimes we consider plugin, etc N/A Pipeline approach (small application), Ntier approach (mid to large application), client server approach (large application). ETL application architecture. Pipeline approach (small application), N-tier approach (mid to large application), client server approach (large application). Com+ application architecture. N/A N/A Layered diagram, Layer to Layer communication diagram, best case sequence diagram Top level and process level architecture with sufficient technical Top level and process level architecture with sufficient technical details. Layered 64 Question Company A Company B for communication with each component, worst case sequence diagram for communication with each component and deployment diagram. Company C Company D details. Layered diagram, Layer to Layer communication diagram, best case sequence diagram for communication with each component, worst case sequence diagram for communication with each component and deployment diagram. N/A Company E diagram, Layer to Layer communication diagram, best case sequence diagram for communication with each component, worst case sequence diagram for communication with each component and deployment diagram. Do you have any implications for not including architecture diagram in detail? What are the technical constraints are considered? N/A N/A N/A N/A We usually consider technological constraint, expertise level Technical constrain includes technical uncertainty and communication agreement between other applications. N/A We usually consider technological constraint, expertise level Do you see any implications for not addressing technical constraints? What are the quality attributes considered? N/A N/A Technical constrain includes technical uncertainty and communication agreement between other applications. N/A Technical constrain includes technical uncertainty and communication agreement between other applications. N/A We consider reliability, modifiability & performance Performance, Security, Availability, Plugability and Future enhancement. We usually consider reliability & performance Performance, Security, Availability, Plugability, reliability, interchangeabili ty, modifiability and Future enhancement Performance, Security, Availability, Plugability, reliability, modifiability and Future enhancement. The Table 11 contains the converted high level status of the surveyed companies for the Step 3 of ATAM framework. 65 Table 11 : High Level Quantitative Feedback on Step 3 Question Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? Does the presentation include architecture diagram in sufficiently detail? Does the presentation address technical constraints? Does the presentation address quality attributes requirement? Does the presentation address quality attribute requirements like performance, reliability, etc.? Total earned points % of Total Point Company A 0 10 10 10 10 10 Company B 4 7 5 8 5 10 Company C 6 7 7 7 7 10 Company D 10 10 9 10 7 7 Company E 10 10 10 10 10 10 50 83.33% 37 61.67% 44 73.33% 43 71.67% 60 100 Following table (Table 12) contains the detail level quantitative points of five different companies for ATAM Step 3. Table 12 : Detail Level Quantitative Feedback on Step 3 Question Do you see any implications for not defining architecture beforehand? What are the architectural approaches considered? Do you have any implications for not considering the architectural approaches? What are the diagrams you produce? Do you have any implications for not including architecture diagram in detail? What are the technical constraints are considered? Do you see any implications for not addressing technical constraints? What are the quality attributes are considered? Total earned points % of Total point Company A 10 5 10 Company B 5 6 10 Company C 10 7 10 Company D 10 7 10 Company E 10 6 10 0 10 5 10 0 10 7 10 5 10 5 10 5 55 61.11% 6 10 6 56 62.22% 5 10 5 57 63.33% 7 10 7 68 75.56% 6 10 6 63 70% The data indicates that the practice level at higher level is much higher than that of the detail level. The overall practice level of ATAM step 3 is high. Following graph represent the overall present the architecture to be evaluated status (Step 3) for different companies. 66 Figure 4 : Practice Level Status of ATAM step 3 "Present the Architecture to be evaluated" 5.2.2 Phase 2 - Investigation and Analysis Step 4: Identify the Architectural Approaches The Table 13 contains the high level status of the surveyed companies for the Step 4 of ATAM framework.   Table 13 : High Level feedback on Step 4 Question Does the architectural team explain the flow of control of the architecture? Does the architectural approach address the critical requirements? Does the architectural team provide explanation how the critical goal is met? Company A Every work flows are explained. Yes, it is addressed. Company B Every work flows are explained. Company C Every work flows are explained. Yes, it is addressed Company D Every work flows are explained. Yes. These are addressed through review processes Yes. They provide all explanation. Company E Yes. Yes, it is addressed. Yes Yes, every complex task are define in details breakdown approach. Yes, to some extent, every complex task is defined. Yes, to some extent. Yes. 67 Following table (Table 14) contains the detail feedback for identify the architectural approach (Step 4). Table 14 : Detail feedback on Step 4 Question Do you see any implications for not explaining the flow of control? Company A Not known. Company B N/A Company C This creates sometimes misunderstandi ng among the stakeholders and quality suffers in the end. The business functions, response time, scaleability, performance, etc. Company D N/A Company E N/A What are the critical requirements are considered? The business functions, response time, scaleability, modifiability, performance, reusability, etc. Does the architectural team provide explanation how the critical goal is met? Usually not. Critical requirements include complex business process and Algorithmic complexity for the business process. Critical goals are considered by the complex business requirement and algorithmic complexity for the business goal. Each complex requirement are break down by details analysis. Review process is used to determine a critical requirement includes complex business process and Algorithmic complexity for the business process. Review process is used to determine a critical requirement includes complex business process and Algorithmic complexity for the business process. Sometimes, high level Critical goals are considered by the complex business requirement and algorithmic complexity and system requirement for the business goal. Critical goals are considered by the complex business requirement and algorithmic complexity and system requirement for the business goal. Do you have any implications in the long run for not considering the critical requirements? N/A N/A Each complex requirement are break down by details analysis. Each complex requirement are break down by details analysis. The Table 15 contains the converted high level status of the surveyed companies for the Step 4 of ATAM framework. Table 15 : High Level Quantitative feedback on Step 4 Question Does the architectural team explain the flow of control of the architecture? Company A 8 Company B 8 Company C 8 Company D 8 Company E 10 68 Does the architectural approach address the critical requirements? Does the architectural team provide explanation how the critical goal is met? Total earned point % of Total Point 9 9 9 6 10 9 26 86.67% 8 25 83.33% 7 24 80% 9 23 76.67% 10 30 100% Following table (Table 16) contains the detail level quantitative points of five different companies for ATAM Step 4. Table 16 : Detail Quantitative feedback on Step 4 Question Do you see any implications for not explaining the flow of control? What are the critical requirements are considered? Does the architectural team provide explanation how the critical goal is met? Do you have any implications in the long run for not considering the critical requirements? Total earned point % of Total point Company A 0 5 0 Company B 10 6 5 Company C 3 5 3 Company D 10 7 5 Company E 10 7 5 10 5 10 5 5 15 37.5% 26 65% 21 52.5% 27 67.5% 27 67.5% The data indicates that the practice level at higher level is very higher than that of detail level. That could happen due to lack of understanding of detail practices. It is also evident that the variation among the companies is significantly high. This indication has been shown in the following figure (Figure 5). 69 Figure 5 : Practice Level Status of ATAM Step 4 "Identify the architectural approaches" Step 5: Generate the Quality Attribute Utility Tree. The Table 17 contains the high level status of the surveyed companies for the Step 5 of ATAM framework. Table 17 : High Level feedback on Step 5 Question Are the important quality attributes goals clearly identified? Company A Yes. Company B Not always. Rarely Yes. Company C Not always. Sometimes Yes. Company D Yes. These are clearly defined from the Mother Company(Laval, Canada) No. No. Company E Yes. Is the utility tree made? Does the utility tree specify the system goals clearly? Are the scenarios are clearly described? Are the quality attributes are prioritized? Are all the stake holders scenarios considered? No utility tree made. We don’t maintain this. Yes, every steps are clearly defined. Yes. No utility tree made. We don’t maintain this. No, little bit. Sometimes. Sometimes. No No More or less. Yes, every steps are clearly defined from mother These are defined from the mother company. Yes. Yes Yes. Not always. Yes. Every work flows are considered. Not all quality attributes are Yes. Not always. Yes. Every work flows are considered. Not all quality Yes, Not always. Yes. Are all the quality attributes like the Yes. Every work flows are considered. Not all quality Try to be considered. Not all 70 Question security, performance, availability, reliability, modifiability, portability, functionality, variability, conceptual integrity, etc. quality attribute considered? If utility tree is not practiced, is there other way in practice? Company A attributes are considered. Company B considered. Company C attributes are considered. Company D Company E No idea. Utility tree is not so helpful No idea. We don’t practice. We do similar type of works little bit. No, other process is not maintained. No Following table (Table 18) contains the feedback of the surveyed companies for generate the quality attribute utility tree (Step 5). Table 18 : Detail feedback on Step 5 Question What are the important qualities attributes goals? Company A Performance, Scalability, Security, Availability, Modifiablity Company B Important quality attributes are as follows: security, performance, availability, reliability, modifiability, portability. N/A Company C Performance, Scalability, Security, Availability Company D Important quality attributes are as follows: security, performance, availability, reliability, modifiability, portability. N/A Company E Important quality attributes are as follows: security, performance, availability, reliability, modifiability, portability. What are the implications for not identifying these? What are the advantages for this? What are the implications of not practicing this? N/A N/A N/A Not known. Quality attributes are not traceable at the design time. After started working we face the problem which can be reduced by this utility tree. Suffer in the long run. Some important quality attributes not considered. Quality attributes are not traceable at the design time. After started working we face the problem which can be reduced by this utility tree. Quality attributes are not traceable at the design time. After started working we face the problem which can be reduced by this utility tree. What are the system goals considered for the utility tree? What are the implications of not Quality requirements are not Sometimes we have to redesign the system after Quality requirements are not clearly Sometimes we have to redesign the system after Sometimes we have to redesign the system after 71 Question practicing this? Company A understood and met.. Company B work started. Company C understood and ensured. Company D work started. This creates additional rework which increases development time. Scenarios are described by best case, worst case and medium case sequence diagram of architecture. N/A Company E work started. This creates additional rework which increases development time. What are the scenarios to be considered? Usually use case scenarios are considered. Scenarios are described by best case, worst case and medium case sequence diagram of architecture. N/A What are the implications of not practicing this? What are the criteria to prioritize quality attributes? N/A Usually use case scenarios are considered. Sometimes growth scenarios are also considered. N/A Scenarios are described by best case, worst case and medium case sequence diagram of architecture. N/A Depends on the project and customer desire. Depends on the project business requirement we prioritize quality attributes. N/A Depends on the business need and customer desire. Depends on the project business requirement we prioritize quality attributes. N/A Depends on the project security. What are the implications for not prioritizing quality attributes? What are the common scenarios considered? N/A N/A N/A What are the implications for not considering all the scenarios? Suffers in the latter stage of the project. Stakeholders related to all business execution process are considered. N/A What are the attributes considered? Usually security, performance, functionality and reliability are considered. Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. Integration with What are the Not known. Suffers in the latter stage of the project and sometimes during production. Quality attributes depends on the project. Usually security, performance, functionality and reliability are considered. The Stakeholders related to all business execution process are considered. N/A Stakeholders related to all business execution process are considered. N/A Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. Integration Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. Integration with 72 Question implications for missing attributes? Company A Company B other application sometimes creates problem. Not known. Company C reusability and extensibility suffers. Company D with other application sometimes creates problem. Not known. Company E other application sometimes creates problem. What are the implications for not considering the quality attributes? Give the descriptions of the practices. Not known. Not known. Not known. What are the implications for practicing nothing like utility tree? Project will suffer in the latter stage. Architecture traceability is not possible and it creates rework after creating architecture. Determine the quality attributes, find out the importance to the project, and prioritize accordingly. Not known. Architecture traceability is not possible and it creates rework after creating architecture. Architecture traceability is not possible and it creates rework after creating architecture. The Table 19 contains the converted high level status of the surveyed companies for the Step 5 of ATAM framework. Table 19 : High Level Quantitative feedback on Step 5 Question Are the important quality attributes goals clearly identified? Is the utility tree made? Does the utility tree specify the system goals clearly? Are the scenarios are clearly described? Are the quality attributes are prioritized? Are all the stake holders scenarios considered? Are all the quality attributes like the security, performance, availability, reliability, modifiability, portability, functionality, variability, conceptual integrity, etc. quality attribute considered? If utility tree is not practiced, is there other way in practice? Total earned points % of Total Company A 10 0 0 6 10 10 5 Company B 5 0 0 3 5 10 5 Company C 6 5 4 5 7 10 5 Company D 10 0 0 10 8 10 3 Company E 10 0 0 10 6 10 5 0 41 51.5% 0 28 35% 3 45 56.25% 0 41 51.255 0 41 51.25% 73 Following table (Table 20) contains the detail quantitative points of five different companies for the ATAM Step 5. Table 20 : Detail Quantitative feedback on Step 5 Question What are the important qualities attributes goals? What are the implications for not identifying these? What are the advantages for this? What are the implications of not practicing this? What are the system goals considered for the utility tree? What are the implications of not practicing this? What are the scenarios to be considered? What are the implications of not practicing this? What are the criteria to prioritize quality attributes? What are the implications for not prioritizing quality attributes? What are the common scenarios considered? What are the implications for not considering all the scenarios? What are the attributes considered? What are the implications for missing attributes? What are the implications for not considering the quality attributes? Give the descriptions of the practices. What are the implications for practicing nothing like utility tree? Total points earned % of Total Company A 5 10 0 0 0 2 5 10 5 0 0 3 5 0 0 0 5 47 27.65% Company B 7 10 0 5 0 2 5 10 7 0 5 10 6 0 0 0 7 74 43.53% Company C 5 10 0 3 0 3 6 10 5 0 0 5 7 0 0 5 0 59 34.71% Company D 7 10 0 5 0 5 6 10 7 0 5 10 6 0 0 0 7 78 45.88% Company E 7 10 0 5 0 5 7 10 5 0 5 10 6 0 0 0 7 77 45.29% The practice level of ATAM Step 5 is significantly low. This is much lower in detail level. This indicates that companies usually do not use step. The variation among the companies is high. The following figure (Figure 6) shows the practice level status of ATAM Step 5 better. 74 Figure 6 : Practice Level Status of ATAM Step 5 "Generate Quality Attributes" Step 6: Analyze the Architectural Approaches The Table 21 contains the high level status of the surveyed companies for the Step 6 of ATAM framework Table 21 : High Level feedback on Step 6 Question How investigation done to define architectural approach? Company A We identify all the functionality and allocate the functionality in each step of investigation. Risk items are identified from the requirement and technological challenges. Company B No defined approach. Company C No defined approach. Company D No investigatio n done. Company E No defined way How risk and non risks are identified? Risk items are identified from the requirement and technologica l challenges. Risk items are identified from the requirement and technologica l challenges. By team meeting. By analyzing requiremen t Following table (Table 22) contains answer for the questionnaires for analyze the architectural approaches (Step 6). Table 22 : Detail feedback on Step 6 Question How investigation Company A Analyzed how Company B We identify all Company C Analyzed Company D We identify all the Company E 75 Question done to define architectural approach? Company A the architectural approach meet the quality attributes. N/A Company B the functionality and allocate the functionality in each step of investigation. N/A Company C how the architectural approach meet the quality attributes. N/A Company D functionality and allocate the functionality in each step of investigation. N/A Company E What are the implications of not doing so? How risk and non risks are identified? Usually find out when and in situation the system can fail. N/A What are the implications of not identifying risks and non-risks? Risk items are identified from the requirement and technological challenges. N/A Usually find out when and in situation the system can fail. N/A Risk items are identified from the requirement and technological challenges. N/A Functionalit y identificatio n cannot be done properly. Risk items are identified By team meeting. N/A The Table 23 contains the converted high level status of the surveyed companies for the Step 6 of ATAM framework. Table 23 : High Level Quantitative feedback on Step 6 Question How investigation done to define architectural approach? How risk and non risks are identified? Total earned points % of Total Point Company A 7 6 13 65% Company B 5 6 11 55% Company C 5 6 11 55% Company D 0 5 5 25% Company E 3 4 7 35% Following table (Table 24) contains the detail quantitative points for five different companies for ATAM Step 6. Table 24 : Detail Quantitative feedback on Step 6 Question How investigation done to define architectural approach? What are the implications of not doing so? How risk and non risks are identified? What are the implications of not identifying risks and non-risks? Total earned points % of Total Point Company A 5 10 5 10 30 75% Company B 7 10 6 10 33 82.5% Company C 6 10 5 10 31 77.5% Company D 7 10 6 10 33 82.5% Company E 0 5 6 10 21 52.5% 76 The data indicates the practice level in detail level is higher than that of higher level. This is very unusual. This probably happened, as this is not the full step. This is only the header of the step. This is also significant variation of the practice level among the companies. Following graph (Figure 7) represents the detail Analyze the architectural approaches status (Step 6) for different companies. Figure 7 : Practice Level Status of ATAM Step 6 "Analyze the architectural approaches" Step 6.1: Investigation of Architectural Approach The Table 25 contains the high level status of the surveyed companies for the Step 6.1 of ATAM framework. Table 25 : High Level feedback on Step 6.1 Question Does the architectural approach support the quality attribute? Company A Yes, we define the quality attributes. Company B Yes, sometimes, we define the quality attributes. We didn’t think of it. Company C Yes, sometimes, we define the quality attributes. We consider variability. Company D Yes. Company E Yes. How does architectural approach support variability? Our approaches are variable. We design taking consideration of every approach Through review. Review. 77 Question How does architectural approach support modifiability? Company A Our approaches are easy to modify as we design taking consideration of this We test our approach to support reliability. We take consideration of integration at the time of design depending on the project category. Company B We consider modifiability if required. Company C We consider modifiability if required. Company D The approach supports modifiabilit y. We approach supports reliability. We take consideratio n of integration at the time of design depending on the project category. Every functionaliti es are distinctly described. Company E At the time of design How does architectural approach support reliability? How does architectural approach support conceptual integrity? We test our approach to support reliability. Considered at the time of design depending on the project category. We test our approach to support reliability. Considered at the time of design depending on the project category. At the time of design At the time of design How does architectural approach support functionality? Every functionalities are distinctly described. Major functionalities are distinctly described. Major functionalitie s are distinctly described. At the time of design Following table contains answer for the questionnaires for investigation of architectural approach. Table 26 : Detail feedback on Step 6.1 Question What are the quality attributes considered? Company A Usually security, scalability, reliability and performance are considered. Company B Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. N/A Company C Usually security, scalability, reliability and performance are considered. Company D Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. N/A Company E Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. N/A What are the implications of not practicing this? What are the processes to identify the variability? What are the implications of not practicing this? What are the processes to N/A N/A Not known Not known Not known Through review variability can be identified. N/A Through review variability can be identified. N/A Not very evident. Future enhancement creates problem. Processes are as follows: Not very evident. Checking the ability to be Checking the ability to be Processes are as follows: Processes are as follows: 78 Question identify the modifiability? Company A modified, changing the different business process and functionality N/A Company B Requirement uncertainty, Functionality, Future enhancement, Plugability. N/A Company C modified, changing the different business process and functionality N/A Company D Requirement uncertainty, Functionality, Future enhancement, Plugability. N/A Company E Requirement uncertainty, Functionality, Future enhancement, Plugability. N/A What are the implications of not practicing this? What are the processes to identify the reliability? Ensures the application can tolerate failure. What are the implications of not practicing this? What are the processes to identify the conceptual integrity? N/A Processes are as follows: Test each flow of execution of architecture, Test plugability of components. N/A Ensures the application can tolerate failure. N/A Processes are as follows: Test each flow of execution of architecture, Test plugability of components. N/A Processes are as follows: Test each flow of execution of architecture, Test plugability of components. N/A What are the implications of not practicing this? What are the processes to identify the functionality of project? Not evident. Considered at the time of design depending on the project category and business requirement. N/A Not evident. Considered at the time of design depending on the project category and business requirement. N/A Considered at the time of design depending on the project category and business requirement. N/A Checking the major functionality against the architecture. What are the implications of not practicing this? N/A Major functionalities are distinctly described from the details analysis of requirement. N/A Checking the major functionality against the architecture. N/A Major functionalities are distinctly described from the details analysis of requirement. N/A Major functionalities are distinctly described from the details analysis of requirement. N/A The Table 27 contains the converted high level status of the surveyed companies for the Step 6.1 of ATAM framework. Table 27 : High Level feedback on Step 6.1 Question Does the architectural approach support the quality attribute? How does architectural approach support variability? Company A 8 6 Company B 6 3 Company C 6 5 Company D 10 3 Company E 10 3 79 Question How does architectural approach support modifiability? How does architectural approach support reliability? How does architectural approach support conceptual integrity? How does architectural approach support functionality? Total earned points % of Total Points Company A 5 7 7 10 43 71.67% Company B 5 7 8 8 37 61.67% Company C 5 7 7 8 38 63.33% Company D 6 6 8 9 42 70% Company E 6 8 8 9 44 73.33% Following table (Table 28) contains the detail quantitative points of five different companies for ATAM Step 6.1. Table 28 : Detail feedback on Step 6.1 Question What are the quality attributes considered? What are the implications of not practicing this? What are the processes to identify the variability? What are the implications of not practicing this? What are the processes to identify the modifiability? What are the implications of not practicing this? What are the processes to identify the reliability? What are the implications of not practicing this? What are the processes to identify the conceptual integrity? What are the implications of not practicing this? What are the processes to identify the functionality of project? What are the implications of not practicing this? Total earned points % of Total Company A 5 10 0 2 5 10 5 10 0 0 5 10 62 51.67% Company B 7 10 0 5 6 10 7 10 3 10 6 10 84 70% Company C 5 10 0 2 5 10 5 10 0 0 5 10 62 51.67% Company D 7 10 3 0 6 10 7 10 3 10 7 10 83 69.17% Company E 7 10 3 0 6 10 7 10 3 10 7 10 83 69.17% The average practice level for ATAM step 6.1 is not very high. The data indicates that the practice level is higher in the high level practice than that of detail level practice. The variation among the companies is not significantly high. Following graph (Figure 80 8) represents the overall investigation of architectural approach (Step 6.1) status for different companies. Figure 8 : Practice Level Status of ATAM Step 6.1 “Investigation on architectural approach” Step 6.2: Creation of Analysis Questions The Table 29 contains the high level status of the surveyed companies for the Step  6.2 of ATAM framework.  Table 29 : High Level feedback on Step 6.2 Question Do you create analysis questions? Can the components of the architecture reused for future projects? Can the framework be expanded in the future to accommodate a new application or new components? Company A Yes. Yes, Utility, Security components are reusable. Yes, But depends on the project. Company B Usually No. Yes, Sometimes. Company C Sometimes, Yes. . Yes, Sometimes. Company D No. Yes, Try to design that can be reused. Yes, we consider that but Project types doesn’t match with other project types. Yes Company E Yes. Yes. Yes, sometimes, depends on the project. . Yes, sometimes, depends on the project. Yes. Depending on project Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Yes Yes Yes Yes Yes. Yes Yes Yes Yes 81 Question Can any new applicationspecific functionality be added to the architecture? Can the system be modified in short time and cost-effective manner? Company A Yes, but depends on the project. Yes, but depends on the project. Company B Usually Yes, but depends on the project. Not always. Sometimes Yes, but depends on the project. Yes, Integration test done always. Company C Usually Yes, but depends on the project. Not always. Sometimes Yes, but depends on the project. Yes, Integration test done always. Company D Yes, but depends on the project. Yes, but depends on the project. Company E Yes. Yes. Do the components interact properly? Yes, Integration test done always. Yes, Thorough QA done for that. Yes. Following table (Table 30) contains the detail feedback of creation of analysis questions (Step 6.2) for the different five companies. Table 30 : Detail feedback on Step 6.2 Question What are the steps followed to create analysis questions? Company A First quality attributes are identified. Then questions are prepared. Company B Steps are requirement analyze, requirement complexity consideration, requirement uncertainty and technical uncertainty. Complex requirement, Complex business process. N/A Company C First quality attributes are identified. Then questions are prepared. Company D Company E What are covered in analysis questions? What are the implications of not practicing this? What are the design level considerations for component reusability? Depends on the project and identified quality attributes. N/A Depends on the project and identified quality attributes. N/A What are the implications of not practicing this? What are the steps considered for the framework expansion? Time is not reduced. Design level considerations are making the component more specific and independent to work. N/A Repeated projects are not lowered. Time is not reduced. Requirement complexity and uncertainty cant be identified properly. Design level considerations are making the component more specific and independent to work. N/A Requirement complexity and uncertainty cant be identified properly. Design level considerations are making the component more specific and independent to work. N/A Each component is made to work as independently without any dependency. Each component is made to work as independently without any dependency. Each component is made to work as independently without any dependency. 82 Question What are the implications of not practicing this? How the validation considered? Company A Inability to expand according to clients need Depends on project. If high reliable project and enforce the validation. N/A Company B N/A Company C Inability to expand according to clients need Depends on project. If high reliable project and enforce the validation. N/A Company D N/A Company E N/A Validation considered as from the business requirement. N/A Validation considered as from the business requirement. N/A Validation considered as from the business requirement. N/A What are the implications of not practicing this? What are the steps followed to identify the architectural behavior? What are the implications of not practicing this? How the decision taken at the time of design to add application specific functionality? Gets unreliable. Steps are requirement complexity, requirement uncertainty and future enhancement. N/A Gets unreliable. Steps are requirement complexity, requirement uncertainty and future enhancement. N/A Steps are requirement complexity, requirement uncertainty and future enhancement. N/A Architecture is designed to accommodate the new functionality. The modifiability and extensibility is considered. What are the implications of not practicing this? How the time and cost calculated for change management? N/A Each component is created to work independently without any dependency, so any other application specific functionality can be added easily. N/A Architecture is designed to accommodate the new functionality. The modifiability and extensibility is considered. N/A Each component is created to work independently without any dependency, so any other application specific functionality can be added easily. N/A Each component is created to work independently without any dependency, so any other application specific functionality can be added easily. N/A What are the implications of not practicing this? Not known. Sometimes we need to reengineer the application architecture which increases time and Sometimes fails to respond to client request. Creates bad impression. Time and cost consideration for the change management considered during the design time. So any new change issue can’t do so much harm to the system. N/A Time and cost consideration for the change management considered during the design time. So any new change issue can’t do so much harm to the system. N/A 83 Question Company A Company B decreases quality. Component interaction tested by unit testing. N/A Company C Company D Company E How the component interaction tested? What are the implications of not practicing this? Component interaction tested by unit testing. The changes do not take effect as expected. N/A Component interaction tested by unit testing. N/A Not known. The Table 31 contains the converted high level status of the surveyed companies for the Step 6.2 of ATAM framework. Table 31 : High Level Quantitative feedback on Step 6.2 Question Do you create analysis questions? Can the components of the architecture reused for future projects? Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Can any new application-specific functionality be added to the architecture? Can the system be modified in short time and costeffective manner? Do the components interact properly? Total earned points % of Total Point Company A 10 7 6 Company B 3 5 7 Company C 6 5 7 Company D 0 6 5 Company E 10 10 6 10 10 5 6 10 62 77.5% 10 10 6 6 10 57 71.25% 10 10 6 6 10 60 75% 10 10 6 6 9 52 65% 10 10 10 10 10 76 95% Following table (Table 32) contains the detail quantitative points for five different companies for ATAM Step 6.2. Table 32 : Detail Quantitative feedback on Step 6.2 Question What are the steps followed to create analysis questions? What are covered in analysis questions? What are the implications of not practicing this? What are the design level considerations for component reusability? What are the implications of not practicing this? What are the steps considered for the framework expansion? Company A 5 4 10 0 2 0 Company B 7 4 10 6 10 5 Company C 5 4 10 0 3 0 Company D 0 0 3 5 10 6 Company E 0 0 4 5 10 6 84 Question What are the implications of not practicing this? How the validation considered? What are the implications of not practicing this? What are the steps followed to identify the architectural behavior? What are the implications of not practicing this? How the decision taken at the time of design to add application specific functionality? What are the implications of not practicing this? How the time and cost calculated for change management? What are the implications of not practicing this? How the component interaction tested? What are the implications of not practicing this? Total earned points % of earned points Company A 5 6 10 0 5 6 Company B 10 5 10 5 10 7 Company C 5 6 10 0 5 6 Company D 10 5 10 5 10 6 Company E 10 6 10 6 10 7 10 0 0 0 0 63 37.06% 10 0 6 5 10 110 64.71% 10 0 5 0 5 74 43.53% 10 5 10 5 10 110 64.71% 10 5 10 5 10 114 67.09% It is evident that practice level variation in higher level and detail level is very high. In the higher level it is seen that the practice level is very high. But, in the detail level the practice is significantly lower than that of higher level. The variation among the companies is also remarkably high. Following graph (Figure 9) represents the overall creation of analysis questions (Step 6.2) status for different companies. 85 Figure 9 : Practice Level Status of ATAM Step 6.2 "Creation of analysis question" Step 6.3: Answers to the Analysis Questions The Table 33 contains the high level status of the surveyed companies for the Step 6.3 of ATAM framework. Table 33: High Level feedback on Step 6.3 Question Are the answer and explanations to the questions analyzed? Company A Yes, we consider every issue. Company B Yes, we consider every issue. Company C Yes, we consider every issue. Company D . Yes, we consider every issue. Company E Yes. Following table (Table 34) contains the detail feedback of the companies for answers to the analysis questions step (Step 6.3). Table 34 : Detail feedback on Step 6.3 Question How the answer and explanations to the questions analyzed? Company A The answers are discussed in the meeting from different angle. Company B Answer and explanations to the questions analyzed by the details breakdown of requirement including complex and uncertain requirement, N/A Company C The answers are discussed in the meeting from different angle. Company D Answer and explanations to the questions analyzed by the details breakdown of requirement including complex and uncertain requirement, N/A Company E Answer and explanations to the questions analyzed by the details breakdown of requirement including complex and uncertain requirement, N/A What are the N/A N/A 86 Question implications of not practicing this? Company A Company B Company C Company D Company E The Table 35 contains the converted high level status of the surveyed companies for the Step 6.3 of ATAM framework. Table 35 : High Level Quantitative feedback on Step 6.3 Question Are the answer and explanations to the questions analyzed? Total earned points % of Total Points Company A 10 10 100% Company B 10 10 100% Company C 10 10 100% Company D 10 10 100% Company E 10 10 100% Following table contains the quantitative points for five different companies. It is evident that all the companies follow this step. Table 36 : Detail Quantitative feedback on Step 6.3 Question How the answer and explanations to the questions analyzed? What are the implications of not practicing this? Total earned points % of Total Point Company A 5 Company B 7 Company C 5 Company D 7 Company E 6 10 10 10 10 10 15 75% 17 85% 15 75% 17 85% 18 90% Following graph (Figure 10) represents the answers to the analysis questions (Step 6.3) status for different companies. 87 Figure 10 : Practice Level Status of ATAM Step 6.3 “Answer to the analysis questions” Step 6.4: Find the risks, non-risks, sensitivity points, trade-off points. The Table 37 contains the high level status of the surveyed companies for the Step 6.4 of ATAM framework. Table 37 : High Level feedback on Step 6.4 Question Are the risks and non-risks identified properly? Are the sensitivity points identified properly? Are the trade off points identified properly? Company A Risks are identified properly. Not in defined process. Not in define process. Company B Not always risks are identified properly. Company C Yes, usually risks are identified. Not really. Not in defined process. Not in define process. Company D Risks are identified properly. Not in defined process. Not in define process. Company E Yes. Not really. Not in defined process. No. Not in define process. No. Following table (Table 38) contains answer for the detail level questionnaires for finding the risk, non-risks, sensitivity points and trade-off points (Step 6.4). Table 38 : Detail feedback on Step 6.4 Question What are the processes to Company A Usually risks are identified. Company B The processes to identify the Company C Usually risks are identified. Company D The processes to identify the Company E The processes to identify the 88 Question identify the risk and non risks? Company A But no defined process is in place Company B risk and non risks are from the requirement complexity, uncertainty and technological uncertainty. Company C But no defined process is in place Company D risk and non risks are from the requirement complexity, uncertainty and technological uncertainty. Company E risk and non risks are from the requirement complexity, uncertainty and technological uncertainty. What are the common risks and non-risks? Common risks are unavailability of client resource, failure of meeting the performance goals, quality requirements N/A What are the implications of not practicing this? What are the processes to identify the sensitivity points? What are the common sensitivity points? What are the implications of not practicing this? Common risks are unavailability of client resource, failure of meeting the performance goals, quality requirements N/A Issues important to the customer can be missed. We cannot verify the architectural approaches are how much feasible for the application. Issues important to the customer can be missed. We cannot verify the architectural approaches are how much feasible for the application. We cannot verify the architectural approaches are how much feasible for the application. What are the processes to identify the trade off points? What are the common tradeoff points? What are the implications of not practicing this? Issues important to the customer can be missed. We cannot verify the architectural approaches are how much feasible for the application. Issues important to the customer can be missed. We cannot verify the architectural approaches are how much feasible for the application. We cannot verify the architectural approaches are how much feasible for the application. 89 The Table 39 contains the converted high level status of the surveyed companies for the Step 6.4 of ATAM framework. Table 39 : High Level Quantitative feedback on Step 6.4 Question Are the risks and non-risks identified properly? Are the sensitivity points identified properly? Are the trade off points identified properly? Total earned points % of Total Point Company A 6 5 3 14 46.67% Company B 3 2 3 8 26.67% Company C 7 2 3. 12 40% Company D 8 3 3 14 46.67% Company E 10 0 0 10 33.33% Following table (Table 40) contains the detail quantitative points for five different companies for Step 6.4. Table 40 : Detail Quantitative feedback on Step 6.4 Question What are the processes to identify the risk and non risks? What are the common risks and non-risks? What are the implications of not practicing this? What are the processes to identify the sensitivity points? What are the common sensitivity points? What are the implications of not practicing this? What are the processes to identify the trade off points? What are the common trade-off points? What are the implications of not practicing this? Total earned points % of Total Point Company A 5 6 10 0 0 4 0 0 3 28 31.11% Company B 6 0 0 0 0 5 0 0 5 16 17.78% Company C 6 6 10 0 0 4 0 0 4 30 33.33% Company D 7 0 0 0 0 6 0 0 7 20 22.22% Company E 7 0 0 0 0 6 0 0 6 19 21.11% The company practice data indicates that the practice level of ATAM Step 6.4 is significantly low. This is much lower for the detail level practice. Usually the companies do not follow this step. The Figure 11 presents the practice status of the surveyed companies for the Step 6.4 of ATAM framework. 90 Figure 11 : Practice Level Status of ATAM Step 6.4 "Find the risks, non-risks, sensitivity points, tradeoff points" 5.2.3 Phase 3 – Testing Step 7: Brainstorm and Prioritize Scenarios The Table 41 contains the high level status of the surveyed companies for the Step 7 of ATAM framework. Table 41 : High Level feedback on Step 7 Question Does all the stake holders involve in brainstorm? Company A Yes. Everyone related to development cooperate this. Yes. Company B No. Some selected members related to development cooperate this. Yes. More or less. Company C Usually Yes.. Company D Yes. Defined by Mother Company. Yes. Done by Mother Company. Yes Yes Yes Yes Company E No Are all scenarios collected after brainstorm? Are use case scenarios considered? Are growth scenarios considered? Are exploratory scenarios considered? Are the scenarios are prioritized? Are the voting Yes. More or less. Yes Sometimes, Yes Sometimes, Yes Yes, to some extent. No voting Yes. Yes. Yes. Yes. Yes. Yes Sometimes, Yes Rarely Yes. Yes, to some extent. No voting process Yes. Yes. Not every time. Yes. No voting process No voting No. 91 Question process used for prioritizing scenarios? Company A is used. Naturally more complex task considered first. Company B is used. Company C process is used. Company D process is used. Naturally more demandable task considered first. Company E Following table (Table 42) contains detail answer for the questionnaires for brainstorm and prioritizes scenarios (Step 7) of five companies. Table 42 : Detail feedback on Step 7 Question What are the criteria to select stakeholders? What are the implications of not practicing this? Company A Company B Company C Company D All related members are selected. Company E Defined by Mother Company. Some scenarios can be missed. Give some example of Scenarios? What are the implications of not practicing this? What are the cases of scenarios considered? What are the implications of not practicing this? What are the steps to consider growth Some scenarios can be missed which can be evident during production. Major application functions are considered. N/A All members are not involved. Only development related members are involved so sometimes we cannot identify the risk and uncertainty. Examples like: Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. N/A Some scenarios can be missed. Every high level case and Complex cases are considered. N/A Some scenarios can be missed which can be evident during production. Major application functions are considered. N/A Examples like: Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. N/A Examples like: Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. N/A Every high level case and Complex cases are considered. N/A Every high level case and Complex cases are considered. N/A Usually considered to some extent. We analyze the requirement and clarify with the Usually considered to some extent. We analyze the requirement and clarify with the We analyze the requirement and clarify with the 92 Question scenarios? Company A How much growth can happen in a certain period of time in terms of users, business volume, storage, etc. N/A Company B client about the further enhancement of the system. Also project growths corresponding to every business workflow are considered. N/A Company C How much growth can happen in a certain period of time in terms of users, business volume, storage, etc. N/A Company D client about the further enhancement of the system. Also project growths corresponding to every business workflow are considered. N/A Company E client about the further enhancement of the system. Also project growths corresponding to every business workflow are considered. N/A What are the implications of not practicing this? What are the steps to consider exploratory scenarios? What are the implications of not practicing this? How the scenarios are prioritized? Not very evident. Steps include the database system portability, Performance, Availability and security. N/A Not very evident. Steps include the database system portability, Performance, Availability and security. N/A Steps include the database system portability, Performance, Availability and security. N/A What are the implications of not practicing this? What is the voting process for prioritizing scenarios? What are the implications of not using voting process? The scenarios are prioritized based on application functionality and stability requirement. N/A Corresponding to the complexity, functionality and business driver scenarios are prioritized. N/A The scenarios are prioritized based on application functionality and stability requirement. N/A Corresponding to the complexity, functionality and business driver scenarios are prioritized. N/A Corresponding to the complexity, functionality and business driver scenarios are prioritized. N/A The priority list may not reflect the actual need. Sometimes we don’t understand at what point we should start working. If the voting process introduce we can decide from where we have to start working. The priority list may not reflect the actual need. Sometimes we don’t understand at what point we should start working. If the voting process introduce we can decide from where we have to start working. Sometimes we don’t understand at what point we should start working. If the voting process introduce we can decide from where we have to start working. The Table 43 contains the converted high level status of the surveyed companies for the Step 7 of ATAM framework. Table 43 : High Level Quantitative feedback on Step 7 Question Does all the stake holders involve in Company A 8 Company B 6 Company C 7 Company D 8 Company E 5 93 Question brainstorm? Are all scenarios collected after brainstorm? Are use case scenarios considered? Are growth scenarios considered? Are exploratory scenarios considered? Are the scenarios are prioritized? Are the voting process used for prioritizing scenarios? Total earned points % of Total Point Company A 10 10 10 10 10 3 61 87.14% Company B 8 10 7 4 7 0 42 60% Company C 8 10 7 6 7 0 45 64.29% Company D Company E 8 10 10 10 10 4 60 85.71% 10 10 10 5 10 0 50 71.43% Following table (Table 44) contains the detail quantitative points for five different companies. Table 44 : Detail Quantitative feedback on Step 7 Question What are the criteria to select stakeholders? What are the implications of not practicing this? Give some example of Scenarios? What are the implications of not practicing this? What are the cases of scenarios considered? What are the implications of not practicing this? What are the steps to consider growth scenarios? What are the implications of not practicing this? What are the steps to consider exploratory scenarios? What are the implications of not practicing this? How the scenarios are prioritized? What are the implications of not practicing this? What is the voting process for prioritizing scenarios? What are the implications of not using voting process? Total earned points % of Total Point Company A 0 5 0 5 5 10 7 10 0 3 5 10 0 3 63 45% Company B 0 7 5 10 7 10 8 10 5 10 6 10 0 6 94 6714% Company C 0 5 0 5 5 10 8 10 0 3 5 10 0 3 64 45.71% Company D 5 10 6 10 7 10 7 10 5 10 6 10 0 5 101 72.14% Company E 4 10 6 10 8 10 8 10 5 10 7 10 0 6 104 74.29% The variation among the high level and detail level practice is significantly high. The high level practice is higher than that of detail level practice. The variation among the 94 companies is also high. The Figure 12 presents the high level status of the surveyed companies for the Step 7 of ATAM framework. Figure 12 : Practice Level Status of ATAM Step “Brainstorm and prioritize scenarios” Step 8: Analyze the Architectural Approaches The Table 45 contains the high level status of the surveyed companies for the Step 8 of ATAM framework. Table 45 : High Level feedback on Step 8 Question Are the new quality attributes considered found in the brainstorm session? Are the new scenarios considered Are the new risks, nonrisks, sensitivity points and trade off points included? Company A Yes, Considered and analyzed properly. Yes. Risk points are included. Company B Yes, Considered and analyzed properly. Yes. Risk points are included, if any. Company C Yes, Considered and analyzed properly. Yes. Risk points are included, if any. Company D Yes Company E Yes. Yes Risk points are included. Yes Risks are included. Following table (Table 46) contains detail answer for the questionnaires for analyze the architectural approach (Step 8) of five companies. 95 Table 46 : Detail feedback on Step 8 Question How do you do that (analyze)? Company A Revise the already selected attributes and consider the new one as well. N/A Company B We compare the new attributes with the old attributes and prioritize the new attributes. N/A Company C Revise the already selected attributes and consider the new one as well. N/A Company D We compare the new attributes with the old attributes and prioritize the new attributes. N/A Company E We compare the new attributes with the old attributes and prioritize the new attributes. N/A What are the implications of not practicing this? What are the processes to consider new scenarios? What are the implications of not practicing this? What are the processes to consider risk and non risk items? For new scenarios all the quality attributes are considered and refined. N/A We compare the existing scenario and new scenario and prioritize the scenarios. N/A For new scenarios all the quality attributes are considered and refined. N/A We compare the existing scenario and new scenario and prioritize the scenarios. N/A We compare the existing scenario and new scenario and prioritize the scenarios. N/A Processes to consider risk and non risk items are complex work flows, Work flows with huge transaction, Work flows with multiple accesses, Workflows with distributed transaction process. Processes to consider sensitivity points are complex work flows, Work flows with huge transaction, Work flows with multiple accesses, Workflows with distributed transaction process. Processes to consider trade off points are complex work flows, Work flows with huge Processes to consider risk and non risk items are complex work flows, Work flows with huge transaction, Work flows with multiple accesses, Workflows with distributed transaction process. Processes to consider sensitivity points are complex work flows, Work flows with huge transaction, Work flows with multiple accesses, Workflows with distributed transaction process. Processes to consider trade off points are complex work flows, Work flows with huge What are the processes to consider sensitivity points? What are the processes to consider trade off points? Processes to consider risk and non risk items are complex work flows, Work flows with huge transaction, Work flows with multiple accesses, Workflows with distributed transaction process. Processes to consider sensitivity points are complex work flows, Work flows with huge transaction, Work flows with multiple accesses, Workflows with distributed transaction process. Processes to consider trade off points are complex work flows, Work flows with huge 96 Question Company A Company B transaction, Work flows with multiple accesses, Workflows with distributed transaction process. Company C Company D transaction, Work flows with multiple accesses, Workflows with distributed transaction process. Company E transaction, Work flows with multiple accesses, Workflows with distributed transaction process. What are the implications of not practicing this? The project may suffer during production. The projects sometimes suffer during production and need costly modification. The Table 47 contains the converted high level status of the surveyed companies for the Step 8 of ATAM framework. Table 47 : High Level Quantitative feedback on Step 8 Question Are the new quality attributes considered found in the brainstorm session? Are the new scenarios considered Are the new risks, non-risks, sensitivity points and trade off points included? Total points earned % of Total Point Company A 10 Company B 10 Company C 10 Company D 10 Company E 10 10 10 10 10 10 10 10 10 10 10 30 100% 30 100% 30 100% 30 100% 30 100% Following table (Table 48) contains the detail quantitative points for five different companies for ATAM Step 8. Table 48 : Detail Quantitative feedback on Step 8 Question How do you do that? What are the implications of not practicing this? What are the processes to consider new scenarios? What are the implications of not practicing this? What are the processes to consider risk and non risk items? What are the processes to consider sensitivity points? What are the processes to consider trade off points? What are the implications of not practicing this? Total earned points Company A 6 10 5 10 0 0 0 6 37 Company B 5 10 6 10 6 7 5 10 59 Company C 5 10 5 10 0 0 0 7 37 Company D 6 10 5 10 6 7 5 10 59 Company E 6 10 6 10 7 6 6 10 61 97 Question % of Total Point Company A 46.25% Company B 73.75% Company C 46.25% Company D 73.75% Company E 76.25 In the high level it is thought that the Step 8 – analyze the architectural approach is fully practiced in the companies. But, the detail level data indicates that the practice level is significantly low. The variation of practices among the companies is also very high. Following graph (Figure 13) represents the overall analyze the architectural approaches (Step 8) status for different companies. Figure 13 : Practice Level Status of ATAM Step 8 "Analyze the architectural approaches" 5.2.4 Phase 4 – Report the ATAM Step 9: Report the ATAM The Table 49 contains the high level status of the surveyed companies for the Step 9 of ATAM framework. Table 49 : High Level feedback on Step 9 Question Is the utility tree Company A No utility tree is Company B No utility tree is Company C Sometimes utility Company D No utility Company E No 98 Question prepared and reported? Are the generated scenarios reported? Company A prepared Yes we report by different diagrams and descriptions Company B prepared Yes we report by different diagrams and descriptions Company C tree is prepared Yes we report by different diagrams and descriptions Company D tree is prepared Yes we report by different diagrams and descriptions Yes analysis questions reported. Risks are reported. Yes reported to the Mother Company. Company E Yes. Are the analysis questions reported? Are the risks and non-risks reported? Is the architectural approach identified, finalized and reported? Yes analysis questions reported. Risks are reported. Final architectural approaches are documented and describes details to every stakeholder. Usually Not. Sometimes, Yes. Yes. Risks are reported sometimes. Final architectural approaches are documented and describes details to every stakeholder. Risks are reported sometimes. Final architectural approaches are documented and describes details to every stakeholder. Risks are reported. Yes. Following table (Table 50) contains answer for the questionnaires for reporting the ATAM (Step 9). Table 50 : Detail feedback on Step 9 Question What are they? What are the implications of not practicing this? Company A Company B Company C Company D Company E What are they? What are the implications of not practicing this? What are they? Not exactly used utility tree. But quality attributes, their importance were identified. No very evident implication. Different type of Use case scenarios is considered. Growth scenarios are also considered sometimes. N/A We cannot prioritize our design and work. We cannot identify the strength and weakness of architecture. Not exactly used utility tree. But quality attributes, their importance were identified. No very evident implication. We cannot prioritize our design and work. We cannot identify the strength and weakness of architecture. We cannot prioritize our design and work. We cannot identify the strength and weakness of architecture. Best case sequence diagram worst case sequence diagram. Also other complex sequence diagram. N/A Different type of scenarios is considered under the category of Use case scenarios. Growth scenarios are also considered. N/A Best case sequence diagram worst case sequence diagram. Also other complex sequence diagram. N/A Best case sequence diagram worst case sequence diagram. Also other complex sequence diagram. N/A Findings of the analysis are presented. Analysis question included new Normally findings of the analysis are Analysis question included new Analysis question included new 99 Question Company A Company B solutions of complex requirements. N/A Company C presented. Company D solutions of complex requirements. N/A Company E solutions of complex requirements. N/A What are the implications of not practicing this? What are they? N/A N/A Different types of project risks are presented. What are the implications of not practicing this? What are the common approaches identified? N/A Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed N/A Different types of project risks as well as organizational risks are presented. N/A Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed N/A Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed N/A Client Server architecture, Web architecture, Data centric architecture are also used. Layered approaches, NTier approaches, Client Server approaches and Pipeline approaches are identified, finalized and reported corresponding to the project type. Usually Client Server architecture, Event Driven architecture and Web architecture are considered. Sometimes data centric architecture, Service Oriented architecture, Plug in, etc are also used. What are the implications of not practicing this? N/A N/A N/A Layered approaches, NTier approaches, Client Server approaches and Pipeline approaches are identified, finalized and reported corresponding to the project type. These are identified by the mother company. N/A Layered approaches, NTier approaches, Client Server approaches and Pipeline approaches are identified, finalized and reported corresponding to the project type. These are identified by the mother company. N/A The Table 51 contains the converted high level status of the surveyed companies for the Step 9 of ATAM framework. Table 51 : High Level Quantitative feedback on Step 9 Question Is the utility tree prepared and reported? Are the generated scenarios reported? Are the analysis questions reported? Company A 0 10 8 Company B 0 10 3 Company C 0 10 6 Company D 0 10 8 Company E 0 10 10 100 Question Are the risks and non-risks reported? Is the architectural approach identified, finalized and reported? Total earned points % of Total Point Company A 7 8 33 66% Company B 5 8 26 52% Company C 5 8 29 58% Company D 6 9 33 66% Company E 6 10 36 72% Following table (Table 52) contains the quantitative points for five different companies. Table 52 : Detail Quantitative feedback on Step 9 Question What are they? What are the implications of not practicing this? What are they? What are the implications of not practicing this? What are they? What are the implications of not practicing this? What are they? What are the implications of not practicing this? What are the common approaches identified? What are the implications of not practicing this? Total earned points % of Total Point Company A 0 6 6 10 5 10 4 10 4 10 65 65% Company B 0 5 5 10 6 10 6 10 6 10 67 67% Company C 0 6 6 10 5 10 4 10 5 10 67 67% Company D 0 5 6 10 6 10 5 10 6 10 68 68% Company E 0 5 7 10 6 10 6 10 6 10 70 70% The data indicate that the practice level of ATAM step 9 is high both in high level and detail. The variation among the companies is not much. That means the practice level is similar for the surveyed companies. Following graph (Figure 14) represents the Report the ATAM (Step 9) status for different companies. 101 Figure 14 : Practice Level Status of ATAM Step 9 "Report the ATAM" 5.3 Analysis Result 5.3.1 Combined Practice Level of the Companies The Table 53 contains the combined high level status of the surveyed companies for all the Steps of ATAM framework. The table indicates that the average earned points of the companies are 409 out of 600 which is significantly high. The practice variations among the companies are not very high. Table 53 : High Level Combined Status of ATAM Practice Level of 5 Companies Steps 1 2 3 4 5 6 6.1 6.2 6.3 6.4 7 8 9 Company A 24 27 50 26 41 13 43 62 10 14 61 30 33 Company B 25 21 37 25 28 11 37 57 10 8 42 30 26 Company C 28 21 44 24 45 11 38 60 10 12 45 30 29 Company D 33 22 43 23 41 5 42 52 10 14 60 30 33 Company E 27 30 60 30 41 7 44 76 10 10 50 30 36 Average 27.4 24.2 46.8 25.6 39.2 9.4 40.8 61.4 10 11.6 51.6 30 31.4 Total Point 40 30 60 30 80 20 60 80 10 30 80 30 50 102 Steps Total Company A 434 Company B 357 Company C 397 Company D 408 Company E 451 Average 409.4 Total Point 600 The Table 54 contains the combined detail level status of the surveyed companies for all the Steps of ATAM framework. Here, the average earned point of the companies – 666 out of 1170 is much less than that of the high level points. The variations of practice level among the companies are comparatively higher. Table 54 : Combined detailed level status of ATAM practice of 5 companies Steps 1 2 3 4 5 6 6.1 6.2 6.3 6.4 7 8 9 Total Company Company Company Company Company Average Total A B C D E Point 34 30 38 31 32 50 33 30 48 36 48 50 60 42.4 55 56 57 68 63 80 59.8 15 26 21 27 27 40 23.2 47 74 59 78 77 170 67 30 33 31 33 21 40 29.6 62 84 62 83 83 130 74.8 63 110 74 110 114 170 94.2 15 17 15 17 18 20 16.4 28 16 30 20 19 90 22.6 63 94 64 101 104 140 85.2 37 59 37 59 61 80 50.6 65 67 67 68 70 100 67.4 544  714 591 743 739  666.2 1170 Following graph represents the combined detailed status of the ATAM practices for the surveyed companies 5.3.2 Combined Analysis Result  The Table 55 shows the combined and comparative status of the ATAM practice level of the surveyed companies. The total point available in high level study is 600 and in detail level study is 1170. The average points earned for the high level is 409 out of 600 and 666 out of 1170 for detail level practice. 103 The Table 56 shows the comparative status of the companies based on the percent points earned out of the available points. The result indicates that at the higher level, the practice level is much higher than the detail level. It is also evident that for some steps the practice level is much higher than the other steps. There is signification variation among the practice level of the companies. 104 Table 55 : Combined Status of ATAM Practice Level (Points earned) ATAM Steps 1 2 3 4 5 6 6.1 6.2 6.3 6.4 7 8 9 Total Company A HLS DLS 34 24 30 27 55 50 15 26 47 41 30 13 62 43 63 62 15 10 28 14 63 61 37 30 65 33 434 544 Company B HLS DLS 30 25 48 21 56 37 26 25 74 28 33 11 84 37 110 57 17 10 16 8 94 42 59 30 67 26 357 714 Company C HLS DLS 38 28 36 21 57 44 21 24 59 45 31 11 62 38 74 60 15 10 30 12 64 45 37 30 67 29 397 591 Company D HLS DLS 31 33 48 22 68 43 27 23 78 41 33 5 83 42 110 52 17 10 20 14 101 60 59 30 68 33 408 743 Company E HLS DLS 32 27 50 30 63 60 27 30 77 41 21 7 83 44 114 76 18 10 19 10 104 50 61 30 70 36 451 739 Average HLS DLS 27.4 33 24.2 42.4 46.8 59.8 25.6 23.2 39.2 67 9.4 29.6 40.8 74.8 61.4 94.2 10 16.4 11.6 22.6 51.6 85.2 30 50.6 31.4 67.4 409.4 666.2 Total Point HLS DLS 50 40 60 30 80 60 40 30 170 80 40 20 130 60 170 80 20 10 90 30 140 80 80 30 100 50 600 1170 HLS: High Level Status DLS: Detail Level Status 105 Table 56 : Comparative Status of ATAM Practice Level (% of available points) ATAM Steps 1 2 3 4 5 6 6.1 6.2 6.3 6.4 7 8 9 Total Company A Company B Company C Company D Company E Average HLS % DLS % HLS% DLS% HLS% DLS% HLS% DLS% HLS% DLS% HLS% DLS% 60 68 62.5 60 70 76 82.5 62 67.5 64 68.5 66 90 50 70 80 70 60 73.33 80 100 83.33 80.67 70.67 83.33 68.75 61.66 70 73.33 71.25 71.66 85 100 78.75 78 74.75 86.66 37.5 83.33 65 80 52.5 76.66 67.5 100 67.5 85.33 58 51.25 27.64 35 43.52 56.25 34.70 51.25 45.88 51.25 45.29 49 39.41 65 75 55 82.5 55 77.5 25 82.5 35 52.5 47 74 71.66 47.69 61.66 64.61 63.33 47.691 70 63.84 73.33 63.85 68 57.54 77.5 37.05 71.25 64.70 75 43.521 65 64.70 95 67.05 76.75 55.41 100 75 100 85 100 75 100 85 100 90 100 82 46.66 31.11 26.66 17.77 40 33.33 46.66 22.22 33.33 21.11 38.67 25.11 76.25 45 52.5 67.14 56.25 45.71 75 72.142 62.5 74.28 64.5 60.85 100 46.25 100 73.75 100 46.25 100 73.75 100 76.25 100 63.25 66 65 52 67 58 67 66 68 72 70 62.8 67.4 72.33 46.49 59.5 61.02 66.16 50.51 68 63.50 75.16 63.16 68.23 56.94 HLS: High Level Status DLS: Detail Level Status 106 The Figure 15-19 represents the comparative practice level status of all ATAM steps of all companies respectively. Figure 15 : Combined ATAM Practice Level Status of Company A Figure 16 : Combined ATAM Practice Level Status of Company B 107 Figure 17 : Combined ATAM Practice Level Status of Company C Figure 18 : Combined ATAM Practice Level Status of Company D 108 Figure 19 : Combined ATAM Practice Level Status of Company E The following figure (Figure 20) depicts the average practice level of ATAM of the companies. It is evident that the variation among the higher level and detail level is significant. The variations among the companies are also significant. It is found that half of the steps and activities are performed in the local companies to some extent. Figure 20 : Average Practice Level Status of the Companies CHAPTER SIX ATAM ADAPTATION 6.1 Introduction ATAM adaptation is a process of combining the industry practices and academic knowledge. Different companies follow different process to define, evaluate and maintain their software architecture, if any. But ATAM provides a very good guidelines and framework to evaluate and refine the architecture that best fits the need of the project doing the tradeoff analysis. 6.2 Adapted ATAM Adapted ATAM fuses the related practices in the industry with the academic knowledge and also provides the additional clarifications to practice ATAM framework focusing on the current practice. Following sections help determine what to do at what segment of ATAM for the local practices to Small and Medium sized companies. Here relevance to local practices and additional clarification to practices of ATAM framework are informed. 6.2.1 Phase 1 - Presentation Phase 1 - Presentation includes “Present the ATAM”, “Present the business drivers” and “Present the architecture to be evaluated” steps of ATAM. Each section is 110 described as relevance to local practices and additional clarifications to consider practicing ATAM framework. 6.2.1.1 Present the ATAM Relevance to local practices Many local companies are practicing ATAM Presentation as stealth mode. They are practicing this but they don’t know the detail representation and the process of representation of ATAM. They practice architecture evaluation process to some extent. They arrange architecture process presentation meeting sometimes. They select participants of meeting corresponding to the practitioners’ experience and skills of different types of projects. Also they are concerned about the implications of not practicing the architecture evaluation process but they realize it after any problem arises during development time. This is relevant step to be practiced by the small and medium sized software companies of Bangladesh. The related source of information and analysis result is given in section 5.2.1.1, table 1 and 2. Additional clarifications to practice ATAM Following steps should be considered for local practices: 1. Present the process of Architecture Evaluation to make sure that the right architecture is being chosen. 2. Collect the functionality and plenty of knowledge about the project before meeting. 3. Involve the related stakeholders of the project who can contribute to the project or might be affected by the project. 111 4. Try to involve client representative in the evaluation process. At least, take their concern about the process and outcome. 5. Take the concerns of every stakeholders related to project even they are not part of Architecture evaluation team. 6.2.1.2 Present the Business Drivers Relevance to local practices Local companies are presenting business drivers but not in full phase. Companies are considering business goal or business drivers in architecture meeting focusing the project category. But that is not in a structured manner. Also they are considering constrains of business like the resource constraint, time constraint, business requirement clarification uncertainty, requirement complexity, lack of domain knowledge and technological uncertainty. Some of the companies involve all the relevant stakeholders of the project and some of the companies don’t do it. The related source of information and analysis result is given in section 5.2.1.2, table 5 and 6. Additional clarifications to practice ATAM Following steps should be considered for local practices: 1. Present the business goals and business drivers which are relevant for the project. 2. Consider business process fitting and flexibility, least time response, customizability, security, Performance, Availability. 3. Consider future enhancement scope for the corresponding to the project. 112 4. Consider business requirement clarification uncertainty, requirement complexity, lack of domain knowledge and technological uncertainty. 5. Consider the resource constraint and time constraint. 6. Involve the relevant stakeholders like Client Representative, System Analyst, Architect, Project Manager, Team Lead and Developers to identify the business goals and system constraints. 7. Clearly describe the system functions to all. 6.2.1.3 Present the Architecture to be evaluated Relevance to local practices The architecture team in local companies describes the architecture to be evaluated. Each company describes in their own way. No defined processes are followed for presenting the architecture to be evaluated. They present the architectural approach. But this also depends on the project criteria. They include architecture diagram in sufficiently detail. Several approaches are followed for defining the architectural diagrams. The strong focus is given on the technical constraints. Technical uncertainties are also considered. For the quality attributes they are not practicing every quality attributes. Depending on the project category they are considering the quality attributes like Performance, Security, Availability, Plugability and Future enhancement. All these practices vary company to company. Some of the companies are following the deployment architecture which is not included or mentioned in the ATAM. The related source of information and analysis result is given in section 5.2.1.3, table 9 and 10. 113 Additional clarifications to practice ATAM Following steps should be considered for local practices: 1. Define the architecture for the project beforehand. 2. Define the architectural approaches like n-tier web architecture, 3-tiered enterprise architecture, pipeline approach, client server approach clearly including the reason for selecting that approach. 3. Include the Layered diagram, Layer to Layer communication diagram, and best case sequence diagrams for communication with each component, worst case sequence diagram for communication with each component and deployment diagrams. 4. Address all constraints and uncertainty. 5. Address all quality attributes like reliability, performance, security, availability, plugability and future enhancement at the initial level. 6.2.2 Phase 2 - Investigations and Analysis Phase 2 - Investigations and Analysis includes “Identify the architectural approaches”, “Generate the Quality Attribute Utility tree”, “Analyze the Architectural Approaches”, “Investigation of architectural approach”, “Creation of analysis questions”, “Answers to the analysis questions” and “Find the risks, non-risks, sensitivity points, trade-off points” steps of ATAM framework. Each section is described as relevance to local practices and additional clarifications to consider practicing ATAM framework. 114 6.2.2.1 Identify the Architectural Approach Relevance to local practices Many of the local companies (around 70%) usually follow this or similar step to identify the architectural approaches. They found it as the very relevant step for architecture evaluation. However, the companies are not very aware and able to explain how the critical goals will be met. The related source of information and analysis result is given in section 5.2.2.1, table 13 and 14. Additional clarifications to practice ATAM Following steps should be considered for local practices: 1. The architectural team should explain the flow of control of the architecture to the team. 2. Flow of controls should be determined by different business workflows considering the best and worst case scenarios. 3. The architectural team should approach the critical requirements like critical business functions, response time, scalability, modifiability, performance, reusability, etc. 4. The critical goals should be considered by the complex business requirement and algorithmic complexity for the business goal. 5. The architectural team should provide explanation how the critical goal will be met. 6.2.2.2 Generate the Quality Attribute Utility Tree Relevance to local practices This step or similar practices are not common in local companies. About 80% companies are not aware of it. However, they usually identify the quality goals, 115 consider the use case scenarios. They usually consider the quality attribute like performance, reliability, security, modifiability, usability, etc. They also prioritize these quality attributes. Following this steps and the quality attributes are totally dependent on the project. But, they agree on the strong relevance and importance of this step for architectural evaluation. The related source of information and analysis result is given in section 5.2.2.2, table 17 and 18. Additional clarifications to practice ATAM Following steps should be considered for local practices: 1. The quality goals should be Cleary identified. 2. The quality goals should consider Security, Performance, Functionality and Reliability at least. 3. The utility tree should be practiced. 4. If utility tree seems to be complex, then similar process should be used to determine the quality attributes, find out the importance to the project, and prioritize accordingly. 5. The utility tree and similar thing should specify the system goals and prioritize the quality attributes. 6. All the stakeholders’ scenarios should be considered. The use case scenarios, and growth scenarios definitely should be considered. The exploratory scenarios can be avoided based on the project nature. 6.2.2.3 Analyze the Architectural Approach Relevance to local practices The local companies usually analyze the architectural approaches. They don’t use all the steps as described in ATAM, but they do the thing in some means. They all agree 116 on the relevance of this step for architectural evaluation. The related source of information and analysis result is given in section 5.2.2.3, table 21 and 22. Additional clarifications to practice ATAM Following steps should be considered for analyzing the architectural approaches: 1. Analyze how the architectural approach can meet the quality and system requirements. 2. Analyze in which situation the System can fail. 3. Figure out all the critical points like the main threats and risks. 6.2.2.4 Investigation of Architectural Approach Relevance to local practices Local companies sometimes practice the investigation of the architectural approaches to some extent. They very objectively do not or cannot relate the architectural approaches with quality goal. But if they can do that could have been better. The related source of information and analysis result is given in section 5.2.2.4, table 25 and 26. Additional clarifications to practice ATAM For the investigation on architectural approach, the following things should be considered: 1. Identify the relevant and required quality attributes for the project that should be met. 117 2. Consider the quality attributes security, modifiability, reliability, conceptual integrity, functionality, performance if not irrelevant to project scope. 3. Investigate how architectural approach support these quality attributes if relevant. 6.2.2.5 Creation of Analysis Questions Relevance to local practices The local companies do ask the analysis questions, but not always make them structured and formal. But, this would be a good practice to make analysis questions for architecture evaluation. The related source of information and analysis result is given in section 5.2.2.5, table 29 and 30. Additional clarifications to practice ATAM Following things should be considered in analysis questions: 1. Ask questions based on quality attributes, their priority and importance. 2. Consider reusability, framework expandability, validation, consistency, modifiability, integrity in making the analysis question. 3. Consider the cost and time implication to achieve these goals. 6.2.2.6 Answers to the Analysis Questions Relevance to local practices The answers are usually followed by the questions in the evaluation meeting. The answers are not also recorded, discussed and analyzed in structured way. But they feel 118 its necessity. The related source of information and analysis result is given in section 5.2.2.6, table 33 and 34. Additional clarifications to practice ATAM Following things should be considered for recording and analyzing the answers: 1. The answers and explanations to the questions should be analyzed in the meeting. 2. All the answers and explanations should be criticized from different angles by the different stakeholders to figure out if there are any flaws. 6.2.2.7 Find the Risk, Non-Risks, Sensitivity and Tradeoff Points Relevance to local practices Local companies usually work on the risks of the projects. They are usually not very aware of and capable of practicing sensitivity points and tradeoff points. But they all agree these are utmost requirement for making right decisions about the architecture. The related source of information and analysis result is given in section 5.2.2.7, table 37 and 38. Additional clarifications to practice ATAM Following are the things to be considered for finding risk, non-risk, sensitivity and trade-off points: 1. Identify the project risks and non-risks properly. The most of the risks are project dependent. 119 2. Consider unavailability of client resource, failure of meeting the performance goals, quality requirements, requirement complexity, technological uncertainty etc. as risk. 3. Figure out the sensitivity points of the project if any. The sensitivity points are dependent on the project and client. 4. Check if there is any trade off points. 5. Make the right allocation of resources to the different points based on the trade off, priority, importance and complexity. 6.2.3 Phase 3 - Testing Phase 3 - Testing includes “Brainstorm and Prioritize Scenarios” and “Analyze the architectural approaches”. Each section is described as relevance to local practices and additional clarifications to consider practicing ATAM framework. 6.2.3.1 Brainstorm and Prioritize Scenarios Relevance to local practices Most of the companies do it in first stage of testing of ATAM but not as structured as ATAM. Some of the companies involve all the related stakeholders for the brainstorm and prioritize scenarios and some of the companies do not involve all the related stakeholders. Some of the companies collect all the scenarios and some of don’t collect all the scenarios after brainstorm. For this reason they miss several scenarios which can be evident during production. Most of the companies consider all the relevant use case scenarios. Most of the companies consider the growth scenarios but not like the ATAM. Some companies consider the exploratory scenarios and some of 120 them don’t consider this. Most of the companies are prioritizing the scenarios. But no companies are practicing the voting process to prioritize scenarios that are selected in the brainstorm session. The related source of information and analysis result is given in section 5.2.3.1, table 41 and 42. Additional clarifications to practice ATAM Following steps should be considered for local practices: 1. All stakeholders need to be involved 2. All the scenarios need to consider. Examples like: Complex work flows, work flows with huge transaction, work flows with multiple access, workflows with distributed transaction process. 3. Every application function should consider for the use case scenarios. 4. Every category of growth scenarios should be considered like how much growth can happen in a certain period of time in terms of users, business volume, storage, etc. 5. Every exploratory scenario should be considered like database system portability, performance, availability and security. 6. Every scenario should be prioritized based on application functionality, complexity, business driver and stability of requirement. 7. Voting process can be avoided to prioritize the scenarios. 6.2.3.1 Analyze the Architectural Approaches Relevance to local practices 121 Most of the companies analyze the architectural approach after getting the quality attribute from the utility tree. They do not create the utility tree but for this they analyze functionality in a detailed manner. Sometimes they consider external plugability with the system. If required then they change the quality attribute and fit the architecture with the external system compatibility. Every complex workflow for the system is also considered as scenario. They also consider the risk and non-risk factors for the newly selected scenario. They consider the tradeoff points but not in a structured manner. For tradeoff points they just consider the newly created or selected scenarios are compatible with the architecture or not. The related source of information and analysis result is given in section 5.2.3.2, table 45 and 46. Additional clarifications to practice ATAM Following work should be done: 1. Revise the already selected attributes and consider the new one as well. Compare the new attributes with the old attributes and prioritize the new attributes. 2. Compare the existing scenario and new scenario and prioritize the scenarios. 3. Scenarios for all the quality attributes should consider and refine. 4. Consider risk and non risk items as complex work flows, work flows with huge transaction, work flows with multiple access, and workflows with distributed transaction process. 5. Consider sensitivity points as complex work flows, work flows with huge transaction, work flows with multiple access and workflows with distributed transaction process. 122 6. Consider trade off points as complex work flows, work flows with huge transaction, work flows with multiple access and workflows with distributed transaction process. 6.2.4 Phase 4 - Report the ATAM Phase 4 – Report the ATAM includes the ATAM reporting. Reporting section is described as relevance to local practices and additional clarifications to consider practicing ATAM framework. 6.2.4.1 Report the ATAM Relevance to local practices Most of the companies practice “Report the ATAM” using their own process. No defined and standard report is produced. Companies generate different scenarios and report those. They prepare analysis questions and collect all possible answers for the analysis questions. Different sort of risk and non-risk factors are considered for different category of the project. Architectural approaches are identified and finalized. All these work are done but except the important part of ATAM “Create Utility Tree”. Companies have no idea about the utility tree. They map different quality attributes but don’t follow anything related to utility tree. For not using this they are facing problem like they cannot prioritize the design and work, identify the strength and weakness of architecture, how quickly change impact can be analyzed. The related source of information and analysis result is given in section 5.2.4.1, table 49 and 50. Additional clarifications to practice ATAM 123 Following work should be done: 1. A utility tree containing all quality attributes should be created. 2. Different sort of scenarios should be considered under the category of Use case scenarios, Growth scenarios. Best case sequence diagram and worst case sequence diagram should be considered. Also other complex sequence diagram should be created for explaining the architecture clearly. 3. All complex workflows and all solutions for the newly generated complex workflows should be considered. This analysis question part should be applied for each and every complex step of the system. 4. Complex workflows, workflows with huge transaction, workflows with multiple access, and workflows with distributed nature should be considered for the risk and non-risk factor identification. Different type of project risks as well as organizational risks should be considered. 5. Architectural approaches like Client Server architecture, Event Driven architecture and Web architecture, data centric architecture, Service Oriented architecture, Layered approaches, N-Tier approaches, Pipeline approaches, etc. should be identified with proper explanation and reasoning for selecting each approach. 6.3 Work Items of Adapted ATAM Adaptation process describes the required practices of ATAM framework. To practice ATAM framework following work items need to be generated focusing on the additional clarification to practice ATAM mentioned in the section 6.2[4]. • • ATAM Representation Business Processes and Drivers Identification 124 • • • Architecture Representation Identify Architectural Approaches Quality Attribute Utility Tree o Quality Attribute Characterizations o Scenarios Use Case Scenarios Growth Scenarios Exploratory Scenarios o Utility Trees • • • Analyze Architectural Approaches Brainstorm and Prioritize Scenarios Result of ATAM o Risks and Non-Risks o Sensitivity Points o Tradeoff Points CHAPTER SEVEN SUMMARY AND PROPOSITION FOR FUTURE WORK For the objectives of this research (mentioned in Chapter 1), this chapter reports what the progress has been made, limitations of the work and recommendations for future work by acknowledging those limitations. 7.1 Progress Made The study revealed more or less the nature of the architecture design process is in practice in the local companies. Basically, the Architectural Practices in small and medium sized software companies are not well evident. They use some common style of architecture without considering of its merits or demerits in detail. The study also revealed the architecture evaluation process in practice. The evaluation of architecture is not very common practice. They sometimes do the evaluation without following any structured framework or methodology like ATAM. The study recorded and analyzed the high level and detailed level gap of practice with standards and corresponding implications to company productivity, maintenance and delivery capacity. The analysis result shown in section 5.3.2 depicts that the average practice level of ATAM in the local companies is medium and half of the steps and activities are performed in the local companies to some extent. It is also evident that the variation among the higher level and detail level practice is significant. The 126 variations among the companies are also significant. Therefore, the full blown ATAM should not be practiced blindly in the small and medium sized software companies. As the full blown ATAM is not the solution for the small and medium sized software companies in Bangladesh, the study defined the ADAPTED ATAM combining the academic knowledge, global standards and local industry practitioners’ knowledge and concerns, for the local practitioners. 7.2 Limitation of the Progress The ATAM is a very good framework for evaluating and refining the architecture. But this is not so simple and easy to apply and practice. Therefore, the study was conducted to make some recommendation and propose simpler ATAM framework which will be suitable for the local companies. For conducting this study the practitioners input was very important. As the ATAM framework itself is a big topic, therefore the participants were not very interested to provide enough inputs. Following are some factors that affected the progress of this study. • Wide knowledge of software architecture and architectural practices is mostly absent to local practitioners. • Concept of architecture evaluation among the local practitioners in small and medium companies is seldom found. • Thorough understanding and knowledge on tradeoff analysis process among the architect is difficult to find in the small and medium companies in software industry in Bangladesh. • Knowledge of ATAM is absent among the practitioners 127 However, the study was conducted on 5 selected companies which are the leading companies of Bangladesh. Therefore, the findings of this study are not 100% true representation of the reality and are not highly generalized. 7.3 Recommendation for Future Work The ATAM framework is a very vast topic, as it is already mentioned. The study conducted is at preliminary level. The study should be conducted to the larger community, i.e. involving the practitioners from the different category of Software companies like local, outsourcing, offshore development center, etc. and of different size employing effective and efficient market research approach. Enough time and resource has to be allocated for the practitioners to provide the real feedback on their practice level. The practitioners can be provided preliminary knowledge of architectural practices and ATAM framework itself so that they can relate the real life practices with the theoretical ones. This can provide the better opportunity to adapt the ATAM more suitably for the local small and medium sized software companies. Furthermore the Adapted ATAM should be applied to the different real life projects and should be validated and refined. 128 REFERENCES [1] Clements, C., Kazman, R., Klein, M.: Evaluating Software Architectures. Addison-Wesley, Boston (2002) “Software Architecture Evaluation: A Key to System Success”, http://interactive.sei.cmu.edu Mildred N. Ambe and Frederick Vizeacoumar, “Evaluation of two architectures using the Architecture Tradeoff Analysis Method (ATAM)” http://en.wikipedia.org/wiki/Architecture_Tradeoff_Analysis_Method http://www.softwarearchitectures.com/go/Discipline/EvaluatingArchitecture/ A TAM/tabid/67/Default.aspx http://www.sei.cmu.edu/architecture/ata_method.html Using the Architecture Tradeoff Analysis MethodSM (ATAMSM) to Evaluate the Software Architecture for a Product Line of Avionics Systems: A Case Study http://www.sei.cmu.edu/pub/documents/03.reports/pdf/03tn012.pdf Bass, L.; Clements, P.; & Kazman, R. Software Architecture in Practice, 2nd edition. Boston, MA: Addison-Wesley, 2003. Clements, P. & Northrop, L. Software Product Lines: Practices and Patterns. Boston, MA: Addison-Wesley, 2002. Clements, P.; Kazman, R.; & Klein, M. Evaluating Software Architectures: Methods and Case Studies. Boston, MA: Addison-Wesley,2002. Clements, P.; Bachmann, F.; Bass, L.; Garlan, D.; Ivers, J.; Little, R.; Nord, R.; & Stafford, J. Documenting Software Architectures: Views and Beyond. Boston, MA: Addison-Wesley, 2003. Colucci, F. “Avionics Upgrade Underway for Special Ops Helicopters.” National Defense 87, 591 (February 2003): 24-26. http://www.nationaldefensemagazine.org/article.cfm?Id=1029 IEEE Computer Society, Software Engineering Standards Committee, Institute of Electrical and Electronics Engineers, IEEE-SA Standards Board, and IEEE Xplore. IEEE Recommended Practice for Architectural Description of Software-Intensive Systems. New York, NY: Institute of Electrical and Electronic Engineers, 2000. Kazman, R.; Klein, M.; & Clements, P. ATAM: Method for Architecture Evaluation (CMU/SEI-2000-TR-004, ADA382629). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 2000. [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] 129 http://www.sei.cmu.edu/publications/documents/00.reports/00tr004.html [15] Applying Architecture Tradeoff Assessment Method (ATAM) As Part Of formal Software Architecture Review http://www.mitre.org/work/tech_papers/tech_papers_07/07_0094/07_0094.pdf Byrnes, C.: Requirements and Design Approach for a High-Reliability System. In: Firesmith, D. (ed.) Proceedings of Fifth International Workshop on Requirements for High Assurance Systems (RHAS 05), IEEE, New York, 2005 Scenario-Based Software Architecture Evaluation Methods: An Overview http://www.win.tue.nl/oas/architecting/aimes/papers/ScenarioBased%20SWA%20Evaluation%20Methods.pdf Kazman, R., Bass, L., Abowd, G. and Webb, S.M. SAAM: A method for analyzing the properties of software architecture. Proceedings of the International Conference on Software Engineering - ICSE'16, pages 81-90, 1994. A Framework for Classifying and Comparing Software Architecture Evaluation Methods http://www.cse.unsw.edu.au/~limingz/publication/ASWEC2004_Zhu.pdf C.-H. Lung and K. Kalaichelvan, "An Approach to Quantitative Software Architecture Sensitivity Analysis," International Journal of Software Engineering and Knowledge Engineering, vol. 10, No. 1, 2000, pp. 97-114. P. Bengtsson, "Towards Maintainability Metrics on Software Architecture: An Adaptation of Object-Oriented Metrics," FirstNordic Workshop on Software Architecture, Ronneby, 1998. N. Lassing, D. Rijsenbrij, and H. v. Vliet, "The goal of software architecture analysis: Confidence building or risk assessment," Proceedings of First BeNeLux conference on software architecture, 1999. G. Abowd, L. Bass, P. Clements, R. Kazman, L. Northrop,and A. Zaremski, "Recommanded Best Industrial Practice for Software Architecture Evaluation," SEI, Carnegie Mellon University CMU/SEI-96-TR-025, 1997. [16] [17] [18] [19] [20] [21] [22] [23] [24] R. Kazman, L. Bass, G. Abowd, and M. Webb, "SAAM: A Method for Analyzing the Properties of Software Architectures," Proceedings of the 16th International Conference on Software Engineering, 1994. [25] R. Kazman, M. Klein, M. Barbacci, T. Longstaff, H. Lipson,and J. Carriere, "The Architecture Tradeoff Analysis Method," Proceedings of IEEE, ICECCS, 1998. P. C. Clements, "Active Reviews for Intermediate Designs," SEI, Carnegie Mellon University CMU/SEI-2000-TN-009, 2000. [26] 130 [27] C.-H. Lung, S. Bot, k. Kalaichelvan, and R. Kazman, "An Approach to Software Architecture Analysis for Evolution and Reusability," Proceedings of CASCON, 1997. P. Bengtsson, N. Lassing, J. Bosch, and H. V. Vliet, "Architecture-Level Modifiability Analysis," Journal of Systems and Software, vol. 69, 2004. P. Bengtsson and J. Bosch, "Architectural Level Prediction of Software Maintenance," Proceedings of 3rd European Conference on Software Engineering Maintenance and Reengineering, 1999. P.-O. Bengtsson and J. Bosch, "Scenario-based Architecture Reengineering," 5th International Conference on Software Reuse, 1998. N. Lassing, D. Rijsenbrij, and H. v. Vliet, "On Software Architecture Analysis of Flexibility, Complexity of Changes: Size isn't Everything," Proceedings of 2nd Nordic Software Architecture Workshop, 1999. G. Molter, "Integrating SAAM in Domain-Centric and Reuse-based Development Processes," Proceedings of the 2nd Nordic Workshop on Software Architecture, Ronneby, 1999. M. Svahnberg, C. Wohlin, L. Lundberg, and M. Mattsson, "A Method for Understanding Quality Attributes in Software Architecture Structures," Proceedings of the 14th International Conference on Soft Engineering and Knowledge Engineering, Ischia, Italy, Jul., 2002. M. Klein and R. Kazman, "Attribute-Based Architectural Styles," Soft Engineering Institute, Carnegie Mellon University, Technical Report CMU/SEI-99-TR-022, October 1999. J. C. Duenas, W. L. d. Oliveira, and J. A. d. l. Puente, "A Software Architecture Evaluation Model," 2nd International Workshop On Development and Evolution of Software Architectures for Product Families, Feb. 1998. L. Dobrica and E. Niemela, "A Survey on Software Architecture Analysis Methods," IEEE Transactions on Software Engineering, vol. 28, No. 7, 2002, pp. 638-653. [28] [29] [30] [31] [32] [33] [34] [35] [36] 131 ACRONYMS ADL ALMA ALPSM ARID ATAM CBSA COM CORBA DoD DBMS EDA EDI EJB ISAAMCR MTS RMI RPC RUP SAAM SAAMER SAAMCS SBAR SEI SOA Architecture Description Language Architecture-Level Modifiability Analysis Architecture-Level Prediction of Software Maintenance Active Reviews for Intermediate Design Architecture Tradeoff Analysis MethodSM Component-Based Software Architecture Component Object Model Common Object Request Broker Architecture Department of Defense Database Management System Event-driven Architecture Electronic Data Interchange Enterprise Java Bean Integrating SAAM in domain-Centric and Reuse-based development Microsoft Transaction Server Remote Method Invocation Remote Procedure Call Rational Unified Process Scenario-based Architecture Analysis Method SAAM for Evolution and Reusability SAAM for Complex Scenarios Scenario-Based Architecture Reengineering Software Engineering Institute of Carnegie Mellon University Service Oriented Architecture 132 APPENDIX A High Level Feedback of Company - A SL ATAM Step Description No. Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. This step involves the description of the evaluation process of the 1 Step 1: Present ATAM. In this step, the evaluation leader presents general the ATAM information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. 2 Step 2: Present In this step, the business goals of the architectural drivers of the the business system are mentioned. This step focuses on the business drivers perspective of the system. It provides additional information on the functions of the system, the major stakeholders, the business goals and other constraints of the system. 3 Step 3: Present the architecture to be evaluated In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that represent the desired qualities of a system. Examples of Question Answer Do you practice architecture evaluation process? If yes do you have any process presentation meeting? If yes who are the participants of the meeting. How the participants are chosen? Do you consider business goal or business drivers in architecture meeting? Do you consider the constraints of the business? Do you involve all the relevant stake holders like end-user, architect, developers, etc? Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? Does the presentation include architecture diagram in sufficiently detail? Does the presentation address technical constraints? Does the presentation address quality attributes requirement? Does the presentation address quality attribute 1.Yes. 2.No define process. 3.Every one related to the development 4.Every one related to the development and requirement analyzing. Project goals are defined. Yes we consider all the constraints. Yes. No. Yes. Yes all diagrams are included. Yes. yes, quality attributes are defined. Yes. 133 SL No. ATAM Step Description Question Answer these attributes include performance, reliability etc. requirements like performance, reliability, etc.? Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. 4 Step 4: Identify This step involves identifying the critical architectural Does the architectural team explain the flow of Every work flows are explained. the architectural approaches that provide an understanding of the critical control of the architecture? approaches requirements of the system. In this step, the architectural team Does the architectural approach address the Yes, it is addressed. explains the flow of control of the architecture and provides a critical requirements? proper explanation of how and whether the critical goal is met. Does the architectural team provide Yes, every complex task are explanation how the critical goal is met? define in details breakdown approach. Yes. Are the important quality attributes goals 5 Step 5: Generate In this phase of the evaluation, the system’s most important quality attribute goals are identified, prioritized and refined. This clearly identified? the Quality No utility tree made. Is the utility tree made? Attribute Utility step is crucial because it focuses the attention of all the We don’t maintain this. Does the utility tree specify the system goals stakeholders and the evaluators on the different aspects of the tree. Yes, every steps are clearly clearly? architecture that are critical to the success of the system. This is defined. Are the scenarios are clearly described? achieved through the construction of a utility tree. Yes. Are the quality attributes are prioritized? Are all the stake holders scenarios considered? The utility tree provides a way of making the system goals more Yes. Every work flows are Are all the quality attributes like the security, specific and more concrete. It also provides a comparison of the considered. performance, availability, reliability, importance of quality attribute goals relative to each other. modifiability, portability, functionality, These requirements are called scenarios. A scenario is a Not all quality attributes are variability, conceptual integrity, etc. quality statement that illustrates the interaction between a stakeholder considered. attribute considered? and the system. These scenarios are the quality goals against If utility tree is not practiced, is there other way which the architecture will be judged. 8. No idea. Utility tree is not so in practice? helpful. Scenario Generation Scenario generation is an important preceding step to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under 134 SL No. ATAM Step Description each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the utility tree. The utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, trade-off points. Question Answer 6 Step 6: Analyze the Architectural Approaches 1. How investigation done to define architectural approach? 2. How risk and non risks are identified? 1. We identify all the functionality and allocate the functionality in each step of investigation. 2. Risk items are identified from the requirement and technological challenges. 6.1 6.1 Investigation of architectural approach After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Does the architectural approach support the quality attribute? How does architectural approach support variability? How does architectural approach support modifiability? How does architectural approach support reliability? 1. Yes, we define the quality attributes. 2. Our approaches are variable. We design taking consideration of every approach. 3. Our approaches are easy to modify as we design taking consideration of this. 135 SL No. ATAM Step Description Question How does architectural approach support conceptual integrity? How does architectural approach support functionality? Answer 4. We test our approach to support reliability. 5. We take consideration of integration at the time of design depending on the project category. 6. Every functionalities are distinctly described. 1. Yes. 2. Yes, Utility, Security components are reusable. 3. Yes, But depends on the project. 4. Yes 5. Yes. 6. Yes, but depends on the project. 7. Yes, but depends on the project. 8. Yes, Integration test done always. 1. Yes, we consider every issue. 6.2 6.2 Creation of analysis questions The next stage in this step involves in gathering analysis questions generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. 6.3 6.4 6.3 Answers to the analysis questions 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. The final stage involved in this step is to identify the risk, nonrisk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. Do you create analysis questions? Can the components of the architecture reused for future projects? Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Can any new application-specific functionality be added to the architecture? Can the system be modified in short time and cost-effective manner? Do the components interact properly? Are the answer and explanations to the questions analyzed? Are the risks and non-risks identified properly? Are the sensitivity points identified properly? Are the trade off points identified properly? Risks are identified properly. Not in defined process. Not in define process. 136 SL No. ATAM Step Description A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Question Answer Phase 3 - Testing 7 Step 7: Brainstorm and Prioritize Scenarios This is the first step in the Testing Phase of the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. This is the last step in the ’Testing’ phase. In this step the highly 8 Step 8: Analyze the architectural voted quality attributes from the previous step are analyzed. The architectural approach that deals with these quality attributes are Approaches found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Phase 4 - Report the ATAM 9 Step 9: Report This is the final phase of the ATAM evaluation, where all the Does all the stake holders involve in brainstorm? Are all scenarios collected after brainstorm? Are use case scenarios considered? Are growth scenarios considered? Are exploratory scenarios considered? Are the scenarios are prioritized? Are the voting process used for prioritizing scenarios? Yes. Everyone related to development cooperate this. Yes. Yes Yes Yes. Yes. No voting process is used. Naturally more complex task considered first. Are the new quality attributes considered found in the brainstorm session? Are the new scenarios considered? Are the new risks, non-risks, sensitivity points and trade off points included? Yes, Considered and analyzed properly. Yes. Risk points are included. Is the utility tree prepared and reported? 1. No utility tree is prepared 137 SL No. ATAM Step the ATAM Description information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches Question Are the generated scenarios reported? Are the analysis questions reported? Are the risks and non-risks reported? Is the architectural approach identified, finalized and reported? Answer Yes we report by different diagrams and descriptions Yes analysis questions reported. Risks are reported. Final architectural approaches are documented and describes details to every stakeholder. 138 B High Level Feedback of Company - B SL ATAM Step Description No. Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. 1 Step 1: Present This step involves the description of the evaluation process of the the ATAM ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. In this step, the business goals of the architectural drivers of the 2 Step 2: Present system are mentioned. This step focuses on the business the business perspective of the system. It provides additional information on drivers the functions of the system, the major stakeholders, the business goals and other constraints of the system. 3 Step 3: Present the architecture to be evaluated Question Answer Do you practice architecture evaluation process? If yes do you have any process presentation meeting? If yes who are the participants of the meeting. How the participants are chosen? Do you consider business goal or business drivers in architecture meeting? Do you consider the constraints of the business? Do you involve all the relevant stake holders like end-user, architect, developers, etc? Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? Does the presentation include architecture diagram in sufficiently detail? Does the presentation address technical constraints? Does the presentation address quality attributes requirement? Does the presentation address quality attribute requirements like performance, reliability, etc.? 1.Yes. 2.No define process. 3. Engineers from the development and marketing. 4. Senior engineers from the development and marketing. Project goals are defined. Yes we consider more or less all the constraints. Yes. But not always. Sometimes. Very high level. In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for Yes. Very high level. the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is Not always. important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical High level. Yes. constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system Some quality attributes are are the issues that represent the desired qualities of a system. defined. Very limited. Examples of Yes. these attributes include performance, reliability etc. Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on 139 SL ATAM Step Description No. during the evaluation. This phase is sub-divided into three steps. 4 Step 4: Identify This step involves identifying the critical architectural the architectural approaches that provide an understanding of the critical approaches requirements of the system. In this step, the architectural team explains the flow of control of the architecture and provides a proper explanation of how and whether the critical goal is met. 5 Step 5: Generate the Quality Attribute Utility tree. In this phase of the evaluation, the system’s most important quality attribute goals are identified, prioritized and refined. This step is crucial because it focuses the attention of all the stakeholders and the evaluators on the different aspects of the architecture that are critical to the success of the system. This is achieved through the construction of a utility tree. The utility tree provides a way of making the system goals more specific and more concrete. It also provides a comparison of the importance of quality attribute goals relative to each other. These requirements are called scenarios. A scenario is a statement that illustrates the interaction between a stakeholder and the system. These scenarios are the quality goals against which the architecture will be judged. Scenario Generation Scenario generation is an important preceding step to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the utility tree. The utility Question Answer Does the architectural team explain the flow of control of the architecture? Does the architectural approach address the critical requirements? Does the architectural team provide explanation how the critical goal is met? Are the important quality attributes goals clearly identified? Is the utility tree made? Does the utility tree specify the system goals clearly? Are the scenarios are clearly described? Are the quality attributes are prioritized? Are all the stake holders scenarios considered? Are all the quality attributes like the security, performance, availability, reliability, modifiability, portability, functionality, variability, conceptual integrity, etc. quality attribute considered? If utility tree is not practiced, is there other way in practice? Every work flows are explained. Yes, it is addressed. Yes, to some extent, every complex task is defined. Not always. Rarely Yes. No utility tree made. We don’t maintain this. No, little bit. Yes. Not always. Yes. Every work flows are considered. Not all quality attributes are considered. No idea. We don’t practice. 140 SL No. ATAM Step Description tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, trade-off points After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Question Answer 6 Step 6: Analyze the Architectural Approaches 1. How investigation done to define architectural approach? 2. How risk and non risks are identified? 1. No defined approach. 2. Risk items are identified from the requirement and technological challenges. 6.1 6.1 Investigation of architectural approach 6.2 6.2 Creation of analysis The next stage in this step involves in gathering analysis questions generated from the high prioritized scenarios already Does the architectural approach support the quality attribute? How does architectural approach support variability? How does architectural approach support modifiability? How does architectural approach support reliability? How does architectural approach support conceptual integrity? How does architectural approach support functionality? Do you create analysis questions? Can the components of the architecture reused 1. Yes, sometimes, we define the quality attributes. 2. We didn’t think of it. We consider modifiability if required. 4. We test our approach to support reliability. 5. Considered at the time of design depending on the project category. 6. Major functionalities are distinctly described. 1. Usually No. 2. Yes, Sometimes. 141 SL No. ATAM Step questions Description discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Question for future projects? Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Can any new application-specific functionality be added to the architecture? Can the system be modified in short time and cost-effective manner? Do the components interact properly? Are the answer and explanations to the questions analyzed? Are the risks and non-risks identified properly? Are the sensitivity points identified properly? Are the trades off points identified properly? Answer 3. Yes, sometimes, depends on the project. 4. Yes 5. Yes. 6. Usually Yes, but depends on the project. 7. Not always. Sometimes Yes, but depends on the project. 8. Yes, Integration test done always. 1. Yes, we consider every issue. Not always risks are identified properly. Not really. Not in defined process. Not in define process. 6.3 6.4 6.3 Answers to the analysis questions 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. The final stage involved in this step is to identify the risk, nonrisk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Phase 3 - Testing 7 Step 7: Brainstorm and Prioritize Scenarios This is the first step in the Testing Phase of the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal Does all the stake holders involve in brainstorm? Are all scenarios collected after brainstorm? Are use case scenarios considered? Are growth scenarios considered? No. Some selected members related to development cooperate this. Yes. More or less. Yes 142 SL No. ATAM Step Description Question Are exploratory scenarios considered? Are the scenarios are prioritized? Are the voting process used for prioritizing scenarios? Answer Sometimes, Yes Rarely Yes. Yes, to some extent. No voting process is used. is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. This is the last step in the ’Testing’ phase. In this step the highly 8 Step 8: Analyze voted quality attributes from the previous step are analyzed. The the architectural architectural approach that deals with these quality attributes are Approaches found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Phase 4 - Report the ATAM 9 Step 9: Report This is the final phase of the ATAM evaluation, where all the the ATAM information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches Are the new quality attributes considered found in the brainstorm session? Are the new scenarios considered? Are the new risks, non-risks, sensitivity points and trade off points included? Yes, Considered and analyzed properly. Yes. Risk points are included, if any. Is the utility tree prepared and reported? Are the generated scenarios reported? Are the analysis questions reported? Are the risks and non-risks reported? Is the architectural approach identified, finalized and reported? 1. No utility tree is prepared Yes we report by different diagrams and descriptions Usually Not. Risks are reported sometimes. Final architectural approaches are documented and describes details to every stakeholder. 143 C High Level Feedback of Company - C  SL ATAM Step Description No. Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. 1 Step 1: Present This step involves the description of the evaluation process of the the ATAM ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. Question Answer Do you practice architecture evaluation process? If yes do you have any process presentation meeting? If yes who are the participants of the meeting. How the participants are chosen? 2 Step 2: Present the business drivers In this step, the business goals of the architectural drivers of the system are mentioned. This step focuses on the business perspective of the system. It provides additional information on the functions of the system, the major stakeholders, the business goals and other constraints of the system. In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that represent the desired qualities of a system. Examples of these attributes include performance, reliability etc. 3 Step 3: Present the architecture to be evaluated Do you consider business goal or business drivers in architecture meeting? Do you consider the constraints of the business? Do you involve all the relevant stake holders like end-user, architect, developers, etc? Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? Does the presentation include architecture diagram in sufficiently detail? Does the presentation address technical constraints? Does the presentation address quality attributes requirement? Does the presentation address quality attribute requirements like performance, reliability, etc.? 1.Yes. 2. Yes, but no define process. 3. Engineers from the development and management. 4. All the engineers from the development and marketing. Project goals are defined. Yes we consider more or less all the constraints. Yes. But not always. Yes, high level. Yes. High level. Yes, usually. High level. Yes. Usually quality attributes are defined. Yes. 144 SL ATAM Step Description Question Answer No. Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. 4 Step 4: Identify This step involves identifying the critical architectural Does the architectural team explain the flow of Every work flows are the architectural approaches that provide an understanding of the critical control of the architecture? explained. approaches requirements of the system. In this step, the architectural team Does the architectural approach address the explains the flow of control of the architecture and provides a critical requirements? Yes, it is addressed. proper explanation of how and whether the critical goal is met. Does the architectural team provide explanation how the critical goal is met? Yes, to some extent. Not always. Sometimes Are the important quality attributes goals 5 Step 5: Generate In this phase of the evaluation, the system’s most important Yes. quality attribute goals are identified, prioritized and refined. This clearly identified? the Quality Is the utility tree made? Attribute Utility step is crucial because it focuses the attention of all the Sometimes. Does the utility tree specify the system goals stakeholders and the evaluators on the different aspects of the tree. Sometimes. clearly? architecture that are critical to the success of the system. This is Are the scenarios are clearly described? achieved through the construction of a utility tree. More or less. Are the quality attributes are prioritized? Are all the stake holders scenarios considered? The utility tree provides a way of making the system goals more Yes. Not always. Are all the quality attributes like the security, specific and more concrete. It also provides a comparison of the performance, availability, reliability, importance of quality attribute goals relative to each other. Yes. Every work flows modifiability, portability, functionality, These requirements are called scenarios. A scenario is a are considered. variability, conceptual integrity, etc. quality statement that illustrates the interaction between a stakeholder attribute considered? and the system. These scenarios are the quality goals against If utility tree is not practiced, is there other way Not all quality attributes which the architecture will be judged. are considered. in practice? Scenario Generation We do similar type of Scenario generation is an important preceding step to the utility works little bit. tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description 145 SL No. ATAM Step Description of the scenarios. The latter form the leaf nodes in the utility tree. The utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, trade-off points. After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Question Answer 6 Step 6: Analyze the Architectural Approaches 1. How investigation done to define architectural approach? 2. How risk and non risks are identified? 1. No defined approach. 2. Risk items are identified from the requirement and technological challenges. 6.1 6.1 Investigation of architectural approach Does the architectural approach support the quality attribute? How does architectural approach support variability? How does architectural approach support modifiability? How does architectural approach support reliability? How does architectural approach support conceptual integrity? How does architectural approach support functionality? 1. Yes, sometimes, we define the quality attributes. 2. We consider variability. We consider modifiability if required. We test our approach to support reliability. Considered at the time of design depending on the project category. 146 SL No. ATAM Step Description Question Answer Major functionalities are distinctly described. 1. Sometimes, Yes. 2. Yes, Sometimes. 3. Yes, sometimes, depends on the project. 4. Yes 5. Yes. 6. Usually Yes, but depends on the project. 7. Not always. Sometimes Yes, but depends on the project. 8. Yes, Integration test done always. 1. Yes, we consider every issue. Yes, usually risks are identified. Not really. Not in defined process. Not in define process. 6.2 6.2 Creation of analysis questions The next stage in this step involves in gathering analysis questions generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. 6.3 6.4 6.3 Answers to the analysis questions 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. The final stage involved in this step is to identify the risk, nonrisk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Do you create analysis questions? Can the components of the architecture reused for future projects? Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Can any new application-specific functionality be added to the architecture? Can the system be modified in short time and cost-effective manner? Do the components interact properly? Are the answer and explanations to the questions analyzed? Are the risks and non-risks identified properly? Are the sensitivity points identified properly? Are the trades off points identified properly? Phase 3 - Testing 7 Step 7: This is the first step in the Testing Phase of the ATAM. The Does all the stake holders involve in Usually Yes.. 147 SL No. ATAM Step Brainstorm and Prioritize Scenarios Description Question brainstorm? Are all scenarios collected after brainstorm? Are use case scenarios considered? Are growth scenarios considered? Are exploratory scenarios considered? Are the scenarios are prioritized? Are the voting process used for prioritizing scenarios? Answer Yes. More or less. Yes Sometimes, Yes Sometimes, Yes. Yes, to some extent. No voting process is used. former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. This is the last step in the ’Testing’ phase. In this step the highly 8 Step 8: Analyze the architectural voted quality attributes from the previous step are analyzed. The architectural approach that deals with these quality attributes are Approaches found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Phase 4 - Report the ATAM 9 Step 9: Report This is the final phase of the ATAM evaluation, where all the the ATAM information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions Are the new quality attributes considered found in the brainstorm session? Are the new scenarios considered? Are the new risks, non-risks, sensitivity points and trade off points included? Yes, Considered and analyzed properly. Yes. Risk points are included, if any. Is the utility tree prepared and reported? Are the generated scenarios reported? Are the analysis questions reported? Are the risks and non-risks reported? Is the architectural approach identified, finalized and reported? 1. Sometimes utility tree is prepared Yes we report by different diagrams and descriptions Sometimes, Yes. Risks are reported 148 SL No. ATAM Step Description - set of identified risks and non-risks - the identified architectural approaches Question Answer sometimes. Final architectural approaches are documented and describes details to every stakeholder. 149 D High Level Feedback of Company - D SL ATAM Step Description No. Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. 1 Step 1: Present This step involves the description of the evaluation process of the the ATAM ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. In this step, the business goals of the architectural drivers of the 2 Step 2: Present system are mentioned. This step focuses on the business the business perspective of the system. It provides additional information on drivers the functions of the system, the major stakeholders, the business goals and other constraints of the system. 3 Step 3: Present the architecture to be evaluated In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that represent the desired qualities of a system. Examples of these attributes include performance, reliability etc. Question Answer Do you practice architecture evaluation process? If yes do you have any process presentation meeting? If yes who are the participants of the meeting. How the participants are chosen? Do you consider business goal or business drivers in architecture meeting? Do you consider the constraints of the business? Do you involve all the relevant stake holders like end-user, architect, developers, etc? Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? Does the presentation include architecture diagram in sufficiently detail? Does the presentation address technical constraints? Does the presentation address quality attributes requirement? Does the presentation address quality attribute requirements like performance, reliability, etc.? 1. Yes. 2. Yes. 3.Development Team 4. Everyone who is and will be related to the development of the project. Project goals are defined. Yes. Yes. But end users are participated through their project manager. Yes Yes. Top level and process level architecture with sufficient technical details. Yes. Yes. These are discussed there. Yes quietly. Phase 2 - Investigation and Analysis 150 SL ATAM Step Description Question Answer No. This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. 4 Step 4: Identify This step involves identifying the critical architectural approaches Does the architectural team explain the flow Every work flows are the architectural that provide an understanding of the critical requirements of the of control of the architecture? explained. approaches system. In this step, the architectural team explains the flow of Does the architectural approach address the Yes. These are addressed control of the architecture and provides a proper explanation of critical requirements? through review processes. how and whether the critical goal is met. Does the architectural team provide Yes. They provide all explanation how the critical goal is met? explanation. Yes. These are clearly Are the important quality attributes goals 5 Step 5: Generate In this phase of the evaluation, the system’s most important defined from the Mother clearly identified? quality attribute goals are identified, prioritized and refined. This the Quality Company(Laval, Canada) Is the utility tree made? Attribute Utility step is crucial because it focuses the attention of all the No. Does the utility tree specify the system goals stakeholders and the evaluators on the different aspects of the tree. No. clearly? architecture that are critical to the success of the system. This is Yes, every steps are clearly Are the scenarios are clearly described? achieved through the construction of a utility tree. defined from mother Are the quality attributes are prioritized? Are all the stake holders scenarios considered? company. The utility tree provides a way of making the system goals more These are defined from the Are all the quality attributes like the security, specific and more concrete. It also provides a comparison of the mother company. performance, availability, reliability, importance of quality attribute goals relative to each other. Yes. These requirements are called scenarios. A scenario is a statement modifiability, portability, functionality, variability, conceptual integrity, etc. quality that illustrates the interaction between a stakeholder and the Try to be considered. attribute considered? system. These scenarios are the quality goals against which the If utility tree is not practiced, is there other architecture will be judged. way in practice? No, other process is not Scenario Generation maintained. Scenario generation is an important preceding step to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the utility tree. The 151 SL No. ATAM Step Description utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, trade-off points. After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Question Answer 6 Step 6: Analyze the Architectural Approaches 1. How investigation done to define architectural approach? 2. How risk and non risks are identified? 1. No investigation done. 2. By team meeting. 6.1 6.1 Investigation of architectural approach 6.2 6.2 Creation of The next stage in this step involves in gathering analysis questions Does the architectural approach support the quality attribute? How does architectural approach support variability? How does architectural approach support modifiability? How does architectural approach support reliability? How does architectural approach support conceptual integrity? How does architectural approach support functionality? Do you create analysis questions? 1. Yes. 2. through review. 3. The approach supports modifiability. We approach supports reliability. We take consideration of integration at the time of design depending on the project category. 6. Every functionalities are distinctly described. 1. No. 152 SL No. ATAM Step analysis questions Description generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Question Can the components of the architecture reused for future projects? Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Can any new application-specific functionality be added to the architecture? Can the system be modified in short time and cost-effective manner? Do the components interact properly? Are the answer and explanations to the questions analyzed? Are the risks and non-risks identified properly? Are the sensitivity points identified properly? Are the trade off points identified properly? Answer 2. Yes, Try to design that can be reused. 3. Yes, we consider that but Project types doesn’t match with other project types. 4. Yes 5. Yes. 6. Yes, but depends on the project. 7. Yes, but depends on the project. 8. Yes, Thorough QA done for that. 1. Yes, we consider every issue. Risks are identified properly. Not in defined process. Not in define process. 6.3 6.4 6.3 Answers to the analysis questions 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. The final stage involved in this step is to identify the risk, nonrisk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Phase 3 - Testing 7 Step 7: Brainstorm and Prioritize Scenarios This is the first step in the Testing Phase of the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality Does all the stake holders involve in brainstorm? Are all scenarios collected after brainstorm? Are use case scenarios considered? Yes. Defined by Mother Company. Yes. Done by Mother Company. 153 SL No. ATAM Step Description Question Are growth scenarios considered? Are exploratory scenarios considered? Are the scenarios are prioritized? Are the voting process used for prioritizing scenarios? Answer Yes Yes Yes. Yes. No voting process is used. Naturally more demandable task considered first. attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. This is the last step in the ’Testing’ phase. In this step the highly 8 Step 8: Analyze the architectural voted quality attributes from the previous step are analyzed. The architectural approach that deals with these quality attributes are Approaches found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Phase 4 - Report the ATAM 9 Step 9: Report This is the final phase of the ATAM evaluation, where all the the ATAM information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches Are the new quality attributes considered found in the brainstorm session? Are the new scenarios considered? Are the new risks, non-risks, sensitivity points and trade off points included? Yes, Yes. Risk points are included. Is the utility tree prepared and reported? Are the generated scenarios reported? Are the analysis questions reported? Are the risks and non-risks reported? Is the architectural approach identified, finalized and reported? 1. No utility tree is prepared Yes we report by different diagrams and descriptions Yes analysis questions reported. Risks are reported. Yes reported to the Mother Company. 154 E High Level Feedback of Company - E SL ATAM Step Description No. Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. This step involves the description of the evaluation process of the 1 Step 1: Present ATAM. In this step, the evaluation leader presents general the ATAM information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. 2 Step 2: Present the business drivers In this step, the business goals of the architectural drivers of the system are mentioned. This step focuses on the business perspective of the system. It provides additional information on the functions of the system, the major stakeholders, the business goals and other constraints of the system. In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that represent the desired qualities of a system. Examples of these attributes include performance, reliability etc. Question Answer Do you practice architecture evaluation process? If yes do you have any process presentation meeting? If yes who are the participants of the meeting. How the participants are chosen? Yes 2.No define process. 3. All the related stake holders. 4. Among the development team 3 Step 3: Present the architecture to be evaluated Do you consider business goal or business drivers in architecture meeting? Do you consider the constraints of the business? Do you involve all the relevant stake holders like end-user, architect, developers, etc? Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? Does the presentation include architecture diagram in sufficiently detail? Does the presentation address technical constraints? Does the presentation address quality attributes requirement? Does the presentation address quality attribute requirements like performance, reliability, etc.? Yes, Project goals are defined. Yes Yes. Yes. Yes. Yes all diagrams are included. Yes. yes, quality attributes are defined. Yes. Phase 2 - Investigation and Analysis 155 SL ATAM Step Description Question Answer No. This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. 4 Step 4: Identify This step involves identifying the critical architectural approaches Does the architectural team explain the flow Yes the architectural that provide an understanding of the critical requirements of the of control of the architecture? Yes, it is addressed. approaches system. In this step, the architectural team explains the flow of Does the architectural approach address the Yes, every complex task are control of the architecture and provides a proper explanation of critical requirements? define in details breakdown how and whether the critical goal is met. Does the architectural team provide approach. explanation how the critical goal is met? Yes. Are the important quality attributes goals 5 Step 5: Generate In this phase of the evaluation, the system’s most important No utility tree made. clearly identified? quality attribute goals are identified, prioritized and refined. This the Quality We don’t maintain this. Is the utility tree made? Attribute Utility step is crucial because it focuses the attention of all the Yes, every steps are clearly Does the utility tree specify the system goals stakeholders and the evaluators on the different aspects of the tree. defined. clearly? architecture that are critical to the success of the system. This is Yes. Not always. Are the scenarios are clearly described? achieved through the construction of a utility tree. Yes. Every work flows are Are the quality attributes are prioritized? Are all the stake holders scenarios considered? considered. The utility tree provides a way of making the system goals more Not all quality attributes are Are all the quality attributes like the security, specific and more concrete. It also provides a comparison of the considered. performance, availability, reliability, importance of quality attribute goals relative to each other. No idea. Utility tree is not so These requirements are called scenarios. A scenario is a statement modifiability, portability, functionality, helpful. variability, conceptual integrity, etc. quality that illustrates the interaction between a stakeholder and the attribute considered? system. These scenarios are the quality goals against which the If utility tree is not practiced, is there other architecture will be judged. way in practice? Scenario Generation Scenario generation is an important preceding step to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the utility tree. The 156 SL No. ATAM Step Description utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, trade-off points. After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Question Answer 6 Step 6: Analyze the Architectural Approaches 1. How investigation done to define architectural approach? 2. How risk and non risks are identified? 1. No defined way 2. By analyzing requirement 6.1 6.1 Investigation of architectural approach 6.2 6.2 Creation of The next stage in this step involves in gathering analysis questions Does the architectural approach support the quality attribute? How does architectural approach support variability? How does architectural approach support modifiability? How does architectural approach support reliability? How does architectural approach support conceptual integrity? How does architectural approach support functionality? Do you create analysis questions? 1. Yes, we define the quality attributes. 2. By review. 3. At the time of design. 4. At the time of design. 5. At the time of design. 6. At the time of design. 1. Yes. 157 SL No. ATAM Step analysis questions Description generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Question Can the components of the architecture reused for future projects? Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Can any new application-specific functionality be added to the architecture? Can the system be modified in short time and cost-effective manner? Do the components interact properly? Are the answer and explanations to the questions analyzed? Are the risks and non-risks identified properly? Are the sensitivity points identified properly? Are the trade off points identified properly? Answer 2. Yes, Utility, Security components are reusable. 3. Yes, But depends on the project. 4. Yes 5. Yes. 6. Yes, but depends on the project. 7. Yes, but depends on the project. 8. Yes, Integration test done always. 1. Yes, we consider every issue. Yes, Risks are identified properly. Not in defined process. Not in define process. 6.3 6.4 6.3 Answers to the analysis questions 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. The final stage involved in this step is to identify the risk, nonrisk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Phase 3 - Testing 7 Step 7: Brainstorm and Prioritize Scenarios This is the first step in the Testing Phase of the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality Does all the stake holders involve in brainstorm? Are all scenarios collected after brainstorm? Are use case scenarios considered? No.. Yes. Yes Yes 158 SL No. ATAM Step Description Question Are growth scenarios considered? Are exploratory scenarios considered? Are the scenarios are prioritized? Are the voting process used for prioritizing scenarios? Answer attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. This is the last step in the ’Testing’ phase. In this step the highly 8 Step 8: Analyze the architectural voted quality attributes from the previous step are analyzed. The architectural approach that deals with these quality attributes are Approaches found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Phase 4 - Report the ATAM 9 Step 9: Report This is the final phase of the ATAM evaluation, where all the the ATAM information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches Not every time.. Yes. No voting process is used. Naturally more complex task considered first. Are the new quality attributes considered found in the brainstorm session? Are the new scenarios considered? Are the new risks, non-risks, sensitivity points and trade off points included? Yes, Considered and analyzed properly. Yes. Risk points are included. Is the utility tree prepared and reported? Are the generated scenarios reported? Are the analysis questions reported? Are the risks and non-risks reported? Is the architectural approach identified, finalized and reported? 1. No utility tree is prepared Yes we report by different diagrams and descriptions Yes analysis questions reported. Risks are reported. Yes, Final architectural approaches are documented and describes details to every stakeholder. 159 F Detail Level Feedback of Company - A SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. Do you practice 1 Step 1: Present the ATAM architecture evaluation process? This step involves the description of the evaluation process of the ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. 2 Step 2: Present the business drivers In this step, the business goals of the architectural drivers of the system are mentioned. This step focuses on the business perspective of the system. It provides additional information on the functions of the system, the major stakeholders, the business goals and other constraints of the system. Do you consider business goal or business drivers in architecture meeting? Yes No 1.1 Why do you practice architecture evaluation process? 1.2 Do you have any Architecture process presentation meeting? 1.3 If yes who are the participants of the meeting? 1.4 How the participants are chosen? 1.5 What are the problems you are facing for not practicing architecture evaluation process? 1.1. What goals are considered? For not doing the mistake of choosing wrong architecture. Normally we do. The Project members. Usually seniors are selected. N/A Yes No Do you consider the constraints of the business? Yes 1.2. Do you have any implications in the long run for not considering business goals? 2.1. What constraints are considered? The business goals depend on project. The business process fitting and flexibility, least time response, customizability, etc. N/A No Do you involve all the relevant stake holders Yes 2.2. Do you have any implications in the long run for not considering the constraints? 3. 1. Who are the relevant stakeholders? The resource constraint and time constraint are usually considered. N/A 160 SL No. ATAM Step & Description Question of the project? Answer No Supplementary Question 3.2. Do you have any implications in the long run for not considering all the relevant stakeholders? 1.1. Do you see any implications for not defining architecture beforehand? Supplementary Answer Not known. 3 Step 3: Present the architecture to be evaluated In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that represent the desired qualities of a system. Examples of these attributes include performance, reliability etc. Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? Yes No N/A Yes 2.1. What are the architectural approaches considered? No 2.2. Do you have any implications for not considering the architectural approaches? 3.1. Do you have any implications for not including architecture diagram in detail? 4.1. What are the technical constraints are considered? 4.2. Do you see any implications for not addressing technical constraints? 5.1. What are the quality attributes are considered? 5.2 Do you have any implications in the long run for not considering the quality attributes requirement? We usually consider 3-tiered architecture, web architecture, different patterns and frameworks, etc N/A Does the presentation include architecture diagram in sufficiently detail? Does the presentation address technical constraints? Yes No N/A Yes No Does the presentation address quality attributes requirement? Yes No We usually consider technological constraint, expertise level N/A We consider reliability, modifiability & performance N/A Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. 161 SL No. 4 ATAM Step & Description Step 4: Identify the architectural approaches This step involves identifying the critical architectural approaches that provide an understanding of the critical requirements of the system. In this step, the architectural team explains the flow of control of the architecture and provides a proper explanation of how and whether the critical goal is met. Question Does the architectural team explain the flow of control of the architecture? Does the architectural approach address the critical requirements? Answer Yes No Yes Supplementary Question Supplementary Answer 1.1 Do you see any implications for not explaining the flow of control? 2.1. What are the critical requirements are considered? 2.2. Does the architectural team provide explanation how the critical goal is met? 2.3 Do you have any implications in the long run for not considering the critical requirements? 1.1 What are the important qualities attributes goals? 1.2 What are the implications for not identifying these? 2.1 What are the advantages for this? 2.2 What are the implications of not practicing this? 3.1 What are the system goals considered for the utility tree? 3.2 What are the implications of not practicing this? 4.1 What are the scenarios to be considered? 4.2 What are the implications of not practicing this? 5.1 What are the criteria to prioritize quality attributes? 5.2 What are the implications for not prioritizing quality attributes? Not known. 2.1 The business functions, response time, scaleability, modifiability, performance, reusability, etc. 2.2 Usually not. N/A No 5 Step 5: Generate the Quality Attribute Utility tree. In this phase of the evaluation, the system’s most important quality attribute goals are identified, prioritized and refined. This step is crucial because it focuses the attention of all the stakeholders and the evaluators on the different aspects of the architecture that are critical to the success of the system. This is achieved through the construction of a utility tree. The utility tree provides a way of making the system goals more specific and more concrete. It also provides a comparison of the importance of quality attribute goals relative to each other. These requirements are called scenarios. A scenario is a statement that illustrates the interaction between a stakeholder and the system. These scenarios are the quality goals against which the architecture will be judged. Are the important quality attributes goals clearly identified? Yes No Is the utility tree made? Does the utility tree specify the system goals clearly? Are the scenarios are clearly described? Yes No Yes No Yes No Are the quality attributes are prioritized? Yes No Performance, Scalability, Security, Availability, Modifiablity N/A Not known. Quality requirements are not understood and met.. Usually use case scenarios are considered. N/A Depends on the project and customer desire. N/A 162 SL No. ATAM Step & Description Question Are all the stake holders scenarios considered? Are the quality attributes like the security, performance, availability, reliability, modifiability, portability, functionality, variability, conceptual integrity, etc. quality attribute considered? If utility tree is not practiced, is there other way in practice? Answer Yes No Yes Supplementary Question 6.1 What are the common scenarios considered? 6.2 What are the implications for not considering all the scenarios? 7.1 What are the attributes considered? 7.2 What are the implications for missing attributes? 7.3 What are the implications for not considering the quality attributes? Supplementary Answer Scenario Generation Scenario generation is an important preceding step to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the utility tree. The utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). 6 Step 6: Analyze the Architectural Approaches This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. No Suffers in the latter stage of the project. 7.1 Usually security, performance, functionality and reliability are considered. 7.2 Not known. N/A Yes 8.1 Give the descriptions of the practices. 8.2 What are the implications for practicing nothing like utility tree? 1.1 How investigation done to define architectural approach? 1.2 What are the implications of not doing so? 2.1 How risk and non risks are identified? 2.2 What are the implications of not identifying risks and non-risks? Project will suffer in the latter stage. Analyzed how the architectural approach meet the quality attributes. N/A Usually find out when and in situation the system can fail. N/A No 1. Do you analyze architectural approaches Yes No 2. Are risk and non risks identified? Yes No 163 SL No. ATAM Step & Description This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, tradeoff points. 6.1 Investigation of architectural approach After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Question Answer Supplementary Question Supplementary Answer 6.1 Does the architectural approach support the quality attribute? Yes 1.1 What are the quality attributes considered? 1.2 What are the implications of not practicing this? 2.1 What are the processes to identify the variability? 2.2 What are the implications of not practicing this? 3.1 What are the processes to identify the modifiability? No Does architectural approach support variability? How does architectural approach support modifiability? Yes No Yes Usually security, scalability, reliability and performance are considered. N/A Not very evident. Checking the ability to be modified, changing the different business process and functionality N/A Ensures the application can tolerate failure. N/A No Does architectural approach support reliability? Does architectural approach support conceptual integrity? Does architectural approach support functionality? Yes No Yes No Yes No 3.2 What are the implications of not practicing this? 4.1 What are the processes to identify the reliability? 4.2 What are the implications of not practicing this? 5.1 What are the processes to identify the conceptual integrity? 5.2 What are the implications of not practicing this? 6.1 What are the processes to identify the functionality of project? 6.2 What are the implications of not practicing this? Not evident. Checking the major functionality against the architecture. N/A 164 SL No. 6.2 ATAM Step & Description 6.2 Creation of analysis questions The next stage in this step involves in gathering analysis questions generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Question Do you create analysis questions? Answer Yes Supplementary Question 1.1 What are the steps followed to create analysis questions? 1.2. What are covered in analysis questions? 1.3 What are the implications of not practicing this? 2.1 What are the design level considerations for component reusability? 2.2 What are the implications of not practicing this? 3.1 What are the steps considered for the framework expansion? 3.2 What are the implications of not practicing this? 4.1 How the validation considered? Supplementary Answer 1.1. First quality attributes are identified. Then questions are prepared. 1.2 Depends on the project and identified quality attributes. N/A No Can the components of the architecture reused for future projects? Yes No Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Yes No Time is not reduced. Inability to expand according to clients need Depends on project. If high reliable project and enforce the validation. N/A Yes No Yes No Can any new application-specific functionality be added to the architecture? Yes 4.2 What are the implications of not practicing this? 5.1 What are the steps followed to identify the architectural behavior? 5.2 What are the implications of not practicing this? 6.1 How the decision taken at the time of design to add application specific functionality? 6.2 What are the implications of not practicing this? 7.1 How the time and cost calculated Gets unreliable. Architecture is designed to accommodate the new functionality. The modifiability and extensibility is considered. N/A No Can the system be Yes 165 SL No. ATAM Step & Description Question modified in short time and cost-effective manner? Do the components interact properly? Answer Supplementary Question for change management? 7.2 What are the implications of not practicing this? 8.1 How the component interaction tested? 8.2 What are the implications of not practicing this? 1.1 How the answer and explanations to the questions analyzed? 1.2 What are the implications of not practicing this? 1.1 What are the processes to identify the risk and non risks? 1.2. What are the common risks and non-risks? Supplementary Answer No Yes No Not known. Not known. The answer are discussed in the meeting from different angle. N/A 1.1Usually risks are identified. But no defined process is in place 1.2 Common risks are unavailability of client resource, failure of meeting the performance goals, quality requirements N/A 6.3 6.4 6.3 Answers to the analysis questions The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The final stage involved in this step is to identify the risk, non-risk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Are the answer and explanations to the questions analyzed? Are the risks and nonrisks identified properly? Yes No Yes No Are the sensitivity points identified properly? Yes No Are the trade off points identified properly? Yes No 1.3 What are the implications of not practicing this? 2.1 What are the processes to identify the sensitivity points? 2.2 What are the common sensitivity points? 2.3 What are the implications of not practicing this? 3.1 What are the processes to identify the trade off points? 3.2 What are the common trade-off points? 3.3 What are the implications of not practicing this? Issues important to the customer can be missed. Issues important to the customer can be missed. 166 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer Phase 3 – Testing 7 Step 7: Brainstorm and Prioritize Scenarios This is the first step in the Testing Phase of the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. Does all the stake holders involve in brainstorm? Are all scenarios collected after brainstorm? Yes No Yes No 1.1 What are the criteria to select stakeholders? 1.2 What are the implications of not practicing this? 2.1 Give some example of Scenarios? 2.2 What are the implications of not practicing this? 3.1 What are the cases of scenarios considered? 3.2 What are the implications of not practicing this? 4.1 What are the steps to consider growth scenarios? Some scenarios can be missed. Are use case scenarios considered? Yes No Some scenarios can be missed which can be evident during production. Major application functions are considered. N/A Usually considered to some extent. How much growth can happen in a certain period of time in terms of users, business volume, storage, etc. N/A Are growth scenarios considered? Yes No Are exploratory scenarios considered? Yes No Are the scenarios are prioritized? Yes 4.2 What are the implications of not practicing this? 5.1 What are the steps to consider exploratory scenarios? 5.2 What are the implications of not practicing this? 6.1 How the scenarios are prioritized? Not very evident. The scenarios are prioritized based on application functionality and stability requirement. N/A No Are the voting process used for prioritizing Yes 6.2 What are the implications of not practicing this? 7.1 What is the voting process for prioritizing scenarios? 167 SL No. ATAM Step & Description Question scenarios? Answer No Yes Supplementary Question 7.2 What are the implications of not using voting process? 1.1 How do you do that? Supplementary Answer The priority list may not reflect the actual need. Revise the already selected attributes and consider the new one as well. N/A For new scenarios all the quality attributes are considered and refined. N/A 8 Step 8: Analyze the architectural Approaches This is the last step in the ’Testing’ phase. In this step the highly voted quality attributes from the previous step are analyzed. The architectural approach that deals with these quality attributes are found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Are the new quality attributes considered found in the brainstorm session? Are the new scenarios considered? No Yes 1.2 What are the implications of not practicing this? 2.1 What are the processes to consider new scenarios? 2.2 What are the implications of not practicing this? 3.1 What are the processes to consider risk and non risk items? 3.2 What are the processes to consider sensitivity points? 3.3 What are the processes to consider trade off points? 3.4 What are the implications of not practicing this? 1.1 What are they? 1.2 What are the implications of not practicing this? No Are the new risks, non-risks, sensitivity points and trade off points included? Yes No Phase 4 - Report the ATAM 9 Step 9: Report the ATAM This is the final phase of the ATAM evaluation, where all the information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches The project may suffer during production. Is the utility tree prepared and reported? Yes No Are the generated scenarios reported? Yes 2.1 What are they? No Are the analysis Yes 2.2 What are the implications of not practicing this? 3.1 What are they? Not exactly used utility tree. But quality attributes, their importance were identified. No very evident implication. Different type of Use case scenarios is considered. Growth scenarios are also considered sometimes. N/A Findings of the analysis is 168 SL No. ATAM Step & Description Question questions reported? Answer Supplementary Question Supplementary Answer presented. No Are the risks and nonrisks reported? Yes No Is the architectural approach identified, finalized and reported? Yes 3.2 What are the implications of not practicing this? 4.1 What are they? 4.2 What are the implications of not practicing this? 5.1 What are the common approaches identified? 5.2 What are the implications of not practicing this? Different type of project risks are presented. N/A Client Server architecture, Web architecture, Data centric architecture are also used. N/A No 169 G Detail Level Feedback of Company - B SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. Do you practice 1 Step 1: Present the ATAM architecture evaluation process? This step involves the description of the evaluation process of the ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. Yes 1.1 Why do you practice architecture evaluation process? 1.2 Do you have any Architecture process presentation meeting? 1.3 If yes who are the participants of the meeting? 1.4 How the participants are chosen? No 1.5 What are the problems you are facing for not practicing architecture evaluation process? 1.1. What goals are considered? 2 Step 2: Present the business drivers In this step, the business goals of the architectural drivers of the system are mentioned. This step focuses on the business perspective of the system. It provides additional information on the functions of the system, the major stakeholders, the business goals and other constraints of the system. Do you consider business goal or business drivers in architecture meeting? Yes 1.1 It helps us track the design decision of architecture. 1.2 We collect all functionality of the project to present meeting. 1.3 System Analyst, Architect, Project Manager, Team Lead and Developers. 1.4 From the project criteria we select the most skilled people for the project from their historical work experience in the company. 1.5 If any change required in the project or any enhanced requirement comes it is difficult to plug the functionality. 1.1 Security, Performance, Availability, Future enhancement. No Do you consider the constraints of the business? Yes 1.2. Do you have any implications in the long run for not considering business goals? 2.1. What constraints are considered? 2.1 Business requirement clarification uncertainty, requirement complexity, lack of domain knowledge and technological uncertainty. No 2.2. Do you have any implications in 170 SL No. ATAM Step & Description Question Answer Supplementary Question the long run for not considering the constraints? 3. 1. Who are the relevant stakeholders? 3.2. Do you have any implications in the long run for not considering all the relevant stakeholders? 1.1. Do you see any implications for not defining architecture beforehand? Supplementary Answer Do you involve all the relevant stake holders of the project? Yes 3.1 System Analyst, Architect, Project Manager, Team Lead and Developers. No 3 Step 3: Present the architecture to be evaluated In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that represent the desired qualities of a system. Examples of these attributes include performance, reliability etc. Does the architecture team define any architecture before evaluation? Yes No Does the presentation address architectural approach? Yes 2.1. What are the architectural approaches considered? 1.1 Problem faced on further enhancement of the project, adding pluggable features and communicating with other applications. 2.1 Pipeline approach (small application), N-tier approach (mid to large application), client server approach (large application). No Does the presentation include architecture diagram in sufficiently detail? Yes 2.2. Do you have any implications for not considering the architectural approaches? 3.1 What are the diagrams you use? 3.1 Layered diagram, Layer to Layer communication diagram, best case sequence diagram for communication with each component, worst case sequence diagram for communication with each component and deployment diagram. No 3.2 Do you have any implications for not including architecture diagram in detail? 171 SL No. ATAM Step & Description Question Does the presentation address technical constraints? Answer Yes Supplementary Question 4.1. What are the technical constraints are considered? Supplementary Answer 4.1 Technical constrain includes technical uncertainty and communication agreement between other applications. No Does the presentation address quality attributes requirement? Yes 4.2. Do you see any implications for not addressing technical constraints? 5.1. What are the quality attributes are considered? 5.2 Do you have any implications in the long run for not considering the quality attributes requirement? 5.1 Performance, Security, Availability, Plugability and Future enhancement. No Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. 4 Step 4: Identify the architectural approaches Does the architectural Yes 1.1 How the flow of control 1.1 Flow of controls is This step involves identifying the critical team explain the flow determined? determined by different business architectural approaches that provide an of control of the workflows considering the best understanding of the critical requirements of the architecture? and worst case scenarios. system. In this step, the architectural team explains No 1.1 Do you see any implications for not the flow of control of the architecture and provides a explaining the flow of control? proper explanation of how and whether the critical goal is met. Does the architectural approach address the critical requirements? Yes 2.1. What are the critical requirements are considered? 2.1 Critical requirements includes complex business process and Algorithmic complexity for the business process. No 3. Does the Yes 2.3 Do you have any implications in the long run for not considering the critical requirements? 3.1 How the critical goals are 3.1 Critical goals are considered 172 SL No. ATAM Step & Description Question architectural team provide explanation how the critical goal is met? Answer Supplementary Question considered? Supplementary Answer by the complex business requirement and algorithmic complexity for the business goal. 3.2 Each complex requirement are break down by details analysis. 1.1 Important quality attributes are as follows: security, performance, availability, reliability, modifiability, portability. 5 Step 5: Generate the Quality Attribute Utility tree. In this phase of the evaluation, the system’s most important quality attribute goals are identified, prioritized and refined. This step is crucial because it focuses the attention of all the stakeholders and the evaluators on the different aspects of the architecture that are critical to the success of the system. This is achieved through the construction of a utility tree. The utility tree provides a way of making the system goals more specific and more concrete. It also provides a comparison of the importance of quality attribute goals relative to each other. These requirements are called scenarios. A scenario is a statement that illustrates the interaction between a stakeholder and the system. These scenarios are the quality goals against which the architecture will be judged. Scenario Generation Scenario generation is an important preceding step to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, Are the important quality attributes goals clearly identified? No Yes 3.2 What are the implications for this? 1.1 What are the important qualities attributes goals? No Is the utility tree made? Yes No 1.2 What are the implications for not identifying these? 2.1 What are the advantages for this? 2.2 What are the implications of not practicing this? 2.2 Quality attributes are not traceable at the design time. After started working we face the problem which can be reduced by this utility tree. Does the utility tree specify the system goals clearly? Yes No 3.1 What is the system goals considered for the utility tree? 3.2 What are the implications of not practicing this? 4.1 What are the scenarios to be considered? Are the scenarios are clearly described? Yes 3.2 Sometimes we have to redesign the system after work started. 4.1 Scenarios are described by best case, worst case and medium case sequence diagram of architecture. No Are the quality attributes are Yes 4.2 What are the implications of not practicing this? 5.1 What are the criteria to prioritize quality attributes? 5.1 Depends on the project business requirement we 173 SL No. ATAM Step & Description with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the utility tree. The utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). Question prioritized? Answer Supplementary Question Supplementary Answer prioritize quality attributes. 5.2 Depends on the project execution process. 5.3 Depends on the number of users of the project 5.4 Depends on the number of data transaction. 5.5 Depends on the project security. No Are all the stake holders scenarios considered? Yes 5.2 What are the implications for not prioritizing quality attributes? 6.1 What are the common scenarios considered? 6.2 What are the implications for not considering all the scenarios? 7.1 What are the attributes considered? 6.1 Stakeholders related to all business execution process are considered. No Are the quality attributes like the security, performance, availability, reliability, modifiability, portability, functionality, variability, conceptual integrity, etc. quality attribute considered? If utility tree is not practiced, is there other way in practice? Yes 7.2 What are the implications for missing attributes? No Yes 7.3 What are the implications for not considering the quality attributes? 8.1 Give the descriptions of the practices. 8.2 What are the implications for practicing nothing like utility tree? 7.1 Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. 7.2 Integration with other application some times creates problem. No 8.2 Architecture traceability is not possible and it creates rework after creating 174 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer architecture. 6 Step 6: Analyze the Architectural Approaches This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, tradeoff points. 6.1 Investigation of architectural approach After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. 1. Do you analyze architectural approaches Yes 1.1 How investigation done to define architectural approach? 1.1 We identify all the functionality and allocate the functionality in each step of investigation. No 2. Are risk and non risks identified? Yes 1.2 What are the implications of not doing so? 2.1 How risk and non risks are identified? 2.2 What are the implications of not identifying risks and non-risks? 2.1 Risk items are identified from the requirement and technological challenges. No 6.1 Does the architectural approach support the quality attribute? Yes 1.1 What are the quality attributes considered? 1.1 Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. No Does architectural approach support variability? How does Yes No Yes 1.2 What are the implications of not practicing this? 2.1 What are the processes to identify the variability? 2.2 What are the implications of not practicing this? 3.1 What are the processes to identify 2.2 Future enhancement creates problem. 3.1 Processes are as follows: 175 SL No. ATAM Step & Description Question architectural approach support modifiability? Answer Supplementary Question the modifiability? Supplementary Answer Requirement uncertainty, Functionality, Future enhancement, Plugability. No Does architectural approach support reliability? Yes 3.2 What are the implications of not practicing this? 4.1 What are the processes to identify the reliability? 4.1 Processes are as follows: Test each flow of execution of architecture, Test plugability of components. No Does architectural approach support conceptual integrity? Yes 4.2 What are the implications of not practicing this? 5.1 What are the processes to identify the conceptual integrity? 5.1 Considered at the time of design depending on the project category and business requirement. No Does architectural approach support functionality? Yes 5.2 What are the implications of not practicing this? 6.1 What are the processes to identify the functionality of project? 6.2 What are the implications of not practicing this? 1.1 What are the steps followed to create analysis questions? 6.1 Major functionalities are distinctly described from the details analysis of requirement. No 6.2 6.2 Creation of analysis questions The next stage in this step involves in gathering analysis questions generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Do you create analysis questions? Yes No Can the components of the architecture reused Yes 1.2. What are covered in analysis questions? 1.3 What are the implications of not practicing this? 2.1 What are the design level considerations for component 1.1 Steps are requirement analyze, requirement complexity consideration, requirement uncertainty and technical uncertainty. 1.2 Complex requirement, Complex business process. 2.1 Design level considerations are making the component more 176 SL No. ATAM Step & Description Question for future projects? Answer Supplementary Question reusability? Supplementary Answer specific and independent to work. No Can the framework be expanded in the future to accommodate a new application or new components? Yes 2.2 What are the implications of not practicing this? 3.1 What are the steps considered for the framework expansion? 3.1 Each component is made to work as independently without any dependency. 3.2 Input and Outputs are distinctly defined. No Will the system handle any input given by the user and handle invalid input? Yes 3.2 What are the implications of not practicing this? 4.1 How the validation considered? 4.1 Validation considered as from the business requirement. 4.2 Third party tools are used to make validation process easier. No Is the architecture consistent in behavior? Yes 4.2 What are the implications of not practicing this? 5.1 What are the steps followed to identify the architectural behavior? 5.1 Steps are requirement complexity, requirement uncertainty and future enhancement. No Can any new application-specific functionality be added to the architecture? Yes 5.2 What are the implications of not practicing this? 6.1 How the decision taken at the time of design to add application specific functionality? 6.1 Each component is created to work independently without any dependency, so any other application specific functionality can be added easily. No Can the system be modified in short time and cost-effective Yes No 6.2 What are the implications of not practicing this? 7.1 How the time and cost calculated for change management? 7.2 What are the implications of not 7.2 Sometimes we need to 177 SL No. ATAM Step & Description Question manner? Answer Supplementary Question practicing this? Supplementary Answer reengineer the application architecture which increases time and decreases quality. 8.1 Component interaction tested by unit testing. Do the components interact properly? Yes No 6.3 6.3 Answers to the analysis questions The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. Are the answer and explanations to the questions analyzed? Yes 8.1 How the component interaction tested? 8.2 What are the implications of not practicing this? 1.1 How the answer and explanations to the questions analyzed? 1.1 Answer and explanations to the questions analyzed by the details breakdown of requirement including complex and uncertain requirement, No 6.4 6.4 Find the risks, non-risks, sensitivity points, tradeoff points. The final stage involved in this step is to identify the risk, non-risk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Are the risks and nonrisks identified properly? Yes 1.2 What are the implications of not practicing this? 1.1 What are the processes to identify the risk and non risks? 1.2. What are the common risks and non-risks? 1.3 What are the implications of not practicing this? 2.1 What are the processes to identify the sensitivity points? 2.2 What are the common sensitivity points? 2.3 What are the implications of not practicing this? 1.1 The processes to identify the risk and non risks are from the requirement complexity, uncertainty and technological uncertainty. No Are the sensitivity points identified properly? Yes No 2.3 We can not verify the architectural approaches are how much feasible for the application. Are the trade off points identified properly? Yes 3.1 What are the processes to identify the trade off points? 3.2 What are the common trade-off points? 178 SL No. ATAM Step & Description Question Answer No Supplementary Question 3.3 What are the implications of not practicing this? Supplementary Answer 3.3 We can not verify the architectural approaches are how much feasible for the application. 3.3 We can not guaranty the architecture will work in every case of business requirement. Phase 3 – Testing 7 Step 7: Brainstorm and Prioritize Scenarios This is the first step in the Testing Phase of the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. Does all the stake holders involve in brainstorm? Yes No 1.1 What are the criteria to select stakeholders? 1.2 What are the implications of not practicing this? Are all scenarios collected after brainstorm? Yes 2.1 Give some example of Scenarios? 1.2 All members are not involved. Only development related members are involved so sometimes we can not identify the risk and uncertainty. 2.1 Examples like: Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. No Are use case scenarios considered? Yes No Are growth scenarios considered? Yes 2.2 What are the implications of not practicing this? 3.1 What are the cases of scenarios considered? 3.2 What are the implications of not practicing this? 4.1 What are the steps to consider growth scenarios? 3.1 Every high level case and Complex cases are considered. 4.1 We analyze the requirement and clarify with the client about the further enhancement of the system. Also project growths corresponding to every business workflow are considered. 179 SL No. ATAM Step & Description Question Answer No Supplementary Question 4.2 What are the implications of not practicing this? 5.1 What are the steps to consider exploratory scenarios? 5.2 What are the implications of not practicing this? 6.1 How the scenarios are prioritized? Supplementary Answer Are exploratory scenarios considered? Yes 5.1 Steps include the database system portability, Performance, Availability and security. No Are the scenarios prioritized? Yes 6.1 Corresponding to the complexity, functionality and business driver scenarios are prioritized. No Are the voting process used for prioritizing scenarios? Yes No 6.2 What are the implications of not practicing this? 7.1 What is the voting process for prioritizing scenarios? 7.2 What are the implications of not using voting process? 8 Step 8: Analyze the architectural Approaches This is the last step in the ’Testing’ phase. In this step the highly voted quality attributes from the previous step are analyzed. The architectural approach that deals with these quality attributes are found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes Are the new quality attributes considered found in the brainstorm session? Are the new scenarios considered? Yes 1.1 How do you do that? 7.2 Sometimes we don’t understand at what point we should start working. If the voting process introduce we can decide from where we have to start working. 1.1 We compare the new attributes with the old attributes and prioritize the new attributes. No Yes 1.2 What are the implications of not practicing this? 2.1 What are the processes to consider new scenarios? 2.2 What are the implications of not practicing this? 3.1 What are the processes to consider risk and non risk items? 2.1 We compare the existing scenario and new scenario and prioritize the scenarios. No Are the new risks, non-risks, sensitivity Yes 3.1 Processes to consider risk and non risk items are complex 180 SL No. ATAM Step & Description from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Question points and trade off points included? Answer Supplementary Question Supplementary Answer work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. Processes to consider sensitivity points are complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. Processes to consider trade off points are complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. 3.2 What are the processes to consider sensitivity points? 3.3 What are the processes to consider trade off points? No Phase 4 - Report the ATAM 9 Step 9: Report the ATAM This is the final phase of the ATAM evaluation, where all the information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches 3.4 What are the implications of not practicing this? 1.1 What are they? 1.2 What are the implications of not practicing this? Is the utility tree prepared and reported? Yes No Are the generated scenarios reported? Yes 2.1 What are they? 1.2 We can not prioritize our design and work. We can not identify the strength and weakness of architecture. 2.2 Best case sequence diagram worst case sequence diagram. Also other complex sequence diagram. No Are the analysis questions reported? Yes 2.2 What are the implications of not practicing this? 3.1 What are they? 3.1 Analysis question included new solutions of complex 181 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer requirements. No Are the risks and nonrisks reported? Yes 3.2 What are the implications of not practicing this? 4.1 What are they? 4.1 Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed No Is the architectural approach identified, finalized and reported? Yes 4.2 What are the implications of not practicing this? 5.1 What are the common approaches identified? 5.1 Layered approaches, N-Tier approaches, Client Server approaches and Pipeline approaches are identified, finalized and reported corresponding to the project type. No 5.2 What are the implications of not practicing this? 182 H Detail Level Feedback of Company - C SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. Do you practice 1 Step 1: Present the ATAM architecture evaluation process? This step involves the description of the evaluation process of the ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. Yes 1.1 Why do you practice architecture evaluation process? 1.2 Do you have any Architecture process presentation meeting? 1.3 If yes who are the participants of the meeting? Just to make sure that we are choosing the right architecture. Usually we have. Usually the Project members. Sometimes client technical person also. Depends on the project. Usually senior and talented members are chosen. N/A 1.4 How the participants are chosen? No 2 Step 2: Present the business drivers In this step, the business goals of the architectural drivers of the system are mentioned. This step focuses on the business perspective of the system. It provides additional information on the functions of the system, the major stakeholders, the business goals and other constraints of the system. Do you consider business goal or business drivers in architecture meeting? Yes 1.5 What are the problems you are facing for not practicing architecture evaluation process? 1.1. What goals are considered? No 1.2. Do you have any implications in the long run for not considering The business goals depend on project. Like for sales application, we considered diversified sales handling, business process change accommodation, least time response to customer, etc. N/A 183 SL No. ATAM Step & Description Question Answer Supplementary Question business goals? 2.1. What constraints are considered? 2.2. Do you have any implications in the long run for not considering the constraints? 3. 1. Who are the relevant stakeholders? 3.2. Do you have any implications in the long run for not considering all the relevant stakeholders? Supplementary Answer Do you consider the constraints of the business? Yes No The resource constraint and time constraint are usually considered. N/A Do you involve all the relevant stake holders of the project? Yes No We involve the stakeholders based on their availability. This sometimes create some problems in the later part of the project, sometimes after delivery. N/A 3 Step 3: Present the architecture to be evaluated In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that represent the desired qualities of a system. Examples of these attributes include performance, Does the architecture team define any architecture before evaluation? Does the presentation address architectural approach? Yes No 1.1. Do you see any implications for not defining architecture beforehand? 2.1. What are the architectural approaches considered? Yes No 2.2. Do you have any implications for not considering the architectural approaches? We usually consider n-tier web architecture, 3-tiered enterprise architecture, Sometimes consider well known framework, sometimes we consider plug-in, etc N/A Does the presentation Yes 184 SL No. ATAM Step & Description reliability etc. Question include architecture diagram in sufficiently detail? Does the presentation address technical constraints? Answer No Supplementary Question 3.1. Do you have any implications for not including architecture diagram in detail? 4.1. What are the technical constraints are considered? 4.2. Do you see any implications for not addressing technical constraints? 5.1. What are the quality attributes are considered? 5.2 Do you have any implications in the long run for not considering the quality attributes requirement? Supplementary Answer N/A Yes No We usually consider technological constraint, expertise level N/A Does the presentation address quality attributes requirement? Yes We usually consider reliability & performance N/A No Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. Yes Does the architectural 4 Step 4: Identify the architectural approaches team explain the flow This step involves identifying the critical of control of the architectural approaches that provide an This creates sometimes No 1.1 Do you see any implications architecture? understanding of the critical requirements of the misunderstanding among for not explaining the flow of system. In this step, the architectural team explains the stakeholders and control? the flow of control of the architecture and provides quality suffers in the end. a proper explanation of how and whether the critical goal is met. 2.1 The business Does the architectural Yes 2.1. What are the critical approach address the critical requirements? requirements are considered? 2.2. Does the architectural team provide explanation how the critical goal is met? 2.3 Do you have any implications in the long run for not considering functions, response time, scaleability, performance, etc. 2.2 Sometimes, high level N/A No 185 SL No. 5 ATAM Step & Description Question Answer Supplementary Question the critical requirements? 1.1 What are the important qualities attributes goals? 1.2 What are the implications for not identifying these? 2.1 What are the advantages for this? 2.2 What are the implications of not practicing this? 3.1 What are the system goals considered for the utility tree? 3.2 What are the implications of not practicing this? 4.1 What are the scenarios to be considered? Supplementary Answer Step 5: Generate the Quality Attribute Utility tree. In this phase of the evaluation, the system’s most important quality attribute goals are identified, prioritized and refined. This step is crucial because it focuses the attention of all the stakeholders and the evaluators on the different aspects of the architecture that are critical to the success of the system. This is achieved through the construction of a utility tree. The utility tree provides a way of making the system goals more specific and more concrete. It also provides a comparison of the importance of quality attribute goals relative to each other. These requirements are called scenarios. A scenario is a statement that illustrates the interaction between a stakeholder and the system. These scenarios are the quality goals against which the architecture will be judged. Scenario Generation Scenario generation is an important preceding step to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the Are the important quality attributes goals clearly identified? Is the utility tree made? Yes No Yes No Performance, Scalability, Security, Availability N/A Suffer in the long run. Some important quality attributes not considered. Does the utility tree specify the system goals clearly? Yes No Are the scenarios are clearly described? Yes No Are the quality attributes are prioritized? Are all the stake holders scenarios considered? Yes No Yes No 4.2 What are the implications of not practicing this? 5.1 What are the criteria to prioritize quality attributes? 5.2 What are the implications for not prioritizing quality attributes? 6.1 What are the common scenarios considered? 6.2 What are the implications for not considering all the scenarios? Quality requirements are not clearly understood and ensured. Usually use case scenarios are considered. Sometimes growth scenarios are also considered. N/A Depends on the business need and customer desire. N/A Are the quality Yes 7.1 What are the attributes Suffers in the latter stage of the project and sometimes during production. 7.1 Quality attributes 186 SL No. ATAM Step & Description utility tree. The utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). Question attributes like the security, performance, availability, reliability, modifiability, portability, functionality, variability, conceptual integrity, etc. quality attribute considered? Answer Supplementary Question considered? 7.2 What are the implications for missing attributes? Supplementary Answer depends on the project. Usually security, performance, functionality and reliability are considered. 7.2 The reusability and extensibility suffers. N/A No If utility tree is not practiced, is there other way in practice? Yes 7.3 What are the implications for not considering the quality attributes? 8.1 Give the descriptions of the practices. No 8.2 What are the implications for practicing nothing like utility tree? 1.1 How investigation done to define architectural approach? Determine the quality attributes, find out the importance to the project, and prioritize accordingly. N/A 6 Step 6: Analyze the Architectural Approaches This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions 1. Do you analyze architectural approaches Yes No 2. Are risk and non risks identified? Yes 1.2 What are the implications of not doing so? 2.1 How risk and non risks are identified? 2.2 What are the implications of not identifying risks and nonrisks? Analyzed how the architectural approach meet the quality attributes. N/A Usually find out when and in situation the system can fail. N/A No 187 SL No. ATAM Step & Description - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, tradeoff points. 6.1 Investigation of architectural approach After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Question Answer Supplementary Question Supplementary Answer 6.1 Does the architectural approach support the quality attribute? Yes 1.1 What are the quality attributes considered? No Does architectural approach support variability? How does architectural approach support modifiability? Yes No Yes 1.2 What are the implications of not practicing this? 2.1 What are the processes to identify the variability? 2.2 What are the implications of not practicing this? 3.1 What are the processes to identify the modifiability? Usually security, scalability, reliability and performance are considered. N/A Not very evident. Checking the ability to be modified, changing the different business process and functionality N/A Ensures the application can tolerate failure. N/A No Does architectural approach support reliability? Does architectural approach support conceptual integrity? Does architectural approach support functionality? Yes No Yes No Yes No 6.2 6.2 Creation of analysis questions Do you create analysis Yes 3.2 What are the implications of not practicing this? 4.1 What are the processes to identify the reliability? 4.2 What are the implications of not practicing this? 5.1 What are the processes to identify the conceptual integrity? 5.2 What are the implications of not practicing this? 6.1 What are the processes to identify the functionality of project? 6.2 What are the implications of not practicing this? 1.1 What are the steps followed to Not evident. Checking the major functionality against the architecture. N/A 1.1. First quality attributes 188 SL No. ATAM Step & Description The next stage in this step involves in gathering analysis questions generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Question questions? Answer Supplementary Question create analysis questions? 1.2. What are covered in analysis questions? Supplementary Answer are identified. Then questions are prepared. 1.2 Depends on the project and identified quality attributes. N/A No Can the components of the architecture reused for future projects? Yes No 1.3 What are the implications of not practicing this? 2.1 What are the design level considerations for component reusability? 2.2 What are the implications of not practicing this? 3.1 What are the steps considered for the framework expansion? 3.2 What are the implications of not practicing this? 4.1 How the validation considered? 4.2 What are the implications of not practicing this? 5.1 What are the steps followed to identify the architectural behavior? 5.2 What are the implications of not practicing this? 6.1 How the decision taken at the time of design to add application specific functionality? Repeated projects are not lowered. Time is not reduced. Can the framework be expanded in the future to accommodate a new application or new components? Will the system handle any input given by the user and handle invalid input? Is the architecture consistent in behavior? Yes No Inability to expand according to clients need Depends on project. If high reliable project and enforce the validation. N/A Yes No Yes No Can any new application-specific functionality be added to the architecture? Yes Gets unreliable. Architecture is designed to accommodate the new functionality. The modifiability and extensibility is 189 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer considered. N/A No Can the system be modified in short time and cost-effective manner? Yes No 6.2 What are the implications of not practicing this? 7.1 How the time and cost calculated for change management? 7.2 What are the implications of not practicing this? 8.1 How the component interaction tested? 8.2 What are the implications of not practicing this? 1.1 How the answer and explanations to the questions analyzed? 1.2 What are the implications of not practicing this? 1.1 What are the processes to identify the risk and non risks? 1.2. What are the common risks and non-risks? Sometimes fails to respond to client request. Creates bad impression. Do the components interact properly? Yes No 6.3 6.3 Answers to the analysis questions The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The final stage involved in this step is to identify the risk, non-risk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. Are the answer and explanations to the questions analyzed? Yes No Are the risks and nonrisks identified properly? Yes The changes do not take effect as expected. The answer are discussed in the meeting from different angle. N/A 1.1Usually risks are identified. But no defined process is in place 1.2 Common risks are unavailability of client resource, failure of meeting the performance goals, quality requirements N/A 6.4 No Are the sensitivity points identified properly? Yes 1.3 What are the implications of not practicing this? 2.1 What are the processes to identify the sensitivity points? 2.2 What are the common sensitivity points? 190 SL No. ATAM Step & Description If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Question Answer No Supplementary Question 2.3 What are the implications of not practicing this? 3.1 What are the processes to identify the trade off points? 3.2 What are the common tradeoff points? 3.3 What are the implications of not practicing this? 1.1 What are the criteria to select stakeholders? 1.2 What are the implications of not practicing this? 2.1 Give some example of Scenarios? 2.2 What are the implications of not practicing this? Supplementary Answer Issues important to the customer can be missed. Are the trade off points identified properly? Yes No Phase 3 – Testing 7 Step 7: Brainstorm and Prioritize Scenarios This is the first step in the Testing Phase of the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. Issues important to the customer can be missed. Does all the stake holders involve in brainstorm? Are all scenarios collected after brainstorm? Yes No Yes Some scenarios can be missed. No Are use case scenarios considered? Yes No Are growth scenarios considered? Yes 3.1 What are the cases of scenarios considered? 3.2 What are the implications of not practicing this? 4.1 What are the steps to consider growth scenarios? Some scenarios can be missed which can be evident during production. Major application functions are considered. N/A Usually considered to some extent. How much growth can happen in a certain period of time in terms of users, business volume, storage, etc. N/A No Are exploratory Yes 4.2 What are the implications of not practicing this? 5.1 What are the steps to consider 191 SL No. ATAM Step & Description Question scenarios considered? Answer Supplementary Question exploratory scenarios? 5.2 What are the implications of not practicing this? 6.1 How the scenarios are prioritized? Supplementary Answer No Are the scenarios are prioritized? Yes Not very evident. The scenarios are prioritized based on application functionality and stability requirement. N/A No Are the voting process used for prioritizing scenarios? 8 Step 8: Analyze the architectural Approaches This is the last step in the ’Testing’ phase. In this step the highly voted quality attributes from the previous step are analyzed. The architectural approach that deals with these quality attributes are found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Are the new quality attributes considered found in the brainstorm session? Yes No Yes 6.2 What are the implications of not practicing this? 7.1 What is the voting process for prioritizing scenarios? 7.2 What are the implications of not using voting process? 1.1 How do you do that? No Are the new scenarios considered? Yes 1.2 What are the implications of not practicing this? 2.1 What are the processes to consider new scenarios? 2.2 What are the implications of not practicing this? 3.1 What are the processes to consider risk and non risk items? 3.2 What are the processes to consider sensitivity points? 3.3 What are the processes to consider trade off points? 3.4 What are the implications of not practicing this? The priority list may not reflect the actual need. Revise the already selected attributes and consider the new one as well. N/A For new scenarios all the quality attributes are considered and refined. N/A No Are the new risks, nonrisks, sensitivity points and trade off points included? Yes No The project sometimes suffer during production and need costly 192 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer modification. Phase 4 - Report the ATAM 9 Step 9: Report the ATAM This is the final phase of the ATAM evaluation, where all the information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches Is the utility tree prepared and reported? Yes No 1.1 What are they? 1.2 What are the implications of not practicing this? Are the generated scenarios reported? Yes 2.1 What are they? No Are the analysis questions reported? Yes No Are the risks and nonrisks reported? Yes 2.2 What are the implications of not practicing this? 3.1 What are they? 3.2 What are the implications of not practicing this? 4.1 What are they? Not exactly used utility tree. But quality attributes, their importance were identified. No very evident implication. Different type of scenarios are considered under the category of Use case scenarios. Growth scenarios are also considered. N/A Normally findings of the analysis is presented. No Is the architectural approach identified, finalized and reported? Yes 4.2 What are the implications of not practicing this? 5.1 What are the common approaches identified? Different type of project risks as well as organizational risks are presented. N/A Usually Client Server architecture, Event Driven architecture and Web architecture are considered. Sometimes data centric architecture, 193 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer Service Oriented architecture, Plug in, etc are also used. N/A No 5.2 What are the implications of not practicing this? 194 I Detail Level Feedback of Company - D  SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. Do you practice 1 Step 1: Present the ATAM architecture evaluation process? This step involves the description of the evaluation process of the ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. Yes 1.1 Why do you practice architecture evaluation process? 1.2 Do you have any Architecture process presentation meeting? 1.3 If yes who are the participants of the meeting? 1.4 How the participants are chosen? No 1.5 What are the problems you are facing for not practicing architecture evaluation process? 2 Step 2: Present the business drivers In this step, the business goals of the architectural drivers of the system are mentioned. This step Do you consider business goal or business drivers in architecture meeting? Yes 1.1. What goals are considered? 1.1 It helps us make the design decision of architecture in a standard way. 1.2 We collect all features of the project. Collect expected enhancement possibilities information. 1.3 Business System Analyst, Business System Architect Team leads, developers. 1.4 Every one who is and will be related to the development of the project. Experience of doing same sort of project. 1.5 If any change required in the project or any enhanced requirement comes it is difficult to plug the functionality. 1.1 Security, Performance, Availability, Future enhancement. 195 SL No. ATAM Step & Description focuses on the business perspective of the system. It provides additional information on the functions of the system, the major stakeholders, the business goals and other constraints of the system. Question Answer No Supplementary Question 1.2. Do you have any implications in the long run for not considering business goals? 2.1. What constraints are considered? Supplementary Answer Do you consider the constraints of the business? Yes 2.1 Business requirement clarification uncertainty, requirement complexity, lack of domain knowledge and technological uncertainty. No Do you involve all the relevant stake holders of the project? Yes 2.2. Do you have any implications in the long run for not considering the constraints? 3. 1. Who are the relevant stakeholders? 3.1 Business System Analyst, Business System Architect, Team Leads, developers. No 3 Step 3: Present the architecture to be evaluated In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that Does the architecture team define any architecture before evaluation? Yes 3.2. Do you have any implications in the long run for not considering all the relevant stakeholders? 1.1 How the architecture teams define architecture before evaluation? 1.1 Analyze the initial requirement and define the initial level of architecture to be evaluated. No Does the presentation address architectural approach? Yes 1.2. Do you see any implications for not defining architecture beforehand? 2.1. What are the architectural approaches considered? 2.1 Pipeline approach (small application), N-tier approach (mid to large application), client server approach (large application). ETL 196 SL No. ATAM Step & Description represent the desired qualities of a system. Examples of these attributes include performance, reliability etc. Question Answer Supplementary Question Supplementary Answer application architecture. No Does the presentation include architecture diagram in sufficiently detail? Yes 2.2. Do you have any implications for not considering the architectural approaches? 3.1 What are the diagrams you use? 3.1 Top level and process level architecture with sufficient technical details. Layered diagram, Layer to Layer communication diagram, best case sequence diagram for communication with each component, worst case sequence diagram for communication with each component and deployment diagram. No Does the presentation address technical constraints? Yes 3.2 Do you have any implications for not including architecture diagram in detail? 4.1. What are the technical constraints are considered? 4.1 Technical constrain includes technical uncertainty and communication agreement between other applications. No Does the presentation address quality attributes requirement? Yes 4.2. Do you see any implications for not addressing technical constraints? 5.1. What are the quality attributes are considered? 5.1 Performance, Security, Availability, Plugability, reliability, 197 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer interchangeability, modifiability and Future enhancement. No 5.2 Do you have any implications in the long run for not considering the quality attributes requirement? Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. Yes 1.1 How the flow of control 1.1 Every work flows are Does the architectural 4 Step 4: Identify the architectural approaches determined? explained. Flow of team explain the flow This step involves identifying the critical controls is determined by of control of the architectural approaches that provide an different business architecture? understanding of the critical requirements of the workflows considering system. In this step, the architectural team explains the best and worst case the flow of control of the architecture and provides scenarios. a proper explanation of how and whether the critical goal is met. No 1.1 Do you see any implications for not explaining the flow of control? Does the architectural approach address the critical requirements? Yes 2.1. What are the critical requirements are considered? 2.1 Review process is used to determine a critical requirement includes complex business process and Algorithmic complexity for the business process. No 3. Does the architectural team Yes 2.3 Do you have any implications in the long run for not considering the critical requirements? 3.1 How the critical goals are considered? 3.1 Critical goals are considered by the 198 SL No. ATAM Step & Description Question provide explanation how the critical goal is met? Answer Supplementary Question Supplementary Answer complex business requirement and algorithmic complexity and system requirement for the business goal. 3.2 Each complex requirement are break down by details analysis. No 5 Step 5: Generate the Quality Attribute Utility tree. In this phase of the evaluation, the system’s most important quality attribute goals are identified, prioritized and refined. This step is crucial because it focuses the attention of all the stakeholders and the evaluators on the different aspects of the architecture that are critical to the success of the system. This is achieved through the construction of a utility tree. The utility tree provides a way of making the system goals more specific and more concrete. It also provides a comparison of the importance of quality attribute goals relative to each other. These requirements are called scenarios. A scenario is a statement that illustrates the interaction between a stakeholder and the system. These scenarios are the quality goals against which the architecture will be judged. Scenario Generation Scenario generation is an important preceding step Are the important quality attributes goals clearly identified? Yes 3.2 What are the implications for this? 1.1 What are the important qualities attributes goals? 1.1 Important quality attributes are as follows: security, performance, availability, reliability, modifiability, portability. No Is the utility tree made? Yes No 1.2 What are the implications for not identifying these? 2.1 What are the advantages for this? 2.2 What are the implications of not practicing this? 2.2 Quality attributes are not traceable at the design time. After started working we face the problem which can be reduced by this utility tree. Does the utility tree specify the system goals clearly? Yes No 3.1 What is the system goals considered for the utility tree? 3.2 What are the implications of not practicing this? 3.2 Sometimes we have to redesign the system after work started. This creates additional rework which increases development 199 SL No. ATAM Step & Description to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the utility tree. The utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). Question Answer Supplementary Question Supplementary Answer time. 4.1 Scenarios are described by best case, worst case and medium case sequence diagram of architecture. Are the scenarios are clearly described? Yes 4.1 What are the scenarios to be considered? No Are the quality attributes are prioritized? Yes 4.2 What are the implications of not practicing this? 5.1 What are the criteria to prioritize quality attributes? 5.1 Depends on the project business requirement we prioritize quality attributes. 5.2 Depends on the number of users of the project. 5.3 Depends on the number of data transaction. 5.4 Depends on the project security. 5.5 Depends on the project execution process. No Are all the stake holders scenarios considered? Yes 5.2 What are the implications for not prioritizing quality attributes? 6.1 What are the common scenarios considered? 6.2 What are the implications for not considering all the scenarios? 7.1 What are the attributes considered? 6.1 Stakeholders related to all business execution process are considered. No Are the quality attributes like the security, performance, availability, reliability, Yes 7.1 Attributes are as follows: security, performance, availability, reliability, modifiability, 200 SL No. ATAM Step & Description Question modifiability, portability, functionality, variability, conceptual integrity, etc. quality attribute considered? Answer Supplementary Question Supplementary Answer portability, functionality, variability. 7.2 Integration with other application sometimes creates problem. 7.2 What are the implications for missing attributes? No 7.3 What are the implications for not considering the quality attributes? 8.1 Give the descriptions of the practices. 8.2 What are the implications for practicing nothing like utility tree? 1.1 How investigation done to define architectural approach? If utility tree is not practiced, is there other way in practice? Yes No 6 Step 6: Analyze the Architectural Approaches This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the trade-off points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, tradeoff points. 1. Do you analyze architectural approaches Yes 8.2 Architecture traceability is not possible and it creates rework after creating architecture. 1.1 We identify all the functionality and allocate the functionality in each step of investigation. No 2. Are risk and non risks identified? Yes 1.2 What are the implications of not doing so? 2.1 How risk and non risks are identified? 2.1 Risk items are identified from the requirement and technological challenges. No 2.2 What are the implications of not identifying risks and nonrisks? 201 SL No. 6.1 ATAM Step & Description 6.1 Investigation of architectural approach After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Question Does the architectural approach support the quality attribute? Answer Yes Supplementary Question 1.1 What are the quality attributes considered? Supplementary Answer 1.1 Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. No Does architectural approach support variability? Yes 1.2 What are the implications of not practicing this? 2.1 What are the processes to identify the variability? 2.2 What are the implications of not practicing this? 3.1 What are the processes to identify the modifiability? 2.1 Through review variability can be identified. No How does architectural approach support modifiability? Yes 3.1 Processes are as follows: Requirement uncertainty, Functionality, Future enhancement, Plugability. No Does architectural approach support reliability? Yes 3.2 What are the implications of not practicing this? 4.1 What are the processes to identify the reliability? 4.1 Processes are as follows: Test each flow of execution of architecture, Test plugability of components. No Does architectural approach support conceptual integrity? Yes 4.2 What are the implications of not practicing this? 5.1 What are the processes to identify the conceptual integrity? 5.1 Considered at the time of design depending on the project category and business requirement. No 5.2 What are the implications of not practicing this? 202 SL No. ATAM Step & Description Question Does architectural approach support functionality? Answer Yes Supplementary Question 6.1 What are the processes to identify the functionality of project? 6.2 What are the implications of not practicing this? 1.1 What are the steps followed to create analysis questions? 1.2. What are covered in analysis questions? 1.3 What are the implications of not practicing this? Supplementary Answer 6.1 Major functionalities are distinctly described from the details analysis of requirement. No 6.2 6.2 Creation of analysis questions The next stage in this step involves in gathering analysis questions generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Do you create analysis questions? Yes No Can the components of the architecture reused for future projects? Yes 2.1 What are the design level considerations for component reusability? 1.3 Requirement complexity and uncertainty can’t be identified properly. 2.1 Design level considerations are making the component more specific and independent to work. No Can the framework be expanded in the future to accommodate a new application or new components? Yes 2.2 What are the implications of not practicing this? 3.1 What are the steps considered for the framework expansion? 3.1 Each component is made to work as independently without any dependency. 3.2 Input and Outputs are distinctly defined. No Will the system handle any input given by the user and handle invalid input? Yes 3.2 What are the implications of not practicing this? 4.1 How the validation considered? 4.1 Validation considered as from the business requirement. 4.2 Validation application 203 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer block is used to validation. No Is the architecture consistent in behavior? Yes 4.2 What are the implications of not practicing this? 5.1 What are the steps followed to identify the architectural behavior? 5.2 What are the implications of not practicing this? 6.1 How the decision taken at the time of design to add application specific functionality? 5.1 Steps are requirement complexity, requirement uncertainty and future enhancement. No Can any new application-specific functionality be added to the architecture? Yes 6.1 Each component is created to work independently without any dependency, so any other application specific functionality can be added easily. No Can the system be modified in short time and cost-effective manner? Yes 6.2 What are the implications of not practicing this? 7.1 How the time and cost calculated for change management? 7.1 Time and cost consideration for the change management considered during the design time. So any new change issue can’t do so much harm to the system. No Do the components interact properly? Yes 7.2 What are the implications of not practicing this? 8.1 How the component interaction tested? 8.2 What are the implications of not practicing this? 8.1 Component interaction tested by unit testing. No 204 SL No. 6.3 ATAM Step & Description 6.3 Answers to the analysis questions The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. Question Are the answer and explanations to the questions analyzed? Answer Yes Supplementary Question 1.1 How the answer and explanations to the questions analyzed? Supplementary Answer 1.1 Answer and explanations to the questions analyzed by the details breakdown of requirement including complex and uncertain requirement, No 6.4 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The final stage involved in this step is to identify the risk, non-risk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Are the risks and nonrisks identified properly? Yes 1.2 What are the implications of not practicing this? 1.1 What are the processes to identify the risk and non risks? 1.2. What are the common risks and non-risks? 1.1 The processes to identify the risk and non risks are from the requirement complexity, uncertainty and technological uncertainty. No Are the sensitivity points identified properly? Yes No 1.3 What are the implications of not practicing this? 2.1 What are the processes to identify the sensitivity points? 2.2 What are the common sensitivity points? 2.3 What are the implications of not practicing this? 2.3 We cannot verify the architectural approaches are how much feasible for the application. Are the trade off points identified properly? Yes No 3.1 What are the processes to identify the trade off points? 3.2 What are the common tradeoff points? 3.3 What are the implications of not practicing this? 3.3 We cannot verify the architectural approaches are how much feasible for the application. 205 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer 3.3 We cannot guaranty the architecture will work in every case of business requirement. Phase 3 – Testing 7 Step 7: Brainstorm and Prioritize Scenarios This is the first step in the Testing Phase of the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. Does all the stake holders involve in brainstorm? Are all scenarios collected after brainstorm? Yes No Yes 1.1 What are the criteria to select stakeholders? 1.2 What are the implications of not practicing this? 2.1 Give some example of Scenarios? 1.1 All related members are selected. 2.1 Examples like: Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. No Are use case scenarios considered? Yes 2.2 What are the implications of not practicing this? 3.1 What are the cases of scenarios considered? 3.2 What are the implications of not practicing this? 4.1 What are the steps to consider growth scenarios? 3.1 Every high level case and Complex cases are considered. No Are growth scenarios considered? Yes 4.1 We analyze the requirement and clarify with the client about the further enhancement of the system. Also project growths corresponding to every business workflow are considered. No 4.2 What are the implications of 206 SL No. ATAM Step & Description Question Answer Supplementary Question not practicing this? 5.1 What are the steps to consider exploratory scenarios? Supplementary Answer Are exploratory scenarios considered? Yes 5.1 Steps include the database system portability, Performance, Availability and security. No Are the scenarios prioritized? Yes 5.2 What are the implications of not practicing this? 6.1 How the scenarios are prioritized? 6.1 Corresponding to the complexity, functionality and business driver scenarios are prioritized. No Are the voting process used for prioritizing scenarios? Yes No 6.2 What are the implications of not practicing this? 7.1 What is the voting process for prioritizing scenarios? 7.2 What are the implications of not using voting process? 8 Step 8: Analyze the architectural Approaches This is the last step in the ’Testing’ phase. In this step the highly voted quality attributes from the previous step are analyzed. The architectural approach that deals with these quality attributes are found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality Are the new quality attributes considered found in the brainstorm session? Yes 1.1 How do you do that? 7.2 Sometimes we don’t understand at what point we should start working. If the voting process introduce we can decide from where we have to start working. 1.1 We compare the new attributes with the old attributes and prioritize the new attributes. No Are the new scenarios considered? Yes 1.2 What are the implications of not practicing this? 2.1 What are the processes to consider new scenarios? 2.1 We compare the existing scenario and new scenario and prioritize the scenarios. No 2.2 What are the implications of 207 SL No. ATAM Step & Description attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, non-risks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Question Answer Supplementary Question not practicing this? 3.1 What are the processes to consider risk and non risk items? Supplementary Answer Are the new risks, nonrisks, sensitivity points and trade off points included? Yes 3.2 What are the processes to consider sensitivity points? 3.3 What are the processes to consider trade off points? 3.1 Processes to consider risk and non risk items are complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. Processes to consider sensitivity points are complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. Processes to consider trade off points are complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. No Phase 4 - Report the ATAM 9 Step 9: Report the ATAM 3.4 What are the implications of not practicing this? 1.1 What are they? 1.2 What are the implications of Is the utility tree prepared and reported? Yes No 1.2 We can not prioritize 208 SL No. ATAM Step & Description This is the final phase of the ATAM evaluation, where all the information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches Question Answer Supplementary Question not practicing this? Supplementary Answer our design and work. We can not identify the strength and weakness of architecture. 2.2 Best case sequence diagram worst case sequence diagram. Also other complex sequence diagram. Are the generated scenarios reported? Yes 2.1 What are they? No Are the analysis questions reported? Yes 2.2 What are the implications of not practicing this? 3.1 What are they? 3.1 Analysis question included new solutions of complex requirements. No Are the risks and nonrisks reported? Yes 3.2 What are the implications of not practicing this? 4.1 What are they? 4.1 Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed No Is the architectural approach identified, finalized and reported? Yes 4.2 What are the implications of not practicing this? 5.1 What are the common approaches identified? 5.1 Layered approaches, N-Tier approaches, Client Server approaches and Pipeline approaches are identified, finalized and reported corresponding to the project type. These are identified by the mother company. 209 SL No. ATAM Step & Description Question Answer No Supplementary Question 5.2 What are the implications of not practicing this? Supplementary Answer 210 J Detail Level Feedback of Company - E SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer Phase 1 - Presentation This is the initial phase in the evaluation of software architectures using the ATAM. Do you practice 1 Step 1: Present the ATAM architecture evaluation This step involves the description of the evaluation process process? of the ATAM. In this step, the evaluation leader presents general information about the ATAM process to all the concerned participants. The leader explains the analysis techniques used in the evaluation, and the expected outcome of the evaluation. The leader addresses any concerns, expectations or questions of the assembled participants. Yes 1.1 Why do you practice architecture evaluation process? 1.2 Do you have any Architecture process presentation meeting? 1.3 If yes who are the participants of the meeting? 1.4 How the participants are chosen? No 1.5 What are the problems you are facing for not practicing architecture evaluation process? 2 Step 2: Present the business drivers In this step, the business goals of the architectural drivers of the system are mentioned. This step focuses on the business perspective of the system. It provides additional information on the functions of the system, the major stakeholders, the business goals and other constraints of the system. Do you consider business goal or business drivers in architecture meeting? Yes 1.1. What goals are considered? 1.1 It helps us make the design decision of architecture in a standard way. 1.2 We collect all features of the project. 1.3 Development team members. 1.4 Every one who is and will be related to the development of the project. 1.5 If any change required in the project or any enhanced requirement comes it is difficult to plug the functionality. 1.1 Security, Performance, Availability, Future enhancement. No Do you consider the constraints of the business? Yes 1.2. Do you have any implications in the long run for not considering business goals? 2.1. What constraints are considered? 2.1 Business requirement clarification uncertainty, requirement complexity, lack of domain knowledge and technological 211 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer uncertainty. No Do you involve all the relevant stake holders of the project? Yes 2.2. Do you have any implications in the long run for not considering the constraints? 3. 1. Who are the relevant stakeholders? 3.1 End users are participated through their project manager and the development team. No 3 Step 3: Present the architecture to be evaluated In this step, the architecture team describes the architecture to be evaluated. It focuses on the architecture, the time availability for the evaluation, and the quality requirements of the architecture under investigation. The architecture presentation in this step is important because it affects the quality of the analysis. Some of the key issues addressed in this presentation are: the technical constraints, other systems that interact with the system being evaluated, and the architectural approaches implemented in order to meet the quality attributes. The quality attributes of a system are the issues that represent the desired qualities of a system. Examples of these attributes include performance, reliability etc. Does the architecture team define any architecture before evaluation? Yes 3.2. Do you have any implications in the long run for not considering all the relevant stakeholders? 1.1 How the architecture teams define architecture before evaluation? 1.2. Do you see any implications for not defining architecture beforehand? 2.1. What are the architectural approaches considered? 1.1 Analyze the initial requirement and define the architecture to be evaluated. No Does the presentation address architectural approach? Yes 2.1 Pipeline approach (small application), N-tier approach (mid to large application), client server approach (large application). Com+ application architecture. No Does the presentation include architecture diagram in sufficiently detail? Yes 2.2. Do you have any implications for not considering the architectural approaches? 3.1 What are the diagrams you use? 3.1 Top level and process level architecture with sufficient technical details. Layered diagram, Layer to Layer communication 212 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer diagram, best case sequence diagram for communication with each component, worst case sequence diagram for communication with each component and deployment diagram. No Does the presentation address technical constraints? Yes 3.2 Do you have any implications for not including architecture diagram in detail? 4.1. What are the technical constraints are considered? 4.1 Technical constrain includes technical uncertainty and communication agreement between other applications. No Does the presentation address quality attributes requirement? Yes 4.2. Do you see any implications for not addressing technical constraints? 5.1. What are the quality attributes are considered? 5.1 Performance, Security, Availability, Plugability, reliability, modifiability and Future enhancement. No 5.2 Do you have any implications in the long run for not considering the quality attributes requirement? Phase 2 - Investigation and Analysis This is the second phase in the evaluation of an architecture using ATAM technique. In this phase a thorough investigation is held of the important issues to be concentrated on during the evaluation. This phase is sub-divided into three steps. Yes 1.1 How the flow of control 1.1 Every work flows are Does the architectural 4 Step 4: Identify the architectural approaches determined? explained. Flow of controls team explain the flow This step involves identifying the critical architectural is determined by different of control of the approaches that provide an understanding of the critical business workflows architecture? requirements of the system. In this step, the architectural 213 SL No. ATAM Step & Description team explains the flow of control of the architecture and provides a proper explanation of how and whether the critical goal is met. Question Answer Supplementary Question Supplementary Answer considering the best and worst case scenarios. No 1.1 Do you see any implications for not explaining the flow of control? Does the architectural approach address the critical requirements? Yes 2.1. What are the critical requirements are considered? 2.1 Review process is used to determine a critical requirement includes complex business process and Algorithmic complexity for the business process. No 3. Does the architectural team provide explanation how the critical goal is met? Yes 2.3 Do you have any implications in the long run for not considering the critical requirements? 3.1 How the critical goals are considered? 3.1 Critical goals are considered by the complex business requirement and algorithmic complexity and system requirement for the business goal. 3.2 Each complex requirement are break down by details analysis. No 5 Step 5: Generate the Quality Attribute Utility tree. In this phase of the evaluation, the system’s most important quality attribute goals are identified, prioritized and refined. This step is crucial because it focuses the Are the important quality attributes goals clearly identified? Yes 3.2 What are the implications for this? 1.1 What are the important qualities attributes goals? 1.1 Important quality attributes are as follows: security, performance, availability, reliability, modifiability, portability. 214 SL No. ATAM Step & Description attention of all the stakeholders and the evaluators on the different aspects of the architecture that are critical to the success of the system. This is achieved through the construction of a utility tree. The utility tree provides a way of making the system goals more specific and more concrete. It also provides a comparison of the importance of quality attribute goals relative to each other. These requirements are called scenarios. A scenario is a statement that illustrates the interaction between a stakeholder and the system. These scenarios are the quality goals against which the architecture will be judged. Scenario Generation Scenario generation is an important preceding step to the utility tree creation. Utility tree generation The utility tree contains ’utility’ as the root node, with the quality attributes forming the secondary level of the utility tree. Under each of the quality attributes, specific quality attribute refinements are included that provide a more precise description of the scenarios. The latter form the leaf nodes in the utility tree. The utility tree is prioritized along two dimensions: the importance of each scenario to the success of the system and an estimation of the degree of difficulty posed by the achievement of this scenario (from the architect’s perspective). The prioritization in the utility tree is based on relative rankings: High (H), Medium (M) and Low (L). Question Answer No Supplementary Question 1.2 What are the implications for not identifying these? 2.1 What are the advantages for this? 2.2 What are the implications of not practicing this? Supplementary Answer Is the utility tree made? Yes No 2.2 Quality attributes are not traceable at the design time. After started working we face the problem which can be reduced by this utility tree. Does the utility tree specify the system goals clearly? Yes No 3.1 What is the system goals considered for the utility tree? 3.2 What are the implications of not practicing this? Are the scenarios are clearly described? Yes 4.1 What are the scenarios to be considered? 3.2 Sometimes we have to redesign the system after work started. This creates additional rework which increases development time. 4.1 Scenarios are described by best case, worst case and medium case sequence diagram of architecture. No Are the quality attributes are prioritized? Yes 4.2 What are the implications of not practicing this? 5.1 What are the criteria to prioritize quality attributes? 5.1 Depends on the project security. 5.2 Depends on the number of users of the project. 5.3 Depends on the project execution process. 5.4 Depends on the number of data transaction. 5.5 Depends on the project 215 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer business requirement we prioritize quality attributes. No Are all the stake holders scenarios considered? Yes 5.2 What are the implications for not prioritizing quality attributes? 6.1 What are the common scenarios considered? 6.2 What are the implications for not considering all the scenarios? 7.1 What are the attributes considered? 6.1 Stakeholders related to all business execution process are considered. No Are the quality attributes like the security, performance, availability, reliability, modifiability, portability, functionality, variability, conceptual integrity, etc. quality attribute considered? Yes 7.2 What are the implications for missing attributes? No 7.3 What are the implications for not considering the quality attributes? 8.1 Give the descriptions of the practices. 8.2 What are the implications for practicing nothing like utility tree? 1.1 How investigation done to define architectural approach? 1.2 What are the implications of not doing so? 7.1 Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. 7.2 Integration with other application some times creates problem. If utility tree is not practiced, is there other way in practice? Yes No 6 Step 6: Analyze the Architectural Approaches This is the final step in the ‘Investigation and Analysis’ phase. In this step, the output of the utility tree generated 1. Do you analyze architectural approaches Yes No 8.2 Architecture traceability is not possible and it creates rework after creating architecture. . 1.2 Functionality identification can not be 216 SL No. ATAM Step & Description in the previous step is analyzed, and a thorough investigation and analysis to find the architectural approach that deals with the corresponding quality attributes are identified. Here, the risks, non-risks, sensitivity points and the tradeoff points for each architectural approach are identified. This step is sub-divided into four main stages: - Investigation of architectural approach - Creation of analysis questions - Answers to the analysis questions - Find the risks, non-risks, sensitivity points, trade-off points. 6.1 Investigation of architectural approach After identifying these quality attributes that are important to the goal of the system, the architectures are determined how they support these quality attributes. A detailed investigation of the architectural approaches is held to understand if these quality attribute requirements are satisfied. Question Answer Supplementary Question Supplementary Answer done properly. 2.1 Risk items are identified By team meeting. 2. Are risk and non risks identified? Yes 2.1 How risk and non risks are identified? 2.2 What are the implications of not identifying risks and nonrisks? No 6.1 Does the architectural approach support the quality attribute? Yes 1.1 What are the quality attributes considered? 1.1 Attributes are as follows: security, performance, availability, reliability, modifiability, portability, functionality, variability. No Does architectural approach support variability? Yes 1.2 What are the implications of not practicing this? 2.1 What are the processes to identify the variability? 2.2 What are the implications of not practicing this? 3.1 What are the processes to identify the modifiability? 2.1 Through review variability can be identified. No How does architectural approach support modifiability? Yes 3.1 Processes are as follows: Requirement uncertainty, Functionality, Future enhancement, Plugability. No 3.2 What are the implications of not practicing this? 217 SL No. ATAM Step & Description Question Does architectural approach support reliability? Answer Yes Supplementary Question 4.1 What are the processes to identify the reliability? Supplementary Answer 4.1 Processes are as follows: Test each flow of execution of architecture, Test plugability of components. No Does architectural approach support conceptual integrity? Yes 4.2 What are the implications of not practicing this? 5.1 What are the processes to identify the conceptual integrity? 5.1 Considered at the time of design depending on the project category and business requirement. No Does architectural approach support functionality? Yes 5.2 What are the implications of not practicing this? 6.1 What are the processes to identify the functionality of project? 6.2 What are the implications of not practicing this? 1.1 What are the steps followed to create analysis questions? 1.2. What are covered in analysis questions? 1.3 What are the implications of not practicing this? 2.1 What are the design level considerations for component reusability? 6.1 Major functionalities are distinctly described from the details analysis of requirement. No 6.2 6.2 Creation of analysis questions The next stage in this step involves in gathering analysis questions generated from the high prioritized scenarios already discussed above. In a real-life setting, the analysis questions are gathered from all the stakeholders. The analysis questions are associated with each architectural approach discussed above; and are geared towards the important quality attributes. Do you create analysis questions? Yes No Can the components of the architecture reused for future projects? Yes 1.3 Requirement complexity and uncertainty cant be identified properly. 2.1 Design level considerations are making the component more specific and independent to work. No 2.2 What are the implications of not practicing this? 218 SL No. ATAM Step & Description Question Can the framework be expanded in the future to accommodate a new application or new components? Answer Yes Supplementary Question 3.1 What are the steps considered for the framework expansion? Supplementary Answer 3.1 Each component is made to work as independently without any dependency. 3.2 Input and Outputs are distinctly defined. No Will the system handle any input given by the user and handle invalid input? Yes 3.2 What are the implications of not practicing this? 4.1 How the validation considered? 4.1 Validation considered as from the business requirement. 4.2 Third party tools are used to make validation process easier. No Is the architecture consistent in behavior? Yes 4.2 What are the implications of not practicing this? 5.1 What are the steps followed to identify the architectural behavior? 5.2 What are the implications of not practicing this? 6.1 How the decision taken at the time of design to add application specific functionality? 5.1 Steps are requirement complexity, requirement uncertainty and future enhancement. No Can any new application-specific functionality be added to the architecture? Yes 6.1 Each component is created to work independently without any dependency, so any other application specific functionality can be added easily. No Can the system be modified in short time Yes 6.2 What are the implications of not practicing this? 7.1 How the time and cost calculated for change 7.1 Time and cost consideration for the 219 SL No. ATAM Step & Description Question and cost-effective manner? Answer Supplementary Question management? Supplementary Answer change management considered during the design time. So any new change issue cant do so much harm to the system. No Do the components interact properly? Yes No 6.3 6.3 Answers to the analysis questions The third stage of this step is to provide reasonable explanations or answers to the above analysis questions. Are the answer and explanations to the questions analyzed? Yes 7.2 What are the implications of not practicing this? 8.1 How the component interaction tested? 8.2 What are the implications of not practicing this? 1.1 How the answer and explanations to the questions analyzed? 8.1 Component interaction tested by unit testing. 1.1 Answer and explanations to the questions analyzed by the details breakdown of requirement including complex and uncertain requirement, No 6.4 6.4 Find the risks, non-risks, sensitivity points, trade-off points. The final stage involved in this step is to identify the risk, non-risk, sensitivity point and the trade-off point. A risk is a problematic point in an architecture, where the latter does not support a given prioritized quality attribute. A non-risk is strength of an architecture, where the latter fulfils a particular prioritized quality attribute. A Sensitivity point is a property of one or more components, that is critical for the achievement of a given quality attribute. Are the risks and nonrisks identified properly? Yes 1.2 What are the implications of not practicing this? 1.1 What are the processes to identify the risk and non risks? 1.2. What are the common risks and non-risks? 1.1 The processes to identify the risk and non risks are from the requirement complexity, uncertainty and technological uncertainty. No Are the sensitivity points identified properly? Yes No 1.3 What are the implications of not practicing this? 2.1 What are the processes to identify the sensitivity points? 2.2 What are the common sensitivity points? 2.3 What are the implications of 2.3 We can not verify the 220 SL No. ATAM Step & Description If the architecture is sensitive at a point for more than one attribute, that point is called Trade-off point. Question Answer Supplementary Question not practicing this? Supplementary Answer architectural approaches are how much feasible for the application. Are the trade off points identified properly? Yes No 3.1 What are the processes to identify the trade off points? 3.2 What are the common tradeoff points? 3.3 What are the implications of not practicing this? 3.3 We can not verify the architectural approaches are how much feasible for the application. 3.3 We can not guaranty the architecture will work in every case of business requirement. 1.1 Defined by Mother Company. Phase 3 – Testing 7 Step 7: Brainstorm and Prioritize Scenarios This is the first step in the Testing Phase of the ATAM. The former represents the interests of the stakeholders and is used to understand the quality attribute requirements. In the utility tree generation step, the main outcome is to perceive the quality attributes from the architect’s point of view. In this step, the goal is to get the larger stakeholder community involved. The prioritized list of brainstormed scenarios is compared with the prioritized scenarios obtained from the tree in Step 5. The stakeholders are expected to brainstorm three kinds of scenarios: · Use case scenarios - In this case, the stakeholder is the end-user. · Growth scenarios - represents the ways in which architecture's growth is perceived · Exploratory scenarios - represent extreme forms of growth in the architecture. Does all the stake holders involve in brainstorm? Are all scenarios collected after brainstorm? Yes No Yes 1.1 What are the criteria to select stakeholders? 1.2 What are the implications of not practicing this? 2.1 Give some example of Scenarios? 2.1 Examples like: Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. No Are use case scenarios considered? Yes 2.2 What are the implications of not practicing this? 3.1 What are the cases of scenarios considered? 3.2 What are the implications of 3.1 Every high level case and Complex cases are considered. No 221 SL No. ATAM Step & Description Question Answer Supplementary Question not practicing this? 4.1 What are the steps to consider growth scenarios? Supplementary Answer Are growth scenarios considered? Yes 4.1 We analyze the requirement and clarify with the client about the further enhancement of the system. Also project growths corresponding to every business workflow are considered. No Are exploratory scenarios considered? Yes 4.2 What are the implications of not practicing this? 5.1 What are the steps to consider exploratory scenarios? 5.1 Steps include the database system portability, Performance, Availability and security. No Are the scenarios prioritized? Yes 5.2 What are the implications of not practicing this? 6.1 How the scenarios are prioritized? 6.1 Corresponding to the complexity, functionality and business driver scenarios are prioritized. No Are the voting process used for prioritizing scenarios? Yes No 6.2 What are the implications of not practicing this? 7.1 What is the voting process for prioritizing scenarios? 7.2 What are the implications of not using voting process? 8 Step 8: Analyze the architectural Approaches Are the new quality Yes 1.1 How do you do that? 7.2 Sometimes we don’t understand at what point we should start working. If the voting process introduce we can decide from where we have to start working. 1.1 We compare the new 222 SL No. ATAM Step & Description This is the last step in the ’Testing’ phase. In this step the highly voted quality attributes from the previous step are analyzed. The architectural approach that deals with these quality attributes are found and check if the architectures support these attributes or not. This step is a repetition of step 6 in the ‘Investigation and analysis’ phase. The only difference being that in step 6, the high priority quality attributes were from the utility tree, while this step takes highly voted brainstormed quality attributes. Some of the prioritized quality attributes from step 6 may reoccur here, while some new ones may emerge. Finally, the risks, nonrisks, sensitivity points and the trade-off points are analyzed for the prioritized scenarios. Question attributes considered found in the brainstorm session? Answer Supplementary Question Supplementary Answer attributes with the old attributes and prioritize the new attributes. No Are the new scenarios considered? Yes 1.2 What are the implications of not practicing this? 2.1 What are the processes to consider new scenarios? 2.1 We compare the existing scenario and new scenario and prioritize the scenarios. No Are the new risks, nonrisks, sensitivity points and trade off points included? Yes 2.2 What are the implications of not practicing this? 3.1 What are the processes to consider risk and non risk items? 3.2 What are the processes to consider sensitivity points? 3.3 What are the processes to consider trade off points? 3.1 Processes to consider risk and non risk items are complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. Processes to consider sensitivity points are complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed transaction process. Processes to consider trade off points are complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with 223 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer distributed transaction process. No Phase 4 - Report the ATAM 9 Step 9: Report the ATAM This is the final phase of the ATAM evaluation, where all the information collected during the evaluation is presented. The ATAM team presents their findings to the group of stakeholders. The main findings of the ATAM team usually include: - a utility tree - set of generated scenarios - set of analysis questions - set of identified risks and non-risks - the identified architectural approaches 3.4 What are the implications of not practicing this? 1.1 What are they? 1.2 What are the implications of not practicing this? Is the utility tree prepared and reported? Yes No Are the generated scenarios reported? Yes 2.1 What are they? 1.2 We can not prioritize our design and work. We can not identify the strength and weakness of architecture. 2.2 Best case sequence diagram worst case sequence diagram. Also other complex sequence diagram. No Are the analysis questions reported? Yes 2.2 What are the implications of not practicing this? 3.1 What are they? 3.1 Analysis question included new solutions of complex requirements. No Are the risks and nonrisks reported? Yes 3.2 What are the implications of not practicing this? 4.1 What are they? 4.1 Complex work flows, Work flows with huge transaction, Work flows with multiple access, Workflows with distributed No Is the architectural approach identified, finalized and reported? Yes 4.2 What are the implications of not practicing this? 5.1 What are the common approaches identified? 5.1 Layered approaches, N-Tier approaches, Client Server approaches and 224 SL No. ATAM Step & Description Question Answer Supplementary Question Supplementary Answer Pipeline approaches are identified, finalized and reported corresponding to the project type. These are identified by the mother company. No 5.2 What are the implications of not practicing this?