Agile Methodology Links

Self-Evaluating Agile Large-Scale Systems SEALS Barry M. Horowitz University of Virginia 1. Prologue Three historical cases provide motivation for the work presented in this paper (the cases are described in Section 6). The cases are highlighted by important systems engineering results that led to major redirection of ongoing system design efforts. The resulting positive outcomes point to the potential value offered by the system architecture and development methodology to be presented in this paper. In each of the three cases, an important risk or opportunity emerged that created a need for major changes to a system that was already in operation. In all cases, new sub-system development activities that promised to address the needed changes were initiated. In all cases, the new sub-systems had very high cost, and the development efforts to create the sub-systems included significant risks. Nonetheless, in each case, the need was sufficient to warrant moving ahead with a new solution. While these new development efforts were already in progress, through happenstance (self-motivation), systems engineers came forward with alternative solutions that offered to address the new system needs by building on the existing design of the systems that were being improved; i.e., there was latent, built-in agility in the existing systems‟ designs that would allow important features to be added that could address the important needs in a lower risk, lower cost manner. While the original designers of the operational systems had not accounted for the future needs, technology advances that had coincidentally occurred during the systems‟ lifetimes permitted old designs to serve as a framework for addressing new needs through the application of these new technologies. In all three cases, the alternate solutions were sufficiently advantageous so as to result in the cancellation of the new sub-system developments and, instead, start the alternative developments. In each of the cases, the new solution was based on: 1) a very detailed understanding of the technical designs of components in the existing system that would end up being the basis for the solution, and 2) a deep understanding of relationships and interdependencies between components, subsystems and the overall system that resulted in identification of how to address the opportunity or risk that initiated the desire for change. This paper suggests a system architecture and methodology that reduces the roles of happenstance and coincidence in creating and exploiting agile systems. 2. Introduction Over the years, the systems engineering community has played a major role in the development of methodologies and architectures for the creation and life-cycle support of large systems [Buede, 1999] [Hughes, 1998]. This paper introduces a new architecture and methodology that promises to remedy well-known weaknesses and corresponding symptoms that are associated with the use of current methods. It also has the potential to provide a vehicle for improving the ability to deal with a growing 1 demand for system developments that are even more complex than the systems that have been worked on in the past. The time for seriously considering new and better architectures and methodologies is upon us, from a need to do better, a need to respond to increased demand for more complex systems, and the availability of affordable information technologies that can be the basis for new and better methodologies. The principal problem that is addressed in this paper is the important role that uncertainty plays in the current development methodologies for building large-scale systems. The architectures and methodologies that are currently used to cope with uncertainties are insufficient, and result in serious problems that are discussed below. Furthermore, as system complexity increases, uncertainties increase, and current approaches become even less viable than they have been to this point in time. Sections 3 and 4 discuss uncertainties as they pertain to large-scale and complex large-scale systems respectively. Section 5 introduces a new methodology and architecture referred to as Self-Evaluating Agile Large-Scale Systems (SEALS), including examples of the potential value of the new approach. Section 6 provides three historical examples that demonstrate the potential value that the SEALS approach offers. These examples are actual situations where a SEALS-type solution was applied, but only after alternate, less desirable solution activities were started and canceled, and the final approach was reached only through happenstance, rather than through use of systematic methods using an architecture tailored for making significant changes to a system. Section 7 discusses the kind of efforts that would be required to shift the system engineering community from using existing methods to the SEALS, or SEALS-like, methodologies. 3. The Problem of Uncertainties in Large-Scale Systems For the purposes of this paper, large-scale systems can be defined as systems that: a. take a long time to build; long enough that internally and externally driven changes in desired system capabilities are very likely to occur, even while the system is under development, b. cost a lot; enough so that they must last a long time, during which it is very likely that significant new system capabilities will be desired, and c. have a large number of components, sub-systems, external interfaces and stakeholders (owners, users, operators, indirectly impacted parties); enough so that, over the life time of the system there will be a diverse set of desired changes for the system that are not likely to be universally agreed upon by the various stakeholder groups Systems with which we all are familiar obviously satisfy this definition of a largescale system. These include Air Traffic Control Systems, DOD Command and Control Systems, large companies‟ Enterprise Resource Planning (ERP) Systems, Telecommunications Systems, etc. This definition has certain implications relative to system engineering methodology for large-scale systems. These include the need to: 2 a. successfully manage the potential instability of system design requirements during development, as this can cause potentially serious problems in the management of delivery schedules and budgets, b. successfully manage the issues that arise due to potential long-range needs for a system to fulfill, and the large uncertainty surrounding predictions about these future needs, and c. create at least temporary harmony among system stakeholders to result in an equilibrium state relative to system objectives and associated design requirements, so that new system efforts, and major system modernizations can get started In general, current system engineering methodologies address these needs by focusing system requirements on current needs and sticking to them. This approach has several positive attributes. It: a. removes instabilities in system design and development, saving time and money, b. avoids predictions about uncertain needs and expenditures on needs that might never result, avoiding wasting of resources, c. avoids additional issues being brought into consideration that could delay stakeholders from arriving at equilibrium in terms of system requirements, so that new efforts can get started more readily, d. permits cost/benefit analyses that are not confused by the uncertainties of the future, which are difficult to evaluate in any case, e. assures buyers of the system that they have not inadvertently disrupted its development through introduction of new requirements that result in development problems, and f. helps to sustain the accountability of the development team to deliver what they signed up for, at the cost and schedule that was agreed upon. However, these advantages are gained at significant costs. These include: a. New systems don‟t do what is desired even at initial delivery because things changed during the initial development period b. Can‟t readily modify systems because they are point designed c. Can‟t readily integrate with new adjoining systems because of the point design d. Possible delays in starting development of new systems because uncertainties need to be resolved among stakeholders prior to starting There are many examples of systems that were ultimately considered as “failures” due to the conditions described above. Perhaps one of the most extreme cases is the Comanche Helicopter, an attempt by the US Army and Boeing to develop a stealthy helicopter that would carry multiple sensors for gaining situational awareness without being observed [CNN, 2004]. This effort was started in 1983 and was terminated in 2004 because: 1) the collapse of the Soviet Union had occurred, 2) the new need was 3 the “war on terrorism”, 3) the cost per helicopter was considered to be too high, and 4) unmanned reconnaissance aircraft were introduced into use. The development effort had cost $6.2 billion dollars at the time of cancellation. While the Army uses component technology results derived from the Comanche efforts, it was not able to adjust the system to fit new needs, even with the very long period of time between changes in need and cancellation of the project. Recognizing the high cost for large-scale systems, the risks involved by introducing uncertainties into the development efforts are simply viewed by the community that buys and builds large systems as too high compared to the benefits associated with fixing requirements. Nonetheless, over time, methodologies and corresponding system architectures have been established with the objective of recognizing the potential for change through sequential development processes [Boehm, 1988]. These methodologies introduce sequential steps in building systems that are organized to delay decisions on the fixed needs until they must be decided upon, so that some of the changes that might have occurred from the start of development are accounted for in the system. Other methods include the possibility of sequentially introducing into operation the usable sub-systems of a system that is under construction, and learning more about operational needs through early use, and, when possible, adjusting accordingly. More recently, methodologies include incorporation of adaptive technologies into systems (referred to as agile technologies in the remainder of this paper), that can readily support the system‟s ability to accommodate to certain needs for change, through either automatic self-adjustment or easily managed system configuration changes. However, in general, the use of the agile technology approach has been opportunistically driven (i.e., the cost of the solutions is sufficiently low that the value is not a critical issue in deciding to use the agile technology), as opposed to being derived from a forecast of new needs over time [Reich, 1999] [de Neufville, 2004]. In addition to these development methodologies, economic methods have been developed that deal with the value of adding design options for future use into a system [Kalligeros, 2006] [Wang, 2006]. In spite of the efforts to be more capable at adjusting to future needs, current methodologies do not directly address the issue of future needs and uncertainties systematically. A methodology to address this subject would include processes for: 1) identifying and continuously learning more about new opportunities and risks that may need to be designed for in a system, 2) estimating the time constants that need to be dealt with in seizing potential opportunities or reducing risks, 3) designing and evaluating the agility of systems so that they can actually adjust within those time constants, including both human organizational agility as well as technical agility, and 4) developing economic strategies for sequentially staging new agilities into systems to support “just in time” response to opportunities and risks. The SEALS architecture and methodology introduced in this paper addresses these objectives. 4. The Exacerbated Problem of Uncertainties in Complex Large-Scale Systems For the purposes of this paper, a large-scale system that is complex is defined as a large system (as defined above) that: 4 i.Is developed through a distributed process involving separate entities contributing in parallel to the implementation of a shared system concept ii.Involves organizations that each has its own objectives and risks and carry out its efforts accordingly iii.Involves a mixture of participants engaged in the development, ranging from government organizations, private sector firms and non-profits. iv.Does not have an integrated budget for the separate development efforts, or a resource allocation plan – each entity operates on its own budget plan based on individual motivations v.Does not have an integrated schedule or integration plan for the separate development efforts – There are no start or finish dates, and integration is accomplished through agreed upon cooperative experiments or in-field operational trials. Where necessary, the integration depends on standards being developed or adopted by the community of developing organizations, either explicitly through standards groups or implicitly through use. Examples of such systems would include a new national health care system, a new international environmental monitoring system, a new intelligent highway system, the integrated intelligence system for homeland security, and the Internet. It is important to note that the Internet was a derivative of a large-scale system that the Department of Defense was hoping to develop, and, in addition, the National Science Foundation played a major role in starting the efforts required to create the Internet. The history of the Internet points to the importance of certain participants as the key stakeholders in the creation of large-scale complex systems, setting the direction, supporting the needed standards, and making the major investments, including higher risk investments, to get things going. When comparing the complex large-scale system to the large-scale system, it is apparent that the uncertainties not only include those related to future opportunities and risks, but also those related to the distributed nature of development. Interestingly, while the overall risks are greater, the risks of the complex system are divided among the participating organizations, as opposed to being focused on the smaller number of organizations involved in large system development. Another interesting aspect of the complex system is that new functions and capabilities can emerge in a manner that depends on the integrated contributions of individual developing organizations, possibly resulting in outcomes that were never anticipated (referred to as emergent). For example, for the case of the Internet, the emergence of capabilities such as Skype to compete with voice communications over the normal telephone systems is something that was not in the original “blue print” for the telecommunications industries that invested large sums of money in the 1990‟s related to the growth of the Internet. 5 An example of a complex large-scale system that has been under consideration for over 25 years is inspired by the Intelligent Highway concept. The basic idea of interest is that a network that supports communications from vehicle to vehicle, between vehicles and off-road safety and traffic management services (e.g., emergency accident response, traffic light control systems), and between vehicles and possible commercial services that are related to vehicle position (e.g., local time varying advertisements), could be of great value [Chadwick, 1993]. The potential opportunities for an Intelligent Highway concept have increased with time as computing, communications and display electronics technologies have advanced, and as new sensors have been developed for use in vehicles. The participating organizations that would be involved in the development of an actual system would include: a) Vehicle manufacturers – Vehicle to vehicle communications systems, in-vehicle sensors, vehicle to vehicle coordinated safety features, and the driver interfaces to the system b) Federal government – Off-road communications gateways to the vehicles, new system concept development for safety and traffic management systems c) State governments – State Departments of Transportation for developing traffic delay prediction capabilities, re-routing capabilities, traffic dependent traffic signal and video camera surveillance control systems d) Emergency responders – Support systems for accident severity predictions and predictions for special rescue equipment that might be needed for particular accidents e) Hospitals – Patient care readiness predictions, prediction-based special information or treatment requests to responders prior to arrival on the scene f) Commercial service providers – On Star, advertising companies While the Intelligent Highway system concept has been under consideration for such a long time, the progress has been very slow. Progress includes the development of a standard for vehicle to vehicle communications (led by auto manufacturers), the demonstrations of a number of possible system capabilities (mainly funded by government organizations), and the introduction of coordinated vehicle to vehicle safety features available to consumers. There are no clear predictions about when capabilities that are comparable to the envisioned system concepts will be available, and in what order one might expect capabilities to emerge. No matter how one views this situation, it seems clear that the system engineering community is not providing a compelling way to analyze and respond to opportunities of this sort, resulting in the possibility of a lost opportunity or a greatly delayed opportunity, or even a result that points to this being an opportunity that requires certain precursors until it is ready. 6 5. Self Evaluating Agile Large-scale Systems (SEALS) Self Evaluating Agile Large-scale Systems (SEALS) is a high-level architectural approach with a corresponding systems engineering methodology to address the development of complex large-scale systems; and in particular, to deal with the uncertainties surrounding systems, as discussed in Sections 2 and 3 above. SEALS is comprised of three components that will be elaborated upon in this section: 1) Multi-scale opportunity and risk analysis 2) Built-in self-evaluation of readiness to respond to opportunities and risks 3) Agile features including architectures, technologies and human organizations that can enable timely responses to seize opportunities and reduce risks The basic concept for SEALS is to regularly perform opportunity and risk analyses in order to: a) Determine the opportunities and risks for which more careful assessments are desired, b) Support decisions on creation of a measurement and analysis subsystem, built-in as part of the overall system under consideration, to gather and assess information pertaining to the opportunities and risks of concern, including estimates of time to successfully seize opportunities or reduce risks, and c) Create a system architecture that can be responsive to possibly needed changes in technical capabilities or human organization in a timely manner, and based on results from the measurement and analysis sub-system, initiate actual implementation efforts for “just in time” availability. Figure 1 illustrates the SEALS architecture. 7 SEALS Architecture Multi-Scale Analysis Self- Evaluation Agility • Agile Technical System Architecture and Design • Agile Human Organizational Architecture and Design New System Features Needed New Operational Capabilities Measurements •Internal •External Operational System Needed New Measurement Capabilities Built-in Measurement And Analysis Sub-Systems For Emerging Opportunities And Risks Multi-Scale Opportunity & Risk Analysis Figure 1 SEALS Architecture The following sections elaborate on each of the SEALS elements. 5.1 Multi-Scale Analysis of Opportunities and Risks Multi-scale analysis refers to the use of an integrated collection of analysis tools that permits an opportunity and risk evaluation of a large-scale complex system at varying levels of granularity, ranging from the macro-level to fine detail micro levels [Bar Yam, 2005]. The need for such modeling within the SEALS framework is related to the desire to identify and evaluate opportunities and risks that can emerge from any level of a largescale complex system. This viewpoint recognizes that an opportunity or risk that resides at the lowest level of a system can result in an outcome that may significantly impact the highest level of the system. For example, the ability to create mobile mass storage devices at relatively low cost enabled the creation of the iPOD, which in turn provided opportunities for Internet-based downloading of music and other entertainment to emerge as a significant new Internet service. As a result, the capacities of mass storage devices are likely to continue to rise, while cost is likely to go down, perhaps permitting new applications requiring more capability, but at lower costs, to emerge. An example might be health self monitoring by comparing one‟s own body measurements (e.g., heart rate) with stored historical patterns. In addition to multiple scales, it is also important to model systems relative to the multiple objectives that they might support. For example, economic models, safety models, environmental models, energy consumption models, and geographical models all 8 might prove useful in the process of evaluating potential opportunities and risks. In turn, these models need to operate at multiple scales, as discussed earlier. The SEALS multi-scale analysis methodology consists of the following five steps: 1) For the system under consideration, select classes of potential opportunities and risks to address. The existing Hierarchical Holographic Modeling (HHM) approach will be used to illustrate this step [Haimes, 1981]. The HHM methodology results in packages of opportunities and risks that are the most important candidates for stimulating changes to the system in question. It also ranks the packages in order of opportunity or concern. 2) For the highest ranking packages, determine the potential observables that, over time, could provide added insight about their likelihoods, benefits, and consequences. 3) Evaluate the possibilities for setting up an information collection and analysis capability as part of the system under consideration, and determine the cost to set up a general infrastructure for collection and analysis as well as for implementing specific collection and analysis capabilities within that infrastructure. 4) Determine thresholds for likelihoods, benefits, and consequences which will be used to help determine that the time has arrived to consider implementation of system changes for seizing opportunities or reducing risks. 5) For the opportunities and risks of interest, determine the changes that would be required to the design of the system under consideration if a response were to be called for. These could be technical or human organization changes. For the identified changes, an economic assessment is required of solutions that can be partially implemented in anticipation of a change so as to result in a reduction of expected cost under the circumstances of either fully, partially or not implementing the possible change. This analysis would result in the desirability of certain degrees of agility being built into the system, in anticipation of what could be important changes that would be faster to accomplish and lower in cost. This analysis could also include an evaluation of the relationship between forecasting errors and degrees of agility. This analysis would recognize that less agility implies slower response, which implies the need to act at an earlier time in order to achieve a given level of responsiveness. Since the need to act at an earlier time generally implies larger forecasting errors, more agility can be seen as reducing the likelihoods of changing a system when it is not desirable to do so. An integrated economic analysis can help define the selection of measurement and analysis capabilities and system agility features (levels of responsiveness), accounting for costs of measurement and analysis, costs of agility features, response time adjustments that agility features offer, and the values of opportunities and risks that could result from a more responsive system. The following Sections discuss HHM and the five SEALS steps in turn. 9 5.1.1 Hierarchical Holographic Modeling (HHM) This section introduces the technique called HHM that has been used for carrying out complex risk analyses at multiple scales (hence the term hierarchical), and relative to multiple objectives (hence the term holographic) [Haimes, 1998]. The HHM technique serves as a starting point for the purposes of SEALS. It can be readily adjusted to perform opportunity analyses as well. HHM supports a structured team analysis effort through the use of collaborative computing tools. The analysis team must consist of a set people who collectively have the multi-scale, multi-objective knowledge about the system, so as to be able to carry out a full opportunity and risk assessment. The three questions that a risk analysis must address are: 1) What can go wrong?, 2) What are the consequences?, and 3) What is the likelihood? To extend to opportunity analysis, one would add the questions: 1) What opportunities might exist? 2) What are their benefits? 3) What are their likelihoods? Based on numerous prior applications of HHM for risk assessment, a critical step in the methodology is the integration of a team of interested parties and experts to identify all opportunities and everything that can possibly go wrong and the corresponding system attributes that make it possible to seize the opportunities and reduce the risks. While in the case of complex systems it may not be possible to develop a complete result, the team is nonetheless inspired to be as complete as possible. 5.1.2 Steps 1 and 2 of SEALS Multi-Scale Analysis, using HHM Figures 2 and 3 together present the results of an HHM risk analysis that was focused on identifying the risks that need to be considered when hardening a water supply system [Haimes, 1997]. This analysis was conducted for the President‟s Commission on Critical Infrastructure Protection. The diagrams represent a system as a set of “Head Topics” that cover the full range of system vulnerabilities, and under each Head Topic is a set of identified risks at varying levels of the hierarchy of subsystems involved. The figures illustrate the scope of an analysis that hopes to deal with both the holographic and hierarchical aspects associated with identifying “what can go wrong”. This HHM analysis was carried out by a substantial team of water supply system experts, who used these results as input to analyses on consequences and likelihoods of failure. Together, the analyses of vulnerabilities, consequences, and likelihood create the output of an HHM activity. 10 Physical A Scope B Temporal C Maintenance D Institutional E Organizational F Management G Resource Allocation H Hardware International Long-long (30+) Affected by Budget Federal DecisionMaking Structure Security Government Pipes National Long (10-30) Standardization Regional Subordinate Structure Short Term Emergency Response Loss of Funds Pumps Regional Intermediate (3-10) Replacement Parts State Communication Long Term Emergency Response Proper Allocation Wells State Short (1-3) Proper Operation Local Daily Operations Desalinization Management Aqueducts Local Now Technical Personnel Individual Emergency Ships O.M. & R. WTPs Individual Plant Planning for: Personnel WWTPs Individual Life-Cycle SCADA Phase Out Figure 2. HHM Analysis Results for Identifying Water Supply System Vulnerabilities (1 of 2) SCADA I System's Configuration J Hydrology K Geography L External Factors M Buffer N Contaminants O Quality of Surface & Groundwater P Temperature Software Centralization Surface Water Lake Lay of Land Threats Overdesign Biological Remote Control Modeling Problem Feedback System Wrong Signals False Positive Decentralization Mountains Insider Bio-organic Chlorination Problem Isolation Intraconnection River Plains InsiderOutsider Outsider Pathogens Salinity Reservoir Water Bodies Climate Chemical Turbidity Interconnection/ Different Systems Interconnection of H/W Interconnection of H/W & SCADA Glacier Group Nuclear pH Ground Water Seasonal Varieties Natural Hazards Toxicity False Negative Sediment Figure 3. HHM Analysis Results for Identifying Water Supply System Vulnerabilities (2 of 2) Once the opportunities and risks are identified, an analysis of benefits and consequences is required. The Head Topics of the HHM model help to identify the many dimensions that benefits and consequences can have (e.g., dollars, lost lives). The expert team is usually able to directly apply its expertise to the estimation of values, but additional team members may be required on an as needed basis. Predicting likelihoods is more difficult. However, one can start by dividing the benefits and consequences into categories (largest, large, ..., small), as well as doing the same for the likelihoods for the opportunities and consequences (e.g., highest, high, …, lowest). The multiple of the size of these two sets is the total number of combinations of consequences and likelihoods. For example, if there are five (5) consequence levels and four (4) likelihood levels, there would be twenty 11 (20) possible combinations across all elements of potential opportunities and benefits. For each of the identified opportunities and risks, specific models would be used to determine the size of the result. Depending on the nature of the opportunity or risk being considered, these models could include mathematical models (deterministic or stochastic), input-output models, systems dynamics models, simulation models, experimental models, etc. The library of models that are used would become a part of the system under consideration (under configuration management), so that future analyses can build on the initial modeling activities and, as time goes by, have a basis for understanding and adjusting prior decisions. As an example that will be carried through all of the five analysis steps in SEALS, consider a large manufacturing company‟s international supply chain system for managing the creation processes for their products. For this example, an HHM analysis would result in ranked opportunities, such as: 1) sustain or increase current market share through price reductions enabled by increasing competition among a larger number of company suppliers, 2) reduce costs and increase profit by reducing inventory levels through increased monitoring of actual sales based upon use of RFID-based systems at points of sale. The HHM might also result in identification of a risk that is growing, such as the theft of Intellectual Property (IP) by manufacturing suppliers. This risk will be expanded upon in the following section. 5.1.3 Steps 3 and 4 of SEALS Multi-Scale Analysis These two steps require specific designs that will become a more tightly integrated part of the system under consideration than the analysis models discussed above. Step 3 deals with determining data sources that can provide insight into the emergent opportunities and risks for the system. The measurements could be based on a wide variety of data sources, including data from: system sensors, surveys, government studies, third party data analysis firms, media articles, etc. These data sources would deal at the levels of system granularity that were necessary to evaluate the specific opportunities and risks identified through the HHM analysis. The combination of data value and data cost would be used to select the desired measurement sources. The combination of data sets would be the input to a data mining effort that would be directed toward evaluating the timing associated with seizing opportunities and reducing risks. As an example of Step 3, return to the case of the large international supply chain management system for which the HHM analysis determined that a high-likelihood, highconsequence risk to be monitored was the theft and exploitation of IP gathered from the system‟s own data bases by suppliers. In general, this risk would be one of several opportunities and risks to be monitored. The potential consequences are that the theft could result in competing products that reduce the market share of the company that owns the IP, and the possibility of negative media visibility that could impact the company‟s stock prices. Elements of potential solutions include organization and process changes to control the situation, use of more secure authentication and authorization methods for access to sensitive information, and use of encryption for sensitive data with company 12 administration of encryption keys. In order to monitor this particular risk, the following data sources could be used to gain a better understanding of the risk: a) State-provided data on cyber incidents, gathered from openly available state reporting sources on cyber incidents, b) Third-party provided data, through paid subscriptions, on industry-wide cyber security investments (e.g., Gartner Group, Forrester), c) Searches for possibly stolen company IP in foreign produced products, d) Business Association studies of IP theft trends, e) Government studies on cyber security, and f) Automated internal auditing of data base users and information use. Recognizing the cost of incorporating the selected data sources and the relevant data from those sources into the overall supply chain management system (under configuration management), decisions are needed on what data to actually use. Step 3 also deals with evaluating this matter. For the supply chain management system example the cost to gather government and business association data is low compared to paying third parties for data, which could be much lower than actually searching for possibly stolen IP and would be small compared to integrated automation for data base auditing. Correspondingly, the discovery of stolen IP would be the surest way to determine the state of the risk under evaluation. As a result, step 3 includes analyses of costs and value of data whose results would provide useful support to a decision on what to do about collection. Having decided on data sources, criteria are needed for initiating the decision-making process on when to act (i.e., start changing the system). This analysis must account for the system response time, the quality of predictions, the cost of decisions, and the value of the opportunity or risk being considered. Step 4 is the part of SEALS that addresses this subject. It requires evaluations of data quality, and model qualities in order to determine the uncertainties of the opportunities and risks at the time of decision-making. Since SEALS is concurrently dealing with a number of opportunities and risks, the relative levels of uncertainty are also important, and part of what is determined in Step 4 is the integrated decision making strategy for the system as a whole, based on the opportunities and risks of greatest interest, the data mining strategy for improving understanding of these opportunities and risks, and factors related to costs and benefits. 5.1.4 Steps 5 of SEALS Multi-Scale Analysis Step 5 involves determining the solution space that corresponds to the opportunities and risks of interest, and expanding on the economic analyses described in Section 4.1.3., to include the new system development costs. For the supply chain management system cyber security example, solutions could include: a) Human Organization Solutions – 1) hire cyber security expert with appropriate experience, 13 2) conduct survey of cyber security vendor referenced clients on organizational issues, 3) develop organization plan for supporting authentication, authorization and key management processes that includes foreign locations, 4) develop possible sequencing plans for organizational plans related to introducing and evolving cyber security capabilities, 5) establish a workforce training and orientation program, and 6) estimate costs for all solutions considered b) Technical Solutions – 1) develop architecture for overall cyber security technical solution, including added authentication, authorization and encryption capabilities, 2) segregate database into sensitive and non-sensitive information, 3) develop possible evolutionary implementation plans, using inputs from vendors and vendor referenced clients, and the relative risks for each of the company‟s suppliers, and 4) estimate costs for all solutions considered As discussed above, the results from the work described in Steps 1 through 5 above, would provide a basis for making trade-offs to determine an operating point for the degree of measurement and analysis capability that would be built into the system, and the level of responsiveness that would be built into the system. The faster the response time, the less sophisticated the measurement and analysis system needs to be. Alternately, the less responsive the system is to change, the more sophisticated the measurement and analysis system needs to be. 5.2 Built-In Self-Evaluation This part of the SEALS architecture is focused on the design, development and configuration control of a sub-system that is usually not included in current system developments; a self-evaluation sub-system. The purpose of this sub-system is to gather measurements and carry out analyses based on the measurements in order to evaluate potential opportunities and risks related to the system. The measurements will serve as the inputs to a library of analysis tools that is part of this sub-system as well, so that the decisions on system changes related to seizing opportunities and reducing risks can be made using data and tools that are compatible. As indicated in Section 4.1, measurements can have a variety of characteristics, including: 1) collected from within the system (e.g., the security audit data example in Section 4.1) as computerized information, 2) collected from surveys, 3) purchased from data collection organizations, and 4) collected from Internet searches. The type of information that is collected can vary from information that directly couples to the opportunities and risks (e.g., business forecasts on growing markets), to 14 information that relates to technology advances that can help to create a new market (e.g., new technologies that reduce component costs significantly), to information that relates to emerging government regulations that can impact a market, to information about capacities and demand for specific system functions that may require modernization. This aspect of SEALS could energize the emergence of organizations that specialize in gathering information about new technologies and their relationships to system opportunities and risks by catalyzing small firms to focus technology development efforts in areas that have high value for systems. Organizations such as the Center for Innovative Technology in the state of Virginia are currently carrying out such a role, but such activities have yet to become an important part of systems development activities because current methodologies do not foster the demand for addressing future opportunities and risks [CIT]. The importance of including collection and analysis as part of the operational system is related to the need to continuously evaluate the candidate opportunities and risks until either they no longer are an issue or system implementation decisions are made related to seizing the opportunity or reducing the risk. The periods of evaluation can be long; long enough to require a system for assuring continuity in the analysis efforts and minimization of lost memory of results and duplicated effort. In addition, the tools for analysis may include automated analysis tools, collaborative computing tools and other tools that should be kept under configuration control as part of the system, so that necessary changes can be made as data collections change over time. 5.3 Agility Features As discussed earlier, if a system is difficult to modify and requires long lead times to make changes, the forecasts for the needs for change must be far in advance. The long lead time frequently can make the uncertainties of prediction sufficiently large so as to discourage action. Consequently, adding agility features to a system can speed up response time, reduce the needed lead time for actions and therefore reduce uncertainties that otherwise might discourage taking action. Recognizing this relationship is extremely important. It helps to explain why systems that can not be readily changed discourage their stakeholders from exploring new opportunities and risks related to the system, and can lead to very undesirable spirals that result in major lost opportunities and inability to respond to major risks that actually occur [Zhao, 2003]. Returning to the Comanche Helicopter example, one can see how such a situation could get as far as it did. The agility of a system can come in many varieties: adaptive software managed functions, easily modifiable human organizations, system policies that are easy to change, technical architectures that are modularized for change, etc. System designers already consider such features as they create a system. Typically this is carried out as a “good design practice” activity, as opposed to being structured around a systematic opportunities and risk activity that is embedded in the system that is being designed. As a practice of good design, there are no guiding economic boundaries to determine which features are more valuable and which are less, and how much is the agility worth. While there are existing economic analysis approaches that address the determination of value, 15 they logically would require the measurement subsystem activities discussed in Section 4.2 as inputs, and this combination of activities is sufficiently unconventional that they do not take place. It is for this reason that the SEALS concept must include all three parts discussed in this paper; without agility the predictions will need to be too far in to the future, and without measurements and forecasts that relate to opportunities and risks, agility is a general rather than specific system need and will not be guided by the factors that stakeholders care about. The following sections discuss some examples of the possible uses for SEALS when considering specific architectural and technology solutions for agility. 5.3.1 Service Oriented Architectures Service Oriented Architecture (SOA) for large information systems is currently receiving major attention in the computer system development domain [Papazoglu, 2003] . By modularizing software systems into services that, through use of integration standards, can be combined into customized processes that consist of many services, the concepts for reusable software can potentially be realized [Boehm, 2005]. This concept requires software management capabilities and tools that support reconfiguration. It requires creating a workforce that can successfully manage such capabilities. And, it also requires a workforce capable of addressing technical issues that surround SOA, including process specific needs related to system performance, security, and reliability. Therefore, service integration may require additional activity beyond the functional integration that is desired. Using SEALS, the system that might include an SOA would: 1) relate the service structure to future opportunities and risks that the stakeholders had identified through the SEALS methodology, 2) determined the scope of potential process modifications and the level of capability required in its workforce that is managing the SOA, including the possibilities for outsourcing functions, and 3) determine the level of investment and the timing for investments that would introduce and convert current system capabilities into SOA-based capabilities. 5.3.2 Peer to Peer Networking Peer to peer (P2P) networking technology is receiving attention as a capability that permits communications relationships in a system to be dynamically adjusted. P2P technologies hold the promise of being able to be the basis for agile information management capabilities in a system [Agre, 2003]. Of course, information management needs are closely related to the functions that a system performs and the roles of humans and machines in that system. In order to understand the value that P2P networking offers, one would need to forecast the potential range of information use that a system could evolve to over time [Bonifacio 2002]. Through the use of SEALS, the opportunities and risks determined by the system stakeholders would provide the basis for addressing the specific uses and values of P2P technology in a particular system. 5.3.3 Wireless Communications Technologies in Enterprise Systems 16 Enterprises recognize that wireless technology can potentially offer significant agility to how and where specific system functions are carried out [Sutherland, 2002]. The pace of change in wireless technology capabilities is extremely fast, currently driven by immediate opportunities in the entertainment domain (iPODS, games, etc). An enterprise considering use of wireless capabilities needs to carry out a complete analysis in order to understand the issues related to use of mobile functions in their systems [Sharma, 2001]. For example: 1) how will configuration control be carried out, assuming that the technology keeps changing, but the enterprise can not continuously retrofit all users, 2) how will security risks be managed (e.g., the highly publicized cases of stolen information from stolen lap tops), and 3) what are the organizational training and management needs that come with use of the wireless technologies. These questions and many similar questions would be continuously addressed as part of SEALS, guided by the opportunities and risks identified for the system. 6. Historical Cases This section provides three historical cases that serve to illustrate the potential for the SEALS approach. In all three cases an opportunity or risk that needed an important system modification was identified by system stakeholders. No multi-scale system analysis was performed. No data had been collected on the agility of the existing system to accommodate the new needs. Instead, because of the importance of making a change to the existing system, major efforts to develop new subsystems that would be added to the existing system were started. These efforts were high-cost and would take substantial time to implement. After these efforts were initiated, engineers involved in activities surrounding the existing system came forward with solutions that evolved out of the existing system design; i.e., the existing system had latent agility that was not accounted for in starting the new efforts. In each case, a much lower cost and faster to implement solution was adopted, tightly integrated with the existing system. The discovery of the “new” solutions was happenstance; i.e., there was no systematic approach such as SEALS to help with discovery. One can only speculate on how many other situations have occurred, where better results could have occurred, but did not. With the use of a SEALS-like architecture and methodology, these examples help to demonstrate the tangible value that SEALS could potentially provide. 6.1 Have Quick/ Seek Talk In the 1980‟s the Air Force recognized the risk of Soviet airborne electronic jamming systems interfering with their communications involving fighter aircraft. At the time US fighters carried fixed frequency UHF radios that could be easily interfered with. Recognizing this risk, the Air Force started a new radio system development activity called Seek Talk. This system would use anti-jam radio communications wave forms requiring synchronization among communicators that required advanced clock technology developed as part of the Seek Talk system. The Seek Talk development was considered to have considerable technical risks and was forecasted to have a very high cost per aircraft (the costs included installation-related costs). After the start of the Seek 17 Talk program, the Mitre Corporation (Seek Talk and Have Quick systems engineer) came forward with a concept for modifying existing UHF radios in fighters. The Have Quick program was the proposed idea [Raytheon]. It recognized that the existing aircraft radios already included all-channel frequency synthesizers along with keyboards and displays for data entry. All that was needed was an accurate clock and a microprocessor to add frequency hopping to these radios. Advances in micro-electronics made it possible to add these new components to the existing UHF radios via a modification kit, mounted on the same electronic boards as already in use (i.e., no extra space was needed to accommodate the upgrade). The costs to accomplish Have Quick were very low compared to Seek Talk, and the technical risks were addressed through a breadboard rapid prototype development activity that created the modified electronic boards for the radio. The final capabilities of Have Quick were less than Seek Talk would have provided, but the advantages were so great that the Air Force decided to cancel Seek Talk and implement Have Quick. The Have Quick radio sub-system with frequency hopping is still in use today. This example is the result of a multi-scale analysis where at the highest level of the system, the Air Force was concerned about air superiority and jamming risks. Accordingly they initiated a solution, without the lower scale analysis of the existing system‟s agility. Indeed, the advances in electronics technology provided the agility to respond quickly and at low cost. The solution was not as capable as starting over, but was deemed sufficient to cancel the effort that was already initiated to develop an entirely new solution. With SEALS the need for Mitre and Motorola to come forward would have been systematic, as opposed to depending on special initiative from the knowledgeable sources. 6.2 BCAS/ACAS In the 1960‟s the Federal Aviation Agency (FAA) and the air line industry, as represented by the Air Transportation Association (ATA) started research efforts related to risk of mid-air collisions due to possible, although unlikely, failures of the Air Traffic Control (ATC) System [ATA, 1972]. The basic solution that was pursued was the development of an airborne collision avoidance system (ACAS) that would serve as a back up to the ATC system [Williamson, 1989]. The ACAS equipment would provide last minute, coordinated collision avoidance directions to pilots on a collision course In order to work, both aircraft involved in a potential collision would need to be carrying the ACAS equipment. This requirement served as a major detraction for implementation of the ACAS system, as the risk was not considered high enough for the FAA to regulate carrying of such equipment. The FAA‟s view on this risk was shaped by the cost of the equipment being prohibitive for general aviation aircraft owners, and the majority of deaths due to mid-air collisions being the result of air carriers colliding with smaller general aviation aircraft [Rucker, 1974]. Nonetheless, investment in ACAS related research efforts continued. In the mid-1970‟s, a well known engineer in the aviation community, George Litchford, came forward with the idea that an airborne collision avoidance system could be developed that processed the signals emanating from the on- 18 board transponders that aircraft carried to respond to FAA radar interrogations to support the ATC system‟s surveillance operation. Litchford‟s proposed system would protect any aircraft carrying the airborne equipment from collisions with any aircraft carrying a transponder. This potential solution was very attractive because it would protect air carriers from the large number of general aviation aircraft carrying transponders and the first aircraft that purchased equipment would be protected from all transponder carriers (which already included all other air carrier aircraft). Litchford‟s proposed solution required very detailed technical knowledge about the design and performance of the FAA‟s transponder-based radar system. Analysis of his proposal determined it to be too complex to be practical, given the state of airborne computing technology at the time. However, the Mitre Corporation (involved in FAA collision avoidance research efforts) suggested a derivative of Litchford‟s solution that was practical, but had potential flaws. The FAA funded research by Mitre and later Lincoln Laboratory, that addressed the development of a transponder-based airborne collision avoidance system that actively interrogated close-by aircrafts‟ transponders and, like ACAS, computed last minute avoidance instructions to pilots, when necessary [Mitre]. This system became known as BCAS (Beacon Collision Avoidance System) and ultimately became an international standard and was regulated by the FAA for use on air carriers. Similar to the Have Quick example, the existing ATC system had a solution embedded within it, but the approach to addressing risks and opportunities did not include systematic use of multi-scale analysis to discover the latent agility. Instead, the happenstance of Litchford coming forward with a proposed solution, and Mitre following up on that opportunity, created an answer that very well could have been missed. With the use of SEALS this type of solution would be systematically sought. 6.3 AWACS Radar System Improvement Program (RSIP) In the 1980‟s the US Air Force became concerned about the risk of the Soviet Union introducing stealthy fighter aircraft into use, and the possibility of the performance of the AWACS aircraft that provided airborne radar surveillance support for US forces becoming seriously impaired [Leppingwell, 1989]. As a result, a significant development effort was initiated to improve the radar system on AWACS aircraft, to be able to detect and track stealthier aircraft. The design concept that was selected for use was to add a sophisticated aircraft tracking system onto the AWACS; one that would start tracks at very low radar signal levels, recognizing that most started tracks would be false (based on noise). However, the tracking system would create paths and velocities from sequential data that could be used to throw out false tracks based on the physical properties that actual aircraft tracks would exhibit. The technical problem to be addressed was the extreme number of false, or phantom, tracks that would need to be processed in order to discover the actual aircraft tracks. This would require very high performance computing capabilities on-board the AWACS, executing yet to be validated tracking algorithms, and there was limited evidence about how well the idea would work. The cost of implementing the solution was high, and evaluation time of the implemented solution would likely be significant. 19 While the tracking system development activity proceeded, the Mitre Corporation (AWACS systems engineer for the Air Force) and Westinghouse (the AWACS radar system developer) came forward with an important analysis. Their analysis indicated that if the analog to digital radar signal converters used on the AWACS radar were improved to be faster and have finer quantization levels, the existing system could be used to detect and track stealthier aircraft. The analysis also indicated that from the time AWACS was originally designed there had been major advances in analog to digital conversion technology. The cost and time to change the converters was significantly less than the tracking system‟s cost, and the certainty of the solution was far greater. The Air Force cancelled the tracking solution activity and in its place initiated and carried out a development activity to upgrade analog to digital converters and make corresponding adjustments to the existing AWACS aircraft tracking system. Similar to the Have Quick and BCAS situations described above, the AWACS had the inherent agility to readily support the desired needs, but a multi-scale assessment was not carried out to exploit the situation until Mitre and Westinghouse, through happenstance, came forward with the observation about analog to digital converters; a very fine level of detailed information that enabled an entirely different and better solution than was being pursued. Again, it would seem that a systematic effort such as SEALS would provide greater assurance that solutions of this sort would indeed be discovered, and in a more timely manner. 7. Transition into Practice This section addresses the question of how a new, SEALS-like approach to system development can be brought into use when current system development approaches are organized to remove considerations of uncertainty. First, it would seem that changes will need to be sequential, so that the management practices related to budgeting for uncertainty, starting developments in the face of uncertainty, contracting with suppliers where uncertainty is properly treated in contracts, etc. can be brought into use without undue risk. The systems engineering research community can play a major role in expediting change, by developing analytical tools, technologies, economic analysis methods, system architectures, work force analysis and design methods, that can be grouped into an overall SEALS-like methodology that addresses uncertainties. Individual system engineering companies can add there own proprietary parts to the methodology that can serve as part of the competitive differentiation among suppliers. In addition, with the proper initiative from the systems engineering community, the framework for a methodology can be standardized through standards organizations, much as the International Organization for Standardization‟s support for the Open Systems Interconnection (OSI) Reference Model. This standard has served the information systems community well, even though it is a standard for a model as opposed to a technical specification standard. Second, because transitions in management practices need to be sequentially addressed, technical transitions can also be sequentially addressed. Using the definitions presented 20 in this paper for large-scale and for complex systems, a sequential approach can be used to first address implementation trials of SEALS with large-scale systems, while in parallel, investing in research to address how to better address the more uncertain situations surrounding complex systems. This approach would raise the rate of adoption of new methods for system development for complex systems, based on the evidence of successful results with large-scale systems. The initial use of a SEALS-like approach would need to be supported by the buyers of large-scale systems. This community will want concrete evidence of value in order to take more and more aggressive steps in adopting a SEALS-like methodology. In order to build the evidence base for the value of SEALS, agreed upon metrics for measuring the benefits and costs of more agile systems would need to be developed, and a process for collecting information to help the systems engineering community observe and report on actual results would be needed. The Cocomo model efforts related to software costs served this role in helping to address the uncertainties associated with predicting the cost of large software development efforts, and gave clear evidence to the economic values of using Commercial Off The Shelf Software products in large-scale systems. As adoption starts to occur, emergent factors would undoubtedly arise that could either benefit or detract from the movement towards directly addressing uncertainty in systems. New problems and issues will arise that the systems engineering community can respond to. Some will be technical, some will be human organizational and some will be political in nature. For example, political processes do not tend to emphasize the uncertainty that goes along with spending policies. In many cases, large-scale system investments are large enough that they can be subjected to open policy scrutiny and debate. This raises interesting issues about open recognition of uncertainty in a manner that would become acceptable to the policy makers dealing with large system investments, and the constituents impacted by the potential investment policies. This potentially volatile situation calls for the systems engineering community to organize itself to foster a new set of activities related to policy and management practices; i.e., providing visibility of ideas, rationale, and results in circles where it usually is limited in its level of participation. This section outlines what could seem to be too hard to do and will require too much effort to accomplish. However the need for change is significant. Everyone involved with large-scale complex systems knows that much better results are possible. The historical cases in Section 6 provide tangible evidence that changes can result in important benefits. The rate of change will be governed by sequential adoption steps that will extend the overall transition time; time that can be very costly and that can result in competitors surpassing the US systems engineering capabilities and in the creation of more flexible systems than we are able to develop. This situation calls for leaders in the systems engineering communities, the research communities, the communities that invest in systems to develop aggressive plans of action to bring about the much needed change addressed in this paper. References 21 [Agre, 2003] Agre, P., “P2P and the promise of Internet Equality,” Communications of the ACM, Vol. 46, No. 2, pp. 39-42. [Allstar] http://www.allstar.fiu.edu/AERO/TCAS.htm [Andrijcic, 2006] Andrijcic E. and B.M. Horowitz, “A Macro-Economic Methodology for Evaluation of Cyber Security Risks Related to Protection of Intellectual Property,” Journal of Risk Analysis, August, 2006. [ATA, 1972] Airborne Collision Avoidance System, Statement of Airline Policy and Requirements and a Tactical Description of the System, Washington, D.C.: Air Transport Association of America, ANTC Report No. 117, March 24, 1972. [Bar-Yam, 2005] Bar-Yam, Yaneer, “About Engineering Complex Systems: Multiscale Analysis and Evolutionary Engineering,” in: Engineering Self Organising Systems: Methodologies and Applications, Brueckner, S., G. Di Marzo Serugendo, A. Karageorgos, R. Nagpal ,Eds., ESOA 2004, LNCS 3464, Springer-Verlag, 2005, pp. 16-31. [Boehm, 1988] Boehm, Barry, “A Spiral Model of Software Development and Enhancement,” Computer, May 1988, pp. 61-72. [Boehm, 2005] Boehm, Barry and Richard Turner, “Management Challenges to Implementing Agile Processes in Traditional Development Organizations,” IEEE Software, September-October 2005, pp. 30-39. [Bonifacio, 2002] Bonifacio, M., R. Cuel., G. Mameli, And M. Nori, “A peer-to-peer architecture for distributed knowledge management,” in: Proceedings of the 3rd International Symposium on Multi-Agent Systems, Large Complex Systems, and E-Businesses, 2002. [Buede, 1999] Buede, Dennis M., The Engineering Design of Systems: Models and Methods, New York: John Wiley & Sons, 1999. [CIT] http://www.cit.nih.gov/home.asp [Chadwick, 1993] Cahdwick, D. j., V. M. Patel, and L. G. Saxton, “Communications Architecture for Early Implementation of Intelligent Vehicle Highway Systems,” Transportation Research Board, 1993. [CNN, 2004] “Army ends 20-year helicopter program,” CNN, February 23, 2004, referenced at: http://www.cnn.com/2004/US/02/23/helicopter.cancel.ap/. [de Neufville, 2004] de Neufville, Richard, “Uncertainty Management For Engineering Systems Planning And Design,” Engineering Systems Monograph, MIT Engineering Systems Division, March 29-31, 2004. 22 [Garcia, forthcoming] Garcia, A. and B.M. Horowitz, “The Potential for Underinvestment in Internet Security: Implications for Regulatory Policy,” forthcoming in: Journal of Regulatory Economics. [Haimes, 1981] Haimes, Y. Y., “Hierarchical holographic modeling.” IEEE Transactions on Systems, Man, and Cybernetics, 11(9), 606–617, 1981. [Haimes, 1997] Haimes, Y. Y., N. C. Matalas, J. H. Lambert, B. A. Jackson, and J. F. R. Fellows, “Reducing the vulnerability of water supply systems to attack,” Journal of Infrastructure Systems, American Society of Civil Engineers, 4(4): 164–177, 1997. [Haimes, 1998] Haimes, Y. Y., Risk Modeling Assessment, and Management. New York: John Wiley & Sons, 1998. [Haimes, 2004] Haimes, Y. Y. and B.M. Horowitz, “Modeling Adaptive Two-Player Hierarchical Holographic Modeling Game for Counterterrorism Intelligence Analysis,” Journal of Homeland Security and Emergency Management, Vol. 1, No.3, 2004, pp. 1-21. [Horowitz, 2006] Horowitz, B. M. and J.H. Lambert, “Learn As You Go Systems Engineering,” IEEE Transactions on Systems, Man, and Cybernetics, Part A, Vol. 36, No. 2, March, 2006. [Hughes,1998] Hughes, T. P., “Rescuing Prometheus, Four Monumental Projects That Changed the Modern World,” Pantheon Books, 1998 [Kalligeros, 2006] Kalligeros, Konstantinos, Richard de Neufville, and David Geltner, “A Practical Approach to Real Options Valuation in System Design,” Unpublished paper, referenced at http://web.mit.edu/kkall/www/KALLIGEROS+AL_06A.pdf. [Leppingwell, 1989] Leppingwell, John W.R., “Soviet Strategic Air Defense and the Stealth Challenge,” International Security, Fall 1989, Vol. 14, No. 2, pp. 64-100. [Mitre] http://www.mitrecaasd.org/work/project_details.cfm?item_id=153 [Papazoglu, 2003] Papazoglu, M.P. and D. Georgakopoulos, Eds. “SPECIAL SECTION: Service-Oriented Computing,” Communications of the ACM, Vol. 46, No. 10, October 2003, pp. 25-28. [Raytheon] http://www.raytheon.com/products/arc164/ [Reich, 1999] Reich, Yoram, et al., “Building Agility for Developing Agile Design Information Systems,” Research in Engineering Design, Vol. 11, No. 2, August 1999, pp. 67-83. 23 [Rucker, 1974] Rucker, R. A. and T. R. Simpson, “Civil Aviation Midair Collisions Analysis: 1972 Added to the 1964-1971 Results,” FAA-EM-8, Addendum 1 (MTR-6334, Addendum 1), Washington, D.C.: Federal Aviation Administration, 1974. [Sharma, 2001] Sharma, Chetan, Wireless Internet Enterprise Application, New York: John Wiley & Sons, 2001. [Sutherland, 2002] Sutherland, J. and W. van den Heuvel, “Enterprise application integration and complex adaptive systems,” Communications of the ACM, Vol. 45, No. 10, October 2002, pp. 59-64. [Wang, 2006] Wang, Tao and Richard de Neufville, “Identification of Real Options „in‟ Projects,” Proceedings of the 16th Annual International Symposium of the International Council on Systems Engineering (INCOSE), Orlando, Florida, July 2006. [Williamson, 1989] Williamson, T. and N. A. Spenser, “Development and operation of the Traffic Alert and Collision Avoidance System (TCAS),” Proceedings of the IEEE, Vol. 77, Issue No. 11, pp. 1735-1744. [Zhao, 2003] T. Zhao and Tseng C. “Valuing flexibility in infrastructure expansion.” Journal of Infrastructure Systems, Vol. 9, No. 3, September 2003 pp. 89-97. 24