Tutorial A: Cost Risk Analysis/Tim Anderson, The Aerospace Corporation
October 25, 2005
8 a.m. to noon
This tutorial is for acquisition managers, systems engineers, risk managers, and engineering specialists
and will focus on how to work with cost risk analysts to develop realistic cost probability distributions and
the probabilistic nature of each program element of cost in a cost estimate. It will identify how risk
information is used to create a probabilistic cost distribution used for decision-making while integrating
program technical uncertainty and cost modeling variation into a probabilistic cost estimate using state-of-
the-art statistical methods. The risk-adjusted cost distribution can be used to determine the confidence of
meeting budget constraints and also identify the budget necessary to have a specific level of confidence.
The tutorial will show how to organize and display the probabilistic nature of the cost estimate in a way
that is explainable to decision-makers with a range of possible costs as well as their likelihoods.
Examples will illustrate how to set up and run the simulations and then properly interpret the results.
Attendees will receive the Aerospace Corporation’s Cost-Risk Reference Card to use in their day-to-day
work.
Tutorial Instructor Background
Tim Anderson is a Senior Engineering Specialist for The Aerospace Corporation. As a member of the
Engineering and Technology Group, he provides cost estimating and operations research services to the
National Reconnaissance Office. He has supported numerous government space programs.
Retired in 2001, Mr. Anderson served for 20 years in the U.S. Navy and began working in the cost
estimating field in 1994 while assigned to the Naval Center for Cost Analysis. From 1997 to 2001, he was
a military professor at the Naval Postgraduate School, teaching cost estimation, operations research, and
other technical courses.
Mr. Anderson has an M.S. in Operations Research from the Naval Postgraduate School and a B.S. in
Industrial and Operations Engineering from the University of Michigan. He is a SCEA certified cost
estimator/analyst, an instructor with The Aerospace Institute, and a frequent presenter of topics related to
cost estimating and cost risk analysis at forums including the Society of Cost Estimating and Analysis
(SCEA), the Military Operations Research Society (MORS), the DoD Cost Analysis Symposium
(DoDCAS) and the Space Systems Cost Analysis Group (SSCAG). Mr. Anderson also lectures at
military, civil and commercial organizations.
Updated 8/29/05
Tutorial B: Best Practices for Acquiring Software-Intensive Space Systems/Suellen
Eslinger, The Aerospace Corporation
October 25, 2005
8 a.m. to noon
This seminar will present a practical, comprehensive set of software acquisition best practices identified
as being significant contributors to the successful acquisition of software-intensive space systems.
Developed by an Aerospace Corporation software acquisition research project, these best practices are
based on experiences with numerous software-intensive space system acquisitions over the past 20
years. This seminar will cover the best practices through each phase of the National Security Space
program acquisition life cycle in chronological order, including both pre-contract award and post-contract
award activities.
Practical information will be provided to the attendees to enable the immediate application of the best
practices to their programs. Other sources of software acquisition best practices (e.g., from the DoD and
associated organizations) will be discussed. The target audience for this seminar is everyone responsible
for acquiring, or supporting the acquisition of, space systems containing complex, mission-critical
software. Note that it is not targeted solely to software specialists.
Tutorial Instructor Background
Suellen Eslinger is a Distinguished Engineer at The Aerospace Corporation working on a variety of
corporate level initiatives to set the benchmark for evaluating contractor performance on US government
space programs. She has over 35 years experience in development and acquisition of complex/critical
software-intensive DoD and NASA space and ground systems; full life cycle software/systems
engineering and software/systems engineering management; and software engineering and software
acquisition process definition and improvement.
Ms. Eslinger has worked a range of space programs; satellite, ground and launch systems including
Space-Based Infrared System (SBIRS) High and its predecessor programs; SBIRS Low and its
predecessor programs; Global Positioning System (GPS); Space Based Radar (SBR); Air Force Satellite
Control Network (AFSCN); Consolidated Space Operations Center (CSOC); Milstar; Evolved Expendable
Launch Vehicle (EELV); Future Imagery Architecture (FIA) and several other classified programs.
Suellen is a key instructor for courses at The Aerospace Institute, the Aerospace Corporation’s University.
Her contributions include Space Systems Software Program Management, Software Engineering,
Software Acquisition, Software Risk Management and Software Architecture.
Ms. Eslinger holds a M.S. in Mathematics from the University of Arizona and a B.S. in Mathematics from
Goucher College in Maryland.
Updated 8/29/05
Tutorial C: PMI Risk Management Approach and Advanced Schedule Risk Analysis/Keith
D. Hornbacher, M.B.A.Hulett & Associates, LLC, Representing the Project Management
Institute RiskSIG
October 25, 2005
8 a.m. to noon
This workshop has two components: (1) introduction to the Risk Management approach of the Project
Management Institute as found in the Guide to the Project Management Body of Knowledge (PMBOK®
Guide), and (2) detailed discussion of project schedule risk analysis, an outstanding example of what PMI
calls “quantitative risk analysis.” The PMI discussion will explain the definition of risk that PMI uses,
including opportunities as well as threats, and describe the six project risk management processes.
The quantitative schedule risk analysis presentation will show that relying on CPM schedules for accurate
estimates of project completion and identity of the critical path is risky. It shows how better estimates can
be made using risk analysis. The results include: (1) an estimate of the likelihood of overrun, (2) the
degree of overrun or schedule contingency reserve that is needed, and (3) which activities and paths are
most likely to delay the project, which indicates where risk responses should be developed and
implemented. We present the impact of correlation between activity durations and the use of probabilistic
branching on the schedule results. Conditional branching provides a way to look at contingency plan
trigger points. The use of constraints, resources and schedule status within a risk analysis concept will be
discussed. Some software that performs simulations will be discussed and demonstrated.
Tutorial Instructor Background
Keith Hornbacher has held senior project risk management and analysis positions as a practitioner in
assignments across North America and Europe. His experience spans a wide range from complex multi-
billion dollar programs to small innovative product development projects. Mr. Hornbacher was vice
president and senior analyst in the firm that developed quantitative risk analysis methodologies and
Monte Carlo simulation software.
He currently consults and trains in project risk management, scheduling, and Project Management Office
development through the firm, Hulett & Associates, LLC of Los Angeles. His clients have included the US
Government and companies in diverse industries: aerospace and defense, energy, pharmaceuticals,
transportation, construction, large science, medical technology and healthcare. He focuses on qualitative
and quantitative technical, cost and schedule risk analysis and on project scheduling. In addition, he
presents seminars and workshops in advanced project management topics as an adjunct faculty member
of the University of Minnesota.
Mr. Hornbacher has been active in the Project Management Institute (PMI) for more than two decades
and contributed to revisions of the Project Management Body of Knowledge (PMBOK®) Guide. He is the
PMI RiskSIG’s project manager developing a Project Risk Management Handbook and is Director,
Project Risk Management, of PMI’s HealthCare SIG. He is an incumbent member of an M.B.A. Advisory
Council and is a past director of INCOSE’s Northstar (Minnesota) Chapter. Mr. Hornbacher has
presented papers on cost and schedule risk analysis to professional societies including components of
PMI, INCOSE, and the conference of the College of Performance Management (CPM).
Updated 8/29/05
Tutorial D: The Systems Engineering/Risk Management Relationship/George Friedman,
The University of Southern California
October 25, 2005
8 a.m. to noon
Recent experience and disappointments in the development of new complex systems has increased the
emphasis on systems engineering and risk management. Both disciplines have been emerging but only
recently has work been done to truly integrate them. Risk management is not only as a recognized
discipline within systems engineering, but as a crucial path to systems engineering’s improved rigor,
academic acceptance, generation of alternative risk mitigation paths, enhanced employment of testing,
and effective decision making. This tutorial will provide a quick refresher on the basics of risk analysis
and risk management, will discuss qualitative and quantitative methods, cover rationality and decision
theory, examine probabilistic logic of imperfect testing and provide examples.
Tutorial information was integrated from a variety of sources: academic course work, research, working
with the space industry, and professional society involvement. USC has extensive systems engineering
and risk management education and this tutorial has been designed to provide the proven information on
integrating risk management with systems engineering.
Tutorial Instructor Background
Dr. Friedman is presently Adjunct Professor in USC’s Viterbi School of Engineering and Associate
Director of their Systems Architecture and Engineering Program, responsible for research in Systems
Engineering. He has developed and teaches the graduate course “Advanced Topics in Systems
Engineering.” Over 300 students from industry and government have taken his courses toward their
master’s degree and he has sat on a dozen PhD committees. He serves on the advisory committee for
AFIT’s system engineering program and has given guest lectures for INFORMS, INCOSE, MIT, and the
Aerospace Corporation.
Previously, he has had over 40 years of industrial experience on a variety of complex programs, primarily
for DoD. He retired from Northrop as their Corporate Vice President of Engineering and Technology,
where he was responsible for all engineering processes, including systems engineering and risk
management.
He is a founder of INCOSE, served as its 3rd president, was elected a Fellow, initiated a risk management
special interest group, and presently serves as an editor on the peer-reviewed journal, Systems
Engineering.
He is a Fellow of the IEEE and served as VP Publications of the AES Society.
He served on several advisory boards for the USAF, NSF, NASA, CIA, and NATO
Education: BS, UC Berkeley; MS and PhD UCLA
Updated 8/29/05
Tutorial E: Principles and Practical Implementation of Space System Risk Assessment and
Risk Management/Sergio Guarro and Karin Feldman, The Aerospace Corporation
October 25, 2005
1 to 5 p.m.
This tutorial is suited to those that have the accountability of running a successful risk management program,
planning the effort, identifying and assessing risks, managing risk information, working with others in this endeavor
across the enterprise and reporting results to management. Attendees will receive the Aerospace Corporation Risk
Management Reference Card used throughout the tutorial and to be used as a guide on the job. This tutorial will
cover both the principles and underpinnings of space system risk assessment (RA) and risk management (RM) and
the practical implementation issues that arise in complex, multi-stakeholder programmatic environments.
The principles covered include basic definitions of RA/RM processes and models, and both qualitative and
quantitative techniques for risk classification and handling. The relationship of programmatic RA/RM models to
“classic PRA” (Probabilistic Risk Assessment) is explored and discussed as well. Practical RM implementation
issues covered in the course include: multi-stakeholder environments; integration of Government and Contractor RM
processes; calibration of risk classification / quantification scales; vertical integration of RM processes in “program of
programs” hierarchical environments; RA/RM support of programmatic decision-making. The course discussion is
complemented by real-life examples and mini-case-studies.
Tutorial Instructor Background
Dr. Sergio Guarro is the Director of the Office of Risk Planning and Assessment organization within The Aerospace
Corporation. He has Over 30 years of professional experience, with early work in nuclear power plant simulation and
control and nuclear materials research, and the last twenty years dedicated to systems effectiveness and
programmatic decision support disciplines, including risk and reliability assessment and software safety analysis. Dr.
Guarro is an internationally recognized expert in risk assessment and management and the programmatic application
of systems engineering, reliability and safety analysis techniques in support of space system programs.
He was an International Fullbright Scholar at UC Berkeley and a Fellow of the Advisory Committee on Nuclear
Safeguards of the U.S. Nuclear Regulatory Commission. Before joining Aerospace he was a Project Leader with the
Nuclear Systems Safety Program at the Lawrence Livermore National Laboratory.
Dr. Guarro has published a scientific textbook and more that fifty technical papers and articles. He has taught courses
on probability and risk assessment in the Department of Mechanical and Aerospace Engineering at UCLA and also
on software safety and quality assurance at the California State University, Long Beach. He has earned Ph.D., E.D.
and M.S. degrees in Nuclear Engineering, University of California Los Angeles as well as a Diploma di Laurea,
Magna cum Laude, in Nuclear Engineering, University of Bologna, Italy.
Karin Feldman is a Senior Member of the Technical Staff in Aerospace’s Office of Risk Planning and Assessment.
Ms. Feldman supports the development and deployment of the Continuous Aerospace Risk Management and
Assessment (CARMA) risk management tool. Ms. Feldman has also been involved in probabilistic risk assessments,
reliability trade studies, and other reliability analysis projects on various space programs.
She is an instructor for Risk Management at The Aerospace Institute and was a teaching assistant for Engineering
Risk Benefit Analysis at MIT.
Ms. Feldman has earned a M.S. in Nuclear Engineering from the Massachusetts Institute of Technology and a B.S.E.
in Nuclear Engineering and Radiological Sciences from the University of Michigan, Ann Arbor.
Updated 8/29/05
Tutorial F: Program Executability Risk/Jack Wormington, The Aerospace Corporation
October 25, 2005
1 to 5 p.m.
Program management, systems engineering management, and risk management are the basis for the
acquisition program office defining the scope of its work and that of its mission partners and contractors.
The government acquisition community has noted problems and deficiencies in acquisitions that have led
to mission failures and acquisition delays. Acquisition professionals will benefit from this tutorial by
gaining an understanding of how acquisition risks can be identified and handled. The workshop will
include current problems in space acquisition, appropriately accepting a program after baselining, risks to
the program, and approaches to handling program risks. Program examples, along with findings from
recent independent program reviews at AF/SMC, will help participants benchmark their program.
This presentation builds on the 5th SSERMS tutorial. It provides information on how to evaluate a
program for risk and identifies common areas of risk across programs. This tutorial is well suited to those
that manage, measure, report and need to correct program risks.
Tutorial Instructor Background
John R. “Jack” Wormington is Vice President of Program Assessment at The Aerospace Corporation.
Wormington's responsibilities include working with senior leaders of the company's customers to establish
independent review processes for national security space programs, conducting reviews of major
acquisition programs, and making recommendations to improve the executability of programs.
Wormington held senior leadership positions at Boeing Satellite Systems and Hughes Space and
Communications before joining The Aerospace Corporation in 2002. His assignments in commercial
space programs included leadership of the successful development of XM Satellite Radio and other
commercial satellite programs, including all HS702 programs at Hughes Space and Communications.
Wormington served in the U.S. Air Force from 1966 to 1995, retiring as a brigadier general. As a leader of
military space programs, his responsibilities included direction of launch and range operations at Cape
Canaveral Air Force Station and the Eastern Test Range at Patrick Air Force Base in Florida and
leadership of the Advanced Launch System Program. He also created and led the Space Surveillance
and Tracking System Program, among other achievements.
In the Pentagon at Headquarters U.S. Air Force he was planning and program officer for space defense,
Deputy Chief of Staff for Operations and Plans, and the space shuttle program element monitor, Deputy
Chief of Staff for Research, Development and Acquisition.
Wormington has Bachelor of Science degrees in astronautical engineering, physics, and engineering
science from the U.S. Air Force Academy, a master's degree in astronautical sciences from Stanford
University, an MBA from Auburn University and a master's in aerospace operations management from the
University of Southern California.
Updated 8/29/05
Tutorial G: Continuous Risk Management/Barney Roberts, Futron Corporation
Representing The INCOSE Risk Management Working Group
October 25, 2005
1 to 5 p.m.
This tutorial is derived from activities of the members of the Risk Management Working Group (RMWG) of
the International Council of System Engineering (INCOSE). The RMWG is investing energies into how the
risk management process is integrated into the system engineering process, and how does one make
decisions based on the risk analysis results. The RMWG is also devoting energies to the understanding
how to measure the performance of the process. Finally, the RMWG is collecting case studies and other
resources for the practitioner to use. The attendees will be provided with a CD of electronic resources.
This tutorial provides the attendee with the benefits of a formal risk management process. The attendee
will receive an overview of risk-based decision support products that have been proven to deliver high
value to space system projects. The formal risk management process of planning, identifying, analyzing,
handling and tracking are covered as well as specialized analytical approaches to assessing risk, both in
the project management arena as well as the system engineering arena.
Tutorial Instructor Background
Mr. Barney Roberts is the chairman of the INCOSE Risk Management Working Group (RMWG). He has
served as an officer and director prior to taking the chairmanship of the RMWG. He is well published in
professional literature contributing over 70 papers and has won a Best Paper award three times for
INCOSE presentations alone.
Currently Mr. Roberts is the Director for the Futron Risk Management Center of Excellence. His role in
that position is to formalize the risk management practices, procedures and tools that he has successfully
developed and utilized in assisting customers in understanding and managing risk since 1994. He has
developed advanced, integrated, quantitative techniques for cost-technical-schedule risk management
assessments. He has provided risk management training and workshops at most NASA centers and
many of their support contractors’ facilities.
Barney has pushed the state-of-the-art in many diverse areas such as project management, system
engineering, hardware and software development and certification, crew training, aerodynamic test and
analysis, flight mechanics, flight control design and analysis, rocket engine analysis and test,
computational fluid mechanics, heat transfer, and thermal systems design.
Barney’s career includes over 30-years at the NASA Johnson Space Center (JSC) serving in a role of
system engineering and project management for most of the agency’s manned spacecraft development
activities. He served as the Manager of the Planet Surface Systems Office coordinating the strategic
planning and development of concepts and operational strategies for human outposts on the Moon and
Mars was a significant contributor to the development of the Space Shuttle. Among his accomplishments
he has received the NASA Exceptional Service Medal and holds a patent for an aerobraking orbit transfer
vehicle.
Updated 8/29/05
Tutorial H: Smarter Buyer: Mitigating Space Acquisition Risks/Pat Maloney and Frank
Wong, The Aerospace Corporation
October 25, 2005
1 to 5 p.m.
The government and industry have been seeking means to improve acquisitions so that they meet
operational needs, are on schedule and on budget, and nurture the organizations involved. This tutorial, a
condensed version of The Aerospace Corporation’s Smarter Buyer course, is a must for acquisition
professionals. It will provide an understanding of the business climate, industry’s need to perform,
government-industry interaction, and alternative means to incentivize proper corporate behaviors.
Tutorial material, based on interviews with top industry and government officials, will cover acquisition
risks and their implications and potential remedies. Topics will include a brief introduction to the space
business environment, the need for industry programs to generate earnings, concerns of the CEO and
space sector manager and how they impact the program, the industry business development process,
how government decisions impact industry financials, and how to improve acquisitions through better
industry program manager/government program director contract structuring and communications.
Tutorial Instructor Background:
Patricia A. Maloney is the Director of the Aerospace Corporation’s Economic and Market Analysis Center
(EMAC). The EMAC analyzes the space business market conditions, market, and industrial base to
advise government customers on how to incentivize industry.
Previously, Ms. Maloney was with Mobil Corporation for 16 years. She held a variety of positions,
including: CFO for the US Western Region, Torrance, CA; Associate Director of Project Finance for Asia
Pacific, with responsibility for Indonesia, Singapore and Jakarta, Indonesia; Wholesale Marketing
Manager and Strategic Planning Associate for US Marketing and Refining, Fairfax, VA, and Manager,
Crude Oil Operations, Gulf and Far East, New York, NY.
She also worked for ARCO Solar, Inc. on the negative electrode for NiH battery and the electrochemical
deposition of copper indium disulfide solar cells
Pat has earned an M.B.A. in Operations Research and International Finance from the Amos Tuck School
of Business Administration at Dartmouth College and a B.S. in Chemistry from the California State
University at Northridge.
Frank Wong is a Senior Engineering Specialist at The Aerospace Corporation. He has worked in the
satellite industry since 1990 as an electrical engineer, financial analyst, and market research analyst.
Since joining The Aerospace Corporation in 2000, he has researched several industries including remote
sensing, hyperspectral imagery (HSI), launch vehicles, TWTA, and GPS. He is also co-author and
teacher of a “Smarter Buyer” course, designed to help government acquisition personnel understand how
corporations operate from a financial perspective. Prior to Aerospace, he worked at Hughes Space and
Communications, designing payload hardware, creating satellite proposals and supporting the Thuraya
Satellite System Program Office. Mr. Wong has a BS and MS in electrical engineering and an MBA from
UCLA. His MSEE focused on microwave electronics and antennas. His MBA was in finance and strategic
marketing.
Updated 8/29/05