Docstoc

SQA Strategy for the Selection_ Evaluation_ Acquisition_ and

Document Sample
SQA Strategy for the Selection_ Evaluation_ Acquisition_ and Powered By Docstoc
					                               Department of Energy Quality Managers
                             Software Quality Assurance Subcommittee
                                        Reference Document SQAS30.01.00 - 2005




                            SQA Strategy
                                  for the
Selection, Evaluation, Acquisition, and Management
                                       of
                   Off-The-Shelf Software




      April 2005




      Prepared by the
      Software Quality Assurance Subcommittee
      of the
      Nuclear Weapons Complex Quality Managers
      under
      United States Department of Energy
      Albuquerque Operations Office
      Albuquerque, New Mexico 87185
OTS Software                                                                      SQAS30.01.00 - 2005




This report was prepared as an account of work sponsored by an agency of the United States Government.
Neither the United States Government nor any agency thereof, nor any of their employees, nor any of their
contractors, subcontractors, or their employees, makes any warranty, express or implied, or assumes any
legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus,
product, or process disclosed, or represents that its use would not infringe privately owned rights.
Reference herein to any specific commercial product, process, or service by trade name, trademark,
manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or
favoring by the United States Government, any agency thereof or any of their contractors or subcontractors.
The views and opinions expressed herein do not necessarily state or reflect those of the United States
Government, any agency thereof or any of their contractors or subcontractors.




This guideline document will be reviewed and updated on a periodic basis. If you have any
corrections, additions, or deletions, please contact the Quality Manager at your site for the name of a
contact on the Software Quality Assurance Subcommittee.



                                                Page 2 of 52
OTS Software                                                     SQAS30.01.00 - 2005


                              SQAS30.01.00-2005




                                  SQA Strategy
                                      for the
       Selection, Evaluation, Acquisition, and Management
                                          of
                           Off-The-Shelf Software
                                    April 2005




                         United States Department of Energy

                            Albuquerque Operations Office

                                       Abstract

This report provides guidance for defining site-specific software quality assurance
activities for the selection, evaluation, acquisition, and management of Off-The-Shelf
(OTS) software for use in support of Nuclear Weapons Complex (NWC) projects. Of
particular interest are weapon and/or weapon-related applications. OTS software
includes sources such as commercial, government, interagency transfer, internal agency,
and open source products intended to be used without modification by the recipient.




                                      Page 3 of 52
OTS Software                                                  SQAS30.01.00 - 2005


                                Acknowledgments
The Software Quality Assurance Subcommittee (SQAS) of the Nuclear Weapons
Complex Quality Managers initiated Work Item #30 to develop a quality report
addressing an SQA Strategy for Off-The-Shelf Software. The SQAS members who have
contributed to this work item are listed below.

               Maria Armendariz                  SA
               Michael Blackledge                SA
               Frank Campbell                    SR
               Sam Chapman                       Y-12
               Ray Cullen                        SR
               Larry Cox                         LA
               George Custodi                    SR
               Mike Elliott                      UK AWE
               Ken Granger                       Y-12
               Sherry Hardgrave                  YSO
               Phil Huffman                      PX
               Hamilton Hunter                   OR
               Cathy Kuhn                        KC
               Mike Lackner                      LA
               Carolyn Owens                     LL
               Dave Peercy Co-Chair              SA
               Greg Pope                         LL
               Don Schilling                     KC
               Debra Sparkman                    LL
               Ann Stewart                       Y-12
               Anton Tran Co-Chair               NN-121.3
               Roger Ward                        PX




The current version of this document can be found on the Software Quality Assurance
Subcommittee website at http://sqas.lanl.gov/.




                                      Page 4 of 52
OTS Software                                                                                                              SQAS30.01.00 - 2005



                                                               Table of Contents
1      INTRODUCTION ................................................................................................................................ 7
    1.1       PURPOSE.......................................................................................................................................... 7
    1.2       SCOPE.............................................................................................................................................. 7
    1.3       BACKGROUND ................................................................................................................................. 8
2      ROLES AND RESPONSIBILITIES................................................................................................. 10
    2.1     NWC SITE CUSTOMER FOR OTS SOFTWARE................................................................................. 10
    2.2     SOFTWARE PRODUCT SUPPORT GROUP ......................................................................................... 11
    2.3     SUPPLIER OF OTS SOFTWARE ....................................................................................................... 12
      2.3.1     Commercial Vendor.............................................................................................................. 12
      2.3.2     NWC Site .............................................................................................................................. 12
      2.3.3     Other Government Organization .......................................................................................... 13
3      GENERAL SQA STRATEGY FRAMEWORK.............................................................................. 14
    3.1     OTS SOFTWARE CATEGORIZATION ............................................................................................... 16
    3.2     RISK BASED PRIORITIES ................................................................................................................ 16
    3.3     SOFTWARE QUALITY CONCERNS AND REQUIREMENTS ................................................................. 17
      3.3.1     Concerns............................................................................................................................... 17
      3.3.2     Software Quality Requirements ............................................................................................ 18
    3.4     SQA PLAN/CASE FORMAT AND CONTENT .................................................................................... 21
      3.4.1     SQA Plan .............................................................................................................................. 22
      3.4.2     SQA Case.............................................................................................................................. 24
4      SQA STRATEGY PROCESS STEPS .............................................................................................. 25
    4.1       SELECTION .................................................................................................................................... 25
    4.2       EVALUATION ................................................................................................................................. 27
    4.3       ACQUISITION ................................................................................................................................. 32
    4.4       MANAGEMENT .............................................................................................................................. 34
APPENDIX A.                  REFERENCES, DEFINITIONS, AND ACRONYMS ............................................ 36
    A.1.      DOE / NNSA / NWC REFERENCES ............................................................................................... 36
    A.2.      MODEL-BASED PRODUCT ACCEPTANCE EXAMPLE REFERENCES .................................................. 36
    A.3.      OTHER SOURCES ........................................................................................................................... 36
    A.4.      DEFINITIONS.................................................................................................................................. 37
    A.5.      ACRONYMS ................................................................................................................................... 38
APPENDIX B.                  SRS OTS EVALUATION AND ACCEPTANCE PROCEDURE.......................... 39
    B.1.      REFERENCES ................................................................................................................................. 39
    B.2.      RESPONSIBILITY ROLES ................................................................................................................. 39
    B.3.      PROCEDURE STEPS ........................................................................................................................ 39
    B.4.      REQUIREMENTS BASED ON SOFTWARE CLASSIFICATION .............................................................. 41
    B.5.      SOFTWARE EVALUATION PACKAGE TEMPLATE ............................................................................ 41
APPENDIX C.                  SUMMARY OF NUREG/CR-6421 ........................................................................... 42

APPENDIX D.                  SPC CEP FOR SOFTWARE SELECTION & EVALUATION ............................ 45
    D.1.          SCOPE EVALUATION EFFORT ..................................................................................................... 46
    D.2.          SEARCH AND SCREEN CANDIDATE PRODUCTS .......................................................................... 46
    D.3.          DEFINE EVALUATION CRITERIA ................................................................................................ 46
    D.4.          EVALUATE ALTERNATIVES ....................................................................................................... 46
    D.5.          ANALYZE EVALUATION RESULTS ............................................................................................. 47



                                                                       Page 5 of 52
 OTS Software                                                                                                      SQAS30.01.00 - 2005

APPENDIX E.                 SOFTWARE NWC SITE TRANSFER METHODS ............................................... 48
   E.1. BACKGROUND ............................................................................................................................... 48
   E.2. CONCERN ...................................................................................................................................... 49
   E.3. ANALYSIS ...................................................................................................................................... 49
     E.3.1. Integrated Contractor Order ................................................................................................ 50
     E.3.2. Office of Scientific and Technical Information..................................................................... 50
     E.3.3. Radiation Safety Information Computational Center ........................................................... 50
     E.3.4. Proposed Standards.............................................................................................................. 51
     E.3.5. Conclusions .......................................................................................................................... 52



                                                               List of Figures
FIGURE 3-1. SQA OTS SOFTWARE STRATEGY FRAMEWORK ....................................................................... 14
FIGURE D-1. INTEGRATION OF THE COMPARATIVE EVALUATION PROCESS AND SQA STRATEGY ............... 45
FIGURE E-1. NWC SOFTWARE TRANSFER APPROACH ................................................................................. 49



                                                                List of Tables
TABLE 3-1. OTS SOFTWARE QUALITY ASSURANCE CONCERNS ................................................................... 17
TABLE 3-2. SQA PLAN TEMPLATE ............................................................................................................... 23
TABLE 3-3. SQA CASE TEMPLATE ............................................................................................................... 24
TABLE 4-1. SELECTION PROCESS CHECKLIST ............................................................................................... 26
TABLE 4-2A. EVALUATION PROCESS CHECKLIST: SYSTEM/SOFTWARE ARCHITECTURE .............................. 28
TABLE 4-2B. EVALUATION PROCESS CHECKLIST: SYSTEM INTEGRATION .................................................... 29
TABLE 4-2C. EVALUATION PROCESS CHECKLIST: FUNCTIONALITY.............................................................. 30
TABLE 4-2D. SUPPLIER CHECKLIST .............................................................................................................. 31
TABLE 4-3. ACQUISITION PROCESS CHECKLIST............................................................................................ 33
TABLE 4-4. MANAGEMENT PROCESS CHECKLIST ......................................................................................... 35
TABLE C-1. NUREG OTS SOFTWARE CLASSIFICATION SCHEME ................................................................ 42
TABLE C-2. PRELIMINARY QUALIFICATION PHASE FOR OTS SOFTWARE ..................................................... 43
TABLE C-3. PRELIMINARY QUALIFICATION PHASE FOR CATEGORY A OTS SOFTWARE .............................. 43
TABLE C-4. PRELIMINARY QUALIFICATION PHASE FOR CATEGORY B OTS SOFTWARE ............................... 44
TABLE C-5. PRELIMINARY QUALIFICATION PHASE FOR CATEGORY C OTS SOFTWARE ............................... 44




                                                                   Page 6 of 52
OTS Software                                                        SQAS30.01.00 - 2005



1     Introduction

1.1     Purpose

This report provides guidance for defining site-specific software quality assurance
activities for the selection, evaluation, acquisition, and management of Off-The-Shelf
(OTS) software for use in support of Nuclear Weapons Complex (NWC) projects. Of
particular interest are weapon and/or weapon-related applications. OTS software
includes sources such as commercial, government, interagency transfer, internal agency,
and open source products intended to be used without modification by the recipient.

1.2     Scope

Off-The-Shelf (OTS) software is any software that is acquired by a customer from a
supplier with the intent to use the software as part of an application without modification
to the software. This report’s guidance applies to all OTS software used to support NWC
projects. The supplier of the OTS software may be from commercial suppliers,
government agencies, other NWC sites, or Open Source providers. The customer for the
OTS software is an NWC site or organization within an NWC site. The application of
the OTS software includes everything from standard office environments to the analysis,
production, operational use, and stockpile support for weapon or weapon-related
products.

Examples of software that are included within the OTS scope include:

      1. General Purpose: general purpose system software such as operating systems,
         compilers, data bases, foundation classes, libraries, and customer interfaces;

      2. Computational Analysis: physics-based computational software used to provide
         finite element analysis or other physics analyses;

      3. Model-Based Evaluation: model-based software used to support the production
         and verification of weapon/weapon related parts;

      4. Development Support: software used in the development or operational
         use/support of weapon and/or weapon-related product;

      5. Equipment Support: software embedded within OTS equipment; and

      6. Operational Software: software integrated within an operational application
         product.

OTS software that includes selectable parameters for use by the customer to tailor the
software for an application is within the scope of this report. This does not preclude the
customer's acquisition of OTS software source code and other documentation from the
supplier, nor the possibility that the OTS software may be changed in the future by the
customer due to lack of supplier support. Whenever such software is changed by


                                        Page 7 of 52
OTS Software                                                         SQAS30.01.00 - 2005


someone other than an official support agent for that software, then the software is no
longer considered OTS software. The software then falls outside the scope of this report.
However, it should be very clear that planning for the possibility that the software will
someday not be OTS, or to mitigate the effects of such an event definitely is within the
scope of this report.

All OTS software will have application requirements, perhaps requirements that vary
over multiple applications. These requirements dictate what activities are required for the
selection, evaluation, acquisition, and overall management of the software. This will
have the effect of defining the level of formality of the activities.

1.3     Background

Off-The-Shelf (OTS) software is becoming more prevalent within important Nuclear
Weapons Complex (NWC) site applications and in the support of such applications. OTS
software acquired from a supplier for use within a Nuclear Weapons Complex (NWC)
site without modification still requires appropriate assurance that the required use will be
achieved. OTS software applications will have varying requirements and thus varying
assurance needs. Every NWC site has experienced the need to use such software – and
the unfortunate experience with associated problems such as the inability of the software
to perform required functions. This can lead to its lack of use, misuse, or modification
for use – in which case it is no longer OTS software.

 Other problems include the inability of the supplier to provide adequate and timely
support such as training users and corrections of defects. In some cases the iteration of
OTS software versions occurs so often as to make it difficult for the NWC customer to
convert to the new versions – which eventually results in the loss of supplier support
capability. New OTS software versions may include functionality that is undesirable for
the customer's application, or may be developed and produced using different personnel
and different quality assurance procedures. In other cases, the supplier may go out of
business or stop supporting the OTS software product leaving the NWC customer without
support, and occasionally without a product at all due to binding legal issues.

As an NWC site our only options are to select an appropriate software supplier, procure
the software under contractual arrangements that mitigate known problems, and ensure
the software satisfies requirements by evaluating/verifying the software against targeted
applications. In other words -- assure that we are appropriately selecting, evaluating,
acquiring, and managing our OTS software.

Based on general NWC experience in procurement of OTS hardware items as well as
lessons learned from previous OTS software acquisitions, there are two basic steps that
can and probably should take place. The first step (reference EP401422) is to conduct a
supplier evaluation that looks for evidence of a supplier software quality assurance
program:
      1. Documentation specifying software requirements;
      2. Processes in place to control software design;



                                         Page 8 of 52
OTS Software                                                         SQAS30.01.00 - 2005


   3. Tests verifying that software complies with requirements;
   4. Defecting tracking and resolution system in place;
   5. Technical support for potential problems;
   6. Support for software replacements and software upgrades;
   7. Escrow contingencies to provide support in the event supplier ceases to exist; and
   8. Proof of prior customer satisfaction in a similar environment,

The second step is to conduct an application verification that looks for evidence of:
   1. Software performance, i.e., that it performs satisfactorily in its application
      environment based on controlled tests, demonstration/observation, and collected
      data;
   2. Software description to identify supplier, version, attributes and characteristics;
   3. Configuration control to prevent unauthorized change of the software; and
   4. Release of the software into a secured and controlled environment.

As an example of one area of OTS software use, Computer Aided Engineering (CAE)
models are used to generate qualified weapon and weapon-related products. OTS
software is a major part of the process to construct the models, verify their correctness,
translate the models into a production process, and validate the manufactured part.
Assessing the adequacy of the software quality is required by QC-1 since such software is
weapon-related and is used for several of the functional activities covered by QC-1:
controls design or design verification; controls production processes or equipment;
controls testing or inspection processes or equipment; provides analysis capability to
determine product acceptability. QC-1 requires such software to be: consistent with
applicable specifications; satisfy error prevention and software engineering principles;
and assessed for quality in a manner commensurate with the complexity and the risk
associated with failure of the software to meet established requirements.

Technical Business Practices have been implemented as a Nuclear Weapons Complex-
wide activity to provide a common process to managing and engineering weapon and
weapon-related products. TBPs such as TBP-307 “Use of Models in the Product
Realization Process“ and TBP-306 “Software Product Processes” have been established
to address MPBA processes and software product processes. EP4101422 provides
Quality Program Requirements for Contractor-Furnished Software.

TBP-307, establishes NWC business practices required to realize weapon products using
CAE models and associated drawings. The model is an authorized substitute for the
previously used 2D drawing. Models are required to be controlled to ensure that design
and production requirements are properly defined in the product definition and to ensure
that the product definition is properly approved and controlled. Changes to models are
required to be controlled to ensure that the models are authorized, and properly
incorporated and that a record of that decision and its incorporation is maintained.




                                        Page 9 of 52
OTS Software                                                         SQAS30.01.00 - 2005


Software that is the subject of this strategy should be controlled so as to support the
requirements of QC-1, TBP-307, and applicable requirements in TBP-306 for OTS
software. This software is critical to the product realization process and to the capability
to regenerate the model or update the model for future changes.

A generic Model Based Product Acceptance (MBPA) process has been defined in order
to provide assurance that the process can be repeated and has adequate quality provisions
built into the activities. Supporting sets of MBPA quality documents and associated
process activities have been identified to support the MBPA process. Some of these
documents are listed as references in Appendix A. When a weapon product is identified
for which CAE models will be developed, these generic process documents provide
guidance for specific quality/qualification activities and documentation that is to be
provided.

Another example of procedural requirements for Evaluation of Existing and Acquired
Software as implemented at the NWC Savannah River Site (SRS) is summarized in
Appendix B. For nuclear reactor/safety applications, the NUREG/CR-6421 provides a
proposed acceptance process for COTS software that could be used for any OTS
software. Key aspects of this approach are summarized in Appendix C and provide an
example of what the SQA strategy might be for OTS software that is part of a high
integrity application. A specific methodology that can support many of the activities
described in this report is summarized in Appendix D. This method, Comparative
Evaluation Process for Software Selection has been developed by the Software
Productivity Consortium (SPC) and is an instance of the Decision Analysis and
Resolution (DAR) process area of the Capability Maturity Model Integration (CMMI).
Methods of transferring software among NWC sites are discussed in some detail in
Appendix E as supplementary information supporting the acquisition of such software.

The approach taken in this report is to provide a strategy for which the examples of
methods and techniques described in the appendices become more detailed application
instances. Thus, the audience for this report should be able to attain a rapid general
knowledge of the strategy with examples of more specific information that may assist in
specific organization implementations.

2     Roles and Responsibilities

This section introduces concepts for the primary roles and responsibilities related to the
selection, evaluation, acquisition, and management of OTS software. Only the basic
roles and responsibilities will be defined in this section since they will vary with the SQA
formality appropriate for the selected OTS software and specific organization roles at
each NWC site.

2.1    NWC Site Customer for OTS Software

For the purposes of this report, the customer (end-user) of the OTS software is always
associated with an NWC Site. The customer is responsible for defining the applications


                                        Page 10 of 52
OTS Software                                                         SQAS30.01.00 - 2005


and associated requirements for the OTS software use and support. In addition, the
customer is responsible for ensuring that the OTS software satisfies the requirements
prior to application use. The customer may have assistance from other site personnel or
from the supplier of the OTS software in accomplishing these responsibilities.

In order to achieve consistency across NWC sites, it is recommended that each site have a
specific Software Product Support Group to coordinate the OTS software activities,
particularly where a high degree of formality might be required. Otherwise, each NWC
site customer will have to become familiar with and perform the functions of the
Software Product Support Group, as described in the next section.

2.2        Software Product Support Group

The Software Product Support Group is a site-specific organizational entity for defining
and controlling the more significant OTS software selection, evaluation, acquisition, and
management activities. This site-specific Group may exist in many different forms,
including the possibility of multiple groups each with specific areas of responsibility. For
certain OTS software categories (e.g., model-based OTS software used at multiple sites),
the NWC should establish a Group with representatives across all sites that serves as an
NWC Software Product Support Group. A Product Realization Team (PRT) may serve
as such a Group for one or more projects related to OTS software associated with weapon
or weapon-related product. Every site has procurement personnel that typically service
the commercial software purchasing aspects for site personnel. In addition, the site may
have a data base of standard operating environment software (e.g., operating systems,
desktop word processing and graphics, and other high-volume commercial software
purchases) that is maintained by procurement or some other personnel group. Such
groups are considered to be part of a Software Product Support Group organization, but
may have very restricted responsibilities.

In general, the Software Product Support Group should be responsible for ensuring the
following activities are conducted for each OTS software product, as appropriate for the
OTS software product use:
      1.    SQA Plan and SQA Case of evidence are developed and maintained;
      2.    Migration Approach is defined and sustained;
      3.    Intellectual property rights and escrow agreements are established as needed;
      4.    Supplier license and maintenance agreement are obtained and maintained;
      5.    Distribution strategy is defined and implemented;
      6.    Training strategy is defined and implemented; and
      7.    Configuration management (e.g., version control and issue tracking) is
            established.
In addition, the Software Product Support Group might provide general help-desk support
for projects regarding the OTS software products and communicate with the suppliers
regarding problem reports, product upgrades, and license/maintenance issues.



                                         Page 11 of 52
OTS Software                                                         SQAS30.01.00 - 2005


The required SQA activities will vary based on the intended application and use of the
associated OTS software products. The specific activities are determined with the
approval of the Software Product Support Group. The general strategy defined within
this document may provide useful guidance.

2.3     Supplier of OTS Software

The OTS software supplier is responsible for activities specified in the purchase
agreement and the maintenance agreement, if these exist. These activities may vary by
software product and the intended Customer. The supplier should provide a mechanism
(e.g., help desk and/or problem reporting system) for addressing potential questions and
issues from the Customer and/or Software Product Support Group. The supplier may
provide product training and guidance on configuration management of the product, as
well as support for software upgrades that correct defects and provide requested
enhancements and adaptations. Some specific categories of suppliers and their
relationship to the NWC Site Customer and Software Product Support Group are
discussed in the following subsections.

2.3.1    Commercial Vendor

A commercial supplier typically has standard products, sold and supported by the
supplier, and provided for multiple customers on the basis of a market price list. Three
simple rules may be used in some combination to define what kind of products are
supported by a commercial supplier.

Rule 1: The product is offered for sale from a catalog and a price list by the supplier who
has developed and is sustaining the product at the supplier’s own expense. Occasionally,
there is a third party supplier that shares this role with the actual developer/supporter of
the product.

Rule 2: The product conforms to an industry standard which by definition has other
suppliers also adopting and following the same standard.

Rule 3: The product was not developed using government funds or is currently no longer
owned or maintained by the government.

As such, commercial suppliers may not have significant incentive to provide product
capabilities/updates or other support activities for an NWC site customer other than as
typically provided to all customers.

2.3.2 NWC Site

NWC site suppliers of software products typically fall into three categories. The last
category is of limited interest since the customer is not another NWC site.

(1) The software product is weapon or weapon-related and is designed to be Mark
Quality product provided to another NWC site and/or end-use government customer as
part of a weapon’s program.


                                        Page 12 of 52
OTS Software                                                          SQAS30.01.00 - 2005


(2) The software product is developed and possibly supported by site personnel for
applications that are of potential use to another NWC site in applications that are not
weapon or weapon-related or at least are not designed with the intent of being qualified
as Mark Quality product.

(3) The software product is developed and possibly supported by site personnel for
applications that are delivered to customers that are outside the NWC complex.

In the first case, the standard NWC TBP system applies, the supplier is typically a PRT,
and quality assurance aspects are covered by the qualification process described within
the TBP system. In addition, when delivery is specifically to another NWC site for
operational use (e.g., tester software that is used in the test system for a weapon or
weapon-related product) or next assembly use in a weapon or weapon-related product,
the Integrated Contractor Order (ICO) transfer mechanism as described in Appendix E
would apply.

In the second case, there might be software products such as computational analysis
codes, application-specific data base management systems, or some other type of
software product whose function is of use to another site. In this case, the quality
assurance activities as well as sustaining support activities by the supplier site can vary
widely. The OTS software SQA strategy in this case becomes a negotiated agreement
between the supplier and customer sites.

2.3.3   Other Government Organization

In some cases, an NWC site may want to obtain OTS software from another government
agency, or from a common repository of government software products such as supported
by the Office of Scientific and Technical Information (OSTI) described in Appendix E.
In this case, the quality assurance activities as well as sustaining support activities by the
supplier/third party site can vary widely. The OTS software SQA strategy in this case
becomes a negotiated agreement between the supplier and customer sites. Typically the
products are obtained “as is” with minimal SQA assurance information, and minimal, if
any, supplier support. Cost is usually reasonable and much of the time all available
documentation, source code, and other supporting information is included in the product
package.




                                        Page 13 of 52
OTS Software                                                                                                                  SQAS30.01.00 - 2005



3            General SQA Strategy Framework

This section defines a general Strategy within which SQA activities for OTS software
products can be defined, based on the application use and required level of formality.
Whether the supplier or customer conduct the activities depends on several factors, but
frequently must be part of the negotiated agreement between the two parties. Elements of
this Strategy are illustrated in Figure 3-1, briefly discussed in the itemized list below, and
described in more detail in subsequent sections. Application of the Framework to the
primary SQA Strategy activities of Selection, Evaluation, Acquisition, and Management
of OTS Software is described in Section 4.


                                                                                                    Project Application Process
 0
     0
         0
                                                    SW Product Support                                    PRO/E
                                                                                                                       Finite Element
                                                                                                                          Analysis
                                                                                                                                           Manual CAD IQ
                                                                                                                                                                    Manual
                                                                                                                                                                    FBTol
                                                                                                                                                                                  Manual FBTol
                                                                                                                                                                                    Tolerance
                                                                                                                                                                                    Analysis
                                                                                                                                                                                                        STEP
                                                                                                                                                                                                        ACIS



                                                         Group
                                                                                                           Create Conceptual       Check Model Geometry            Create Drawing,            Optional Neutral
             Supplier C                                                                                    Model                   and Creation Standards          Dimensions & Tol            Format Created

                                                                                                                                                                              Manual         CMS/PDM IMS
                                                                                                          Partner With                                                        Check            DA Releases

                Supplier B                                                                  Identify SW
                                                                                                          Production Agency                                                   Model           Model/Drawing

                                                                                                             Manual           FBMach Unigraphics       CMS/PDM                    Manual
                                                                                                              Check            Create In-process        PA Releases               PA Model             DTER
                   Supplier A                                                                 Products      In-Process
                                                                                                             Models
                                                                                                                                   Models              Model/Drawing              Check


                                                                  Site2                                                  Unigraphics
                                                                                                                         Programming
                                                                                                                                              Raw Material
                                                                                                                                             Part Fabrication
                                                                                                                                             and on Machine

                                   Acquisition                    Group                                                   for Machine
                                                                                                                                               Verification

                                                                                                            CimStation     Manual       CimStation

                                        &                                                                 Translate
                                                                                                           Model
                                                                                                                           Check
                                                                                                                           Model
                                                                                                                                         Create CMM
                                                                                                                                           Program
                                                                                                                                                                  CMM
                                                                                                                                                                Inspection
                                                                                                                                                                               Manual
                                                                                                                                                                               Inspection
                                                                                                                                                                                                   CER/QER
                                                                                                                                                                                                   Production



                                  Update Support                   NWC
                                                       Site3                  Site1                                                                                                                              Tailor
                                                       Group                  Group                                                                                                                             Process
                                                                  0 0     0                                                                                                                         0
                                                                                                                                                                                               0
                                                                                                                                                                                              0
                                                                •Select
                                                                •Evaluate                                                                               Project C
                                                                                              Select SW
                                                                •Acquire                       Products                                    Project B
                                                                •Distribute
                                                                                                                               Project A
                                                                •Migrate
                                                                •Manage                 0
                                                                                       0
                                                                                      0
                                                                SW Product C
                                                            SW Product B                                                                                                     Apply
                                                                                                                                                                             Process
                                                        SW Product A
                                                      SQA Plan Activities
                                                      •Acquisition Assmt
                                                      •Capability Eval                                                                                                                            Application
                                                      •CM & Migration Eval
                                                      •Mgmt & Oper Use
                                                      SQA Case Evidence
                                                      •User Requirements
                                                      •Use License Agreement
                                                      •Maintenance Agreement
                                                      •Verification Tests/Results
                                                      •Migration Approach




                                                   SW Product List

                                Figure 3-1. SQA OTS Software Strategy Framework
         1. Identification of Need for OTS SW Product: This activity is typically
            conducted by the ultimate user within the Project Application, possibly with the
            assistance of the Software Product Support Group. The need may be filled with
            existing OTS SW Products within the Software Product List or it may be
            necessary to look for other possible Suppliers.
         2. Selection, Evaluation, Acquisition, and Management of OTS SW Product:
            Given that existing software product capabilities do not satisfy the need, it is
            necessary to go through a process where potential software products are identified


                                                           Page 14 of 52
OTS Software                                                       SQAS30.01.00 - 2005


       and Selected for more thorough evaluation; Evaluated against required
       operational needs including the prioritization and determination whether any
       candidates satisfy the need and, if so, which one is the best solution; contractual
       arrangements negotiated and agreements for any appropriate support established;
       and the selected SW Product is Acquired. Additional verification/validation
       activities may be conducted by the Software Product Support Group and/or the
       ultimate Project Application user
  3.   SQA Plan and Case: The information generated as part of the Selection,
       Evaluation, Acquisition, and Management activities may be usefully organized
       into a SQA Plan (activities to be conducted by the supplier and customer) and
       SQA Case (results of the conducted activities). Concerns are translated into
       specific SQA activities defined in the SQA Plan for each product. The results of
       the SQA activities are captured in the SQA Case and provide evidence for
       qualification of the OTS SW Product for Application use.
  4.   Acquisition & Update Support: These activities occurs only after all the issues
       related to Selection, Evaluation, Acquisition, Distribution, Migration, and overall
       Management have been considered, resolved, and a Supplier and SW Product
       have been determined. These activities are related to the actual purchase/transfer
       of the OTS software product and/or its evolving updates through contractual
       channels. This would also include support activities associated with anomalous
       conditions such as default acquisition of the OTS software source code through a
       legal escrow agreement.
  5.   Application Use and Tailoring: The use of the OTS Software may require some
       site-specific and/or application-specific process tailoring as well as product
       tailoring in the form of input parameter selection and/or wrappers to constrain
       and/or enhance the use of the OTS Software within an Application.
  6.   Software Product Support Group: For some more critical categories of OTS
       SW Product, it is useful to establish a site organization group with responsibility
       for Selecting, Evaluating, Acquiring, Distributing, Migrating, and Managing the
       OTS SW Products for use on Application projects. For management of OTS SW
       Products (e.g., PRO E Computer Aided Design tool) that affect multiple NWC
       sites, there should be a Software Product Support Group with representation from
       all affected sites.
  7.   SW Product List and Repository: In order to leverage the use of OTS SW
       Products, it is proposed that each site maintain a configuration controlled list
       and/or repository of any OTS SW Products in use within the site. For COTS SW
       Products, there are obvious cost benefits for purchase and maintenance contracts
       if there are multiple users within the site. Such a List could include a variety of
       attribute information, including a summary of the SQA Plan/Case information,
       developer, supporter, contact information, and so forth.
  8.   Communication Feedback: Creation of adequate feedback and communication
       mechanisms to facilitate the selection and use of OTS software products by an
       application project and support the transfer and update of software products by the
       supplier is essential.



                                      Page 15 of 52
OTS Software                                                         SQAS30.01.00 - 2005


3.1     OTS Software Categorization

There are many possible categorizations of OTS software. The following categorization
is in terms of the OTS software application use. In some cases, such software may have
multiple applications, in which case the SQA strategy would have to be appropriate to the
most critical applications for the customer. A general set of application categories (many
of which might overlap) for OTS software includes:
      1. general office use;
      2. general corporate information management;
      3. model-based production;
      4. model-based analysis;
      5. weapon or weapon-related development;
      6. weapon or weapon-related operational use; and
      7. safety-related applications.

3.2     Risk Based Priorities

In order to establish what level of formality means for SQA of OTS software, it is
appropriate to establish a general set of risk categories. The following approach
illustrates this strategy, although much more detailed risk strategies can be developed,
such as illustrated in Appendices B and C.

Risk-Based Approach: Each OTS software product used in an application is given a
prioritization ranking. The prioritization ranking is based on the typical use of the OTS
software product and determines the set of activities that should be applied achieve the
requisite quality assurance described in this strategy document. In short, the set of
quality assurance activities should be at least those necessary to ensure requirements for
the specified application are met. A general prioritization ranking is provided below:

a.       High Priority: OTS software product is critical to the application production, use,
and/or support; OTS software product must pass all SQA evaluation activities; any
critical risks will be addressed with an appropriate risk mitigation plan.

b.     Medium Priority: OTS software product is an important part of the application
production, use, and/or support; SQA activities may take a graded approach; OTS
software product must pass all [graded] SQA evaluation activities; any critical risks will
be addressed with an appropriate risk mitigation plan.

c.     Low Priority: OTS software product is required to support the application
production, use, and/or support; limited informal SQA activities may be defined as
necessary; software product must pass all [limited] SQA evaluation activities; any critical
risks will be addressed with an appropriate risk mitigation plan.




                                        Page 16 of 52
OTS Software                                                                    SQAS30.01.00 - 2005


3.3     Software Quality Concerns and Requirements

3.3.1    Concerns

Commercial industries and government agencies develop a wide range of OTS software
to meet a variety of customer needs. Commercial OTS (COTS) software is typically built
for a large customer market and is, compared with custom software, relatively
inexpensive to procure. The commercial supplier typically carries out any maintenance
under license agreement, although such maintenance may not be specifically responsive
to any one customer. COTS software should be, if properly selected, of general
applicability, flexible, reliable and maintained within the supplier’s environment.

The software quality assurance concerns that are addressed in this Strategy primarily
relate to those activities necessary to ensure the OTS software used in an application is
adequately selected, evaluated, acquired, and managed throughout its useful support life.
Some OTS software quality concerns that should be addressed by the SQA activities are
listed below in Table 3-1:
                   Table 3-1. OTS Software Quality Assurance Concerns
 SQA Concern                               SQA Plan and Case Might Address
 1.   Functionality may not meet all       •   Verification of OTS software products in accordance with
      requirements or may provide              application qualification requirements
      undesirable functionality.
                                           •   Requirements specification and verification test plan/case
                                               for each critical software product
 2.   Faults and other problems with       •   Software supplier surveys to provide up front assurance
      the software product will tend not       about the quality methods of each supplier
      to be made apparent by the
      supplier.                            •   Software supplier Maintenance Agreement that provides
                                               known defect conditions with each release
                                           •   Use of a reporting and corrective action process for
                                               identifying, reporting, and resolving software problems
                                           •   Documentation of an application Migration Strategy for
                                               upgrading software products
 3.   Obsolescence may be a problem.       •   Software supplier Maintenance Agreement that provides
      Upgrade releases may be frequent         known defect conditions with each release
      and irregular.
                                           •   Documentation of an Application Migration strategy for
                                               upgrading software products
 4.   Support for obsolete software        •   Documentation of an application Migration Strategy for
      products will tend to be very            upgrading software products
      expense, if not impractical.
 5.   Integration with custom software     •   Such software may require custom operational
      and hardware may require special         parameterization or custom interface scripts specific to the
      OTS software wrappers                    application use
                                           •   Customization artifacts will be retained with the
                                               configuration identification of software as applied to
                                               specific applications.




                                               Page 17 of 52
OTS Software                                                                      SQAS30.01.00 - 2005


 SQA Concern                                SQA Plan and Case Might Address
 6.   Stated conformance to a standard      •   Verification of OTS software products in accordance with
      will be no guarantee of                   application qualification requirements
      interoperability.
                                            •   Requirements specification and verification test plan/case
                                                for each critical software product
 7.   Testing to validate integrated        •   A graded approach to testing; only specific application
      performance may be extensive.             requirements will be tested; selected existing verification
                                                test suites will be run and results compared with known
                                                outputs
                                            •   Appropriate action will be taken based on the test results
 8.   Specialty engineering activities to   •   Specific specialty tests will be an integrated part of the
      ensure reliability, security, and         software verification testing:
      safety requirements may be
      difficult.                            •   Reliability (correctness will be verified as part of the
                                                verification process)
                                            •   Security (installation of any OTS software is done only
                                                after site-specific security procedures have been addressed)
                                            •   Safety (installation of any OTS software is done only after
                                                site-specific safety procedures have been addressed).
 9.  Support activities by the software     •   Documentation of an Application Migration strategy for
     customer (where OTS software is            upgrading software products
     an integrated part of a larger
     system) can be extensive               •   Strategy must address frequency, deployment, installation,
     (frequency and/or complexity of            acceptance, and training related to new releases
     upgrade).
 10. Software integration and retest,       •   Documentation of an Application Migration strategy for
     distribution of new releases.              upgrading software products
                                            •   Strategy must address frequency, deployment, installation,
                                                acceptance, and training related to new releases
 11. Installation, acceptance and           •   Documentation of an Application Migration strategy for
     training for new releases.                 upgrading software products across NWC applications
                                            •   Strategy must address frequency, deployment, installation,
                                                acceptance, and training related to new releases
 12. Response of the OTS supplier to        •   Verification of OTS software products in accordance with
     correction of defects may not              application qualification requirements
     satisfy operational needs.
                                            •   Requirements specification and verification test plan/case
                                                for each critical software product


3.3.2    Software Quality Requirements

This section defines a general framework of required activities to include in the OTS
software product SQA Plan of activities to conduct for the OTS software product and the
SQA Case of evidence/results that show the requirements have been met. The end user
and/or Software Product Support Group organization will provide specific guidance on
which requirements are applicable to which software products. This may be a risk-based
approach with a defined level of formality depending on the criticality of use as part of


                                                Page 18 of 52
OTS Software                                                         SQAS30.01.00 - 2005


the application. Once the OTS software product has satisfied its quality requirements, it
is “certified/qualified” for use in any similar application. However, depending on the
tailoring of the OTS software for the application and the quality assurance activities
conducted on the OTS software product, the application project may require further SQA
activities for any “certified/qualified” OTS software product.

3.3.2.1 General Concerns and Issues

The following issues should be analyzed as part of the OTS software selection and
evaluation in accordance with the specific quality criteria that are established. The OTS
software maintenance agreement(s) should satisfy these criteria. OTS licenses and
agreements should clearly define the legal extent of the supplier responsibilities regarding
through life support including fault repair, on-site support, on-line advice, and escrow for
software source and any associated documentation.

3.3.2.1.1 Technical Competence

Ensure the OTS software supplier has an established, stable support and maintenance
organization. Assess any factors that might jeopardize future essential support
requirements.

3.3.2.1.2 Third Party Aspects

Ensure clear responsibilities are addressed in separate licenses / agreements. Review the
stability of supplier and third party support services.

3.3.2.1.3 Component Life Cycle

Consider the stated / anticipated life span for which the OTS software product(s) will be
supported. Commercial practice is to support products through two major upgrades (2-7
years). Review the effect OTS upgrades will have on the implemented system. Assess
effects of product releases on the overall cost / effectiveness of the system. Recognize
that OTS software product will be non-viable without supplier support.

3.3.2.1.4 Supplier Responsiveness

Ensure the supplier mechanisms for support satisfy the customer’s requirements such as:
continuous on-site supplier field representatives, on-call fault repair services or
workaround solutions, and on-line advice. Identify the various critical supplier software
support scenarios, e.g., emergency response time of one hour or less.

3.3.2.1.5 Product Updating and Replacement

Ensure there is an understanding of the supplier’s policy on product upgrades and new
releases. Assess what support the supplier would provide for the integration of new OTS
packages into the operational system. Will the supplier participate if necessary in the
establishment of an integration and test environment? What are the implications
(configuration, availability, cost) of operational verification of new OTS software


                                        Page 19 of 52
OTS Software                                                        SQAS30.01.00 - 2005


products?   The following guidance is provided regarding timing of OTS software
releases:
   1. Ensure that all through life support-related activities for the OTS software are
      considered in advance of the procurement to ensure that updates and replacements
      are an integral part of the overall acquisition strategy. Plan for change through a
      well-defined OTS software support framework and a “Pre Planned Improvement”
      strategy.
   2. Compatibility problems can exist with inter-dependent OTS software products.
      Use a reference test bed to test and integrate different OTS software and
      application products. Apply strict configuration management for timely release of
      update versions to the operational system.
   3. Harmonize the system software upgrade strategy with the supplier for release of
      new products onto the market. A policy of “no change” may work for a while,
      but the OTS software versions, particularly for commercial suppliers, will quickly
      become out of date and support from the supplier will become increasingly
      difficult.
3.3.2.1.6 Support for Integration Software

Integration software (integrates OTS software with its operational environment) will
either be in-house developed or other OTS software that interfaces the system’s major
OTS software packages. This “wrapper” software must be supported.

3.3.2.1.7 Training

Ensure required training for customers and system integrators is timely and application-
specific. Depending on the complexity of the OTS software and its level of integration
with other software, customer training should be acquired before the operational fielding
of the system. Documentation and training material should be available in advance of
any proposed acquisition in order to ensure that all relevant customer requirements can be
adequately met. Consider training requirements as integral to the acquisition and support
of OTS software components. A training license agreement may be the most cost-
effective approach.

3.3.2.2 Assuring Software Product Capabilities

3.3.2.2.1 Customer Requirements

The SQA Plan should state provisions for assuring that OTS software provided by
suppliers meets established customer requirements. The SQA Plan should state the
methods used to define the customer requirements.

3.3.2.2.2 Verification Testing

The SQA Plan should state provisions for assuring that the software is adequately tested
against appropriate test criteria to ensure customer requirements are satisfied. This may
be satisfied in some cases by having a pre-defined checklist of verification tests that can


                                       Page 20 of 52
OTS Software                                                         SQAS30.01.00 - 2005


be run to check out the customer requirements. Existing verification evidence for the
OTS software product may satisfy this. Customer-generated verification tests may be
required to satisfy specific application requirements. Typical information might include:
adequate ratio of software testers to developers; functional and structural test coverage
measured and exceeds a specified percentage; testing is performed on platforms and
versions (O/S, compiler, hardware, peripherals) similar to the anticipated environment.

3.3.2.3 Configuration Management and Migration of Software Products

3.3.2.3.1 Product and Media Control

The SQA Plan should state provisions for assuring that there are adequate methods and
facilities used to store, secure, document, and distribute released baseline versions of the
software products. These provisions should include the methods and facilities to identify
the product storage media, documentation to store and retrieve the media, and any
protection and access controls necessary for the media. The baseline versions should be
controlled through standard OTS software configuration management identification,
control, status accounting, and audit activities.

3.3.2.3.2 Migration Approach

The SQA Plan should state provisions for assuring that migration of a baseline software
release to an updated release or to a different software product satisfies customer
requirements. The Migration Approach may provide specifics of the maintenance
agreement, verification testing, and configuration control responsibilities.

3.3.2.3.3 Problem Reporting and Corrective Action

The SQA Plan should state provisions for assuring that a problem reporting and
corrective action capability is established in order to document and communicate
software problems with the supplier. This capability should be coordinated with the
configuration management of software products and the migration of software products
when updated by the supplier.

3.3.2.4 Management and Operational Use of Software Products

The SQA Plan should describe assurance work product documentation, standard practices
such as configuration management, reviews and audits, and risk management, and
automated tools and methods for implementing the standard practices that together
facilitate the management and operational use of the software products.

The SQA Plan should describe any operational constraints or guidance that assures the
proper use of the software products and their output results.

3.4   SQA Plan/Case Format and Content

The SQA Plan describes what activities will be conducted for a specific software product
to provide assurance that the product can be used in applications. The SQA Case


                                        Page 21 of 52
OTS Software                                                       SQAS30.01.00 - 2005


describes the results of the quality assurance activities and serves to document the
evidence for “certifying/qualifying” the OTS software product can be used in specific
applications. Guidelines on the general format and content of the Plan and Case that
address the SQA concerns and requirements are provided in the following subsections. A
more detailed discussion of the specific SQA activities of Selection, Evaluation,
Acquisition, and Management will be described in Section 4.

3.4.1   SQA Plan

The SQA Plan should use a standard template. Since the SQA Plan is specific to an OTS
software product, one possible representation for weapon or weapon-related applications
would be to use the NWC support drawing system. In that system, a part number would
define the software product with a six-digit part drawing number, e.g., 123456. Thus, the
SQA Plan would have a slaved support drawing number, e.g., SQ123456. Standard
version and issue information would be used with a title as follows:

                    Software Quality Plan, <software product name>

The Software Product Support Group should determine a reasonable standard format for
each SQA Plan. The end user, Software Product Support Group, and/or designated
representative should develop this plan. The Software Product Support Group might
enforce the standard format. The format for a specific product SQA Plan should include
the general topic areas as illustrated in Table 3-2. Where not applicable, a statement to
that effect is suitable. Additional sections may be added as needed or required by site-
specific procedures. The SQA Plan information may be included in other appropriate
documentation to simplify the number of documents. It is expected that the specific
format and content will be appropriately tailored for the application.




                                      Page 22 of 52
      OTS Software                                                                   SQAS30.01.00 - 2005


                                      Table 3-2. SQA Plan Template
            SQA Plan Section                                  SQA Plan “how to” description
1. Introduction
1.1 Purpose                            Identify the boundaries of the OTS software products covered by this Plan.
                                       Why is the SQA Plan being written? How will plan contribute to success of
                                       project? What is intended use of SW covered by SQA Plan?
1.2 Scope                              What is scope of SQA Plan? Who does it apply to? Who is intended
                                       audience? Which OTS software items does this SQA Plan cover? (Names
                                       and abbreviations) Which portions of the general software life cycle apply to
                                       each OTS software item?
1.3 Deviations                         What, if any, are the deviations from the SQA Strategy? May not be
                                       applicable for some plans.
1.4 Document Content                   Include a brief roadmap of the document content.
2. Product Description
2.1 Software Products                  Identify the specific OTS software product(s) covered by this Plan.
2.2 Software Functions                 Summarize the required software product functions.
3. SQA Activities
3.1. Selection of Software Products    Describe the process for determining candidate software products and
                                       suppliers. Describe the methods for ensuring the technical competence and
                                       responsiveness of the OTS software product supplier. Describe methods used
                                       for a preliminary screening of the candidates and suppliers.
3.2. Evaluation of Software Product    Describe the methods used for evaluating that the OTS software product
Capabilities                           satisfies the established customer requirements. Describe the methods used
                                       for evaluating that the maintenance agreement will contain adequate
                                       provisions for product updates, licenses for use, training, and the legal extent
                                       of the supplier responsibilities regarding through life support including fault
                                       repair, on-site support, on-line advice, and escrow for source (if necessary).
3.3 Acquisition of Software            Describe the methods used for establishing the contractual agreements,
                                       maintenance agreements, escrow agreements, and actual purchase of the OTS
                                       software product determined to be the best fit from the evaluation process.
                                       Describe the methods used for ensuring the OTS software products are
                                       adequately identified, verified, distributed to customers as a common
                                       baseline, and supported when updated by the supplier.
3.4 Management of Software             Describe the methods used for ensuring the OTS software products and
Products                               associated media are identified and controlled for consistent access and use.
                                       Describe the methods for ensuring problems are reported and corrected, the
                                       status of updated OTS software products is reported, and changes to OTS
                                       software products are controlled. Describe assurance work product
                                       documentation, standard assurance methods, automated tools for
                                       implementing assurance practices, and risk management methods that
                                       enhance the assurance of the OTS software products. Describe practices that
                                       assure the proper operational use of the OTS software products and their
                                       output results.
Appendices
References                             Include references to other documents used to support, supplement, or
                                       complete this document.
Definitions and Acronyms               Use this section to identify terminology that may require explanation for the
                                       audience for which the document is intended. Include a comprehensive list of
                                       the acronyms used throughout the document.




                                                  Page 23 of 52
      OTS Software                                                                 SQAS30.01.00 - 2005


     3.4.2    SQA Case

     The SQA Case should use a standard document template. Since the SQA Case is specific
     to an OTS software product, one possible representation would be to use the NWC
     support drawing system. In that system, a part number would define the OTS software
     product with a six-digit part drawing number, e.g., 123456. The Case results could be
     included in an appendix to the SQA Plan, or as a separate volume of the SQA Plan.

     The Software Product Support Group should determine a reasonable standard format for
     each SQA Case. The end user, Software Product Support Group and/or designated
     representative conducts the SQA activities and capture the results in this Case. The
     Software Product Support Group might enforce the standard format. The format for a
     specific product SQA Case should include the general result information areas as
     illustrated in Table 3-3. Where not applicable, a statement to that effect is suitable.
     Additional sections may be added as needed to support site-specific requirements. The
     quality evaluation evidence may be obtained by different methods depending upon the
     availability of the information, including: observation of existing practices, interviews
     with suppliers and/or customers, supplier surveys, review of existing quality assurance
     information, and conduct of verification tests.
                                       Table 3-3. SQA Case Template
         SQA Plan Section                                   SQA Plan “how to” description
Appendix X:                            SQA Activity Results
X.1. Selection of Software Products    Selection of potential OTS Software products results:
                                       - Technical competence and responsiveness of the supplier matrix.
                                       - Maintenance agreement evaluation.
                                       - License agreement evaluation.
                                       - Compatibility of supplier update process and OTS software product
                                       migration approach.
X.2. Evaluation of Software Products   Customer requirements definition evaluation results:
                                       - Software product verification strategy and results evaluation
                                       - Transition and migration evaluation
                                           Compatibility of supplier update process
                                           OTS software product migration approach
X.3 Acquisition of Software Product    Contractual agreements, maintenance agreements, escrow agreements results:
                                       - Specific OTS software product selected by evaluation process
                                       Identification of the OTS software product(s) and supplier(s) results:
                                       - Common baseline identification
                                       - Delivery schedule, update schedule and identification per agreements
X.4 Management of Software             Management evaluation results:
Product                                - Identification and assessment of assurance work product documentation.
                                       - Identification and assessment of operational practices.
                                       - Configuration Management Practices Evaluation
                                           Version control; Issue tracking; Release management.




                                                  Page 24 of 52
OTS Software                                                          SQAS30.01.00 - 2005



4     SQA Strategy Process Steps

Section 3 provides a framework for guiding a customer in the general concerns and
activities that might be conducted in acquiring an OTS software product from a supplier.
The framework includes construction of a “Plan” of activities related to the potential use
of the OTS software product and a “Case” of evidence from those activities that
documents what was actually done. As appropriate, this “Case” of evidence should be
updated throughout the life of the subject OTS software product, including the support
for migration of new versions of the OTS software for the target applications.

This section summarizes a process-oriented strategy for the primary activities of the
Plan/Case in terms of selection, evaluation, acquisition, and management processes. All
the elements of the Software Quality Plan/Case framework introduced in Section 3 are
covered in somewhat more detail in this section. The appendices provide additional
concepts that may be applied within this process framework. The checklists provided in
this section are derived from several of the references, particularly [SPC-CHK], with
additional information provided to address concerns presented in Section 3, and tailored
to cover the specific process steps of the strategy. These process activities and checklists
can be tailored to satisfy an appropriate level of formality depending on the specified
requirements of the customer, supplier, OTS software product, and targeted applications.

4.1   Selection

The selection process consists of identifying the criteria concerning a required OTS
software product capability, the potential OTS software products, and the suppliers that
may satisfy the criteria through an appropriate evaluation process. This criteria may be
evolved from an initial screening version used in this Selection process step to a more
detailed version used for the Evaluation process step. There are existing processes and
criteria such as in [SRS-5.07], [EP401422], [SPC-COTS], [JA1005], [NATO96], and
[NATO97] that can apply to any OTS software. The activities related to this selection
process can be part of an overall application software quality plan or a plan specific to the
selection, evaluation, acquisition, and management of OTS software products.

The Selection process for the OTS Software should include the following activities:
       a.      identify the application need and use;
       b.      define technical requirements the software is to satisfy;
       c.      determine programmatic (cost, schedule, legal) requirements;
       d.      identify at least one OTS software product/supplier for evaluation;
       e.      develop the evaluation criteria/approach based on graded formality; and
       f.      establish roles and responsibilities, including involvement of a site
               Software Product Support Group.
A checklist of questions and concerns related to these selection activities is provided in
Table 4-1.


                                        Page 25 of 52
OTS Software                                                                      SQAS30.01.00 - 2005


                               Table 4-1. Selection Process Checklist
         Questions                                               Remarks

1.   Have the application          It is important that the software architecture within the application
     need and use been             context is understood and that the potential need for OTS products is
     explicitly and clearly        established within that architecture. Potential use and integration
     described?                    with the application must be reasonably understood prior to the
                                   selection of possible OTS candidates.

2.   Have the architecture,        The evaluation checklists of the Evaluation Process in Section 4.2
     integration, and              can serve as a guide for understanding what these requirements
     functionality                 might include. Typically it is useful to develop a high level (initial
     requirements been             screening) set of requirements to cull the potential OTS candidates,
     identified?                   followed by development of the more detailed (evaluation) set of
                                   requirements that will be used during the Evaluation process.

3.   Have any specific             The formality with which the evaluation evidence must be developed
     safety, security,             depends on the application use, and in particular on whether there are
     reliability concerns been     any significant specialty engineering concerns. If the application has
     identified?                   significant safety, security, or high reliability requirements, then it
                                   may be necessary to develop more detailed requirement
                                   specifications based on level of formality criteria such as in
                                   Appendices B & C.

4.   Have programmatic             It is useful to determine the difference between through life (not just
     cost, schedule, and legal     development vs purchase) cost and schedule estimates for full scale
     concerns been                 custom development and support of software for which the potential
     addressed?                    OTS software is to satisfy. Potential legal concerns such as
                                   addressed by the Evaluation Process Supplier checklist in Section 4.2
                                   and many of the Management Process checklist items apply during
                                   the Selection process as well.

5.   Are there any potential       An initial screening of existing OTS software candidates based on
     OTS software                  the initial criteria from checklist items #2, #3, #4 above should result
     candidates?                   in at least one, but hopefully more than one candidate. If there are no
                                   initial candidates, it is likely that custom software will need to be
                                   developed. If there are candidates, then the Evaluation Process can
                                   be conducted.

6.   Have evaluation criteria      The initial screening criteria per checklist items #2, #3, and #4 above
     covering technical,           should be fully developed such as illustrated in the Evaluation
     programmatic, and             Process of Section 4.2. Process details for specific tasks that might
     supplier capability been      be of use can be found in the examples and references in Appendix B
     developed?                    and Appendix D.

7.   Have evaluation,              Roles and responsibilities for the evaluation of potential OTS
     integration, and support      software products as well as for the subsequent integration and
     roles and                     through-life support should be identified and documented. Of
     responsibilities been         particular interest is the potential role and responsibility of a
     identified?                   Software Product Support Group.




                                               Page 26 of 52
OTS Software                                                        SQAS30.01.00 - 2005


4.2   Evaluation

The evaluation process consists of analysis activities to determine how well the selected
OTS software product(s) and OTS software supplier(s) satisfy the specified evaluation
criteria, and as appropriate which OTS software product/supplier is most qualified for
subsequent acquisition activities.

The evaluation of the selected OTS software candidates is done per specified evaluation
criteria using a defined process. There are existing processes and criteria such as in
[SRS-5.07], [EP401422], [SPC-COTS], [JA1005], [NATO96], and [NATO97] that can
apply to any OTS software. The evaluation process includes documentation of results as
a “case” of evidence and could include involvement of a site Software Product Support
Group, depending on the application and the intended user group.

The evaluation process should include the following activities based on the level of
formality and type of application:
       a. evaluate each OTS software product and supplier using the architecture,
          integration, and functionality product checklists and the supplier checklist;
       b. analyze the evaluation data using a decision support model with a scoring
          strategy, weighted/prioritization scheme, and decision method;
       c. consider any additional constraints, limitations, expert judgment that might
          impact the analysis results;
       d. identify the best OTS software product(s) from the analysis results; if no OTS
          software product satisfies the minimum evaluation criteria, then report that
          result to the appropriate management; and
       e. obtain engineering and management consensus on the acceptance of the best
          (if any) OTS software product(s) for acquisition.
A checklist of questions and concerns related to these evaluation activities is provided in
Tables 4-2a, 4-2b, 4-2c, and 4-2d.




                                       Page 27 of 52
       OTS Software                                                                   SQAS30.01.00 - 2005


              Table 4-2a. Evaluation Process Checklist: System/Software Architecture
              Questions                                                    Remarks

1.   Has the software architecture    It is important that the software architecture is designed first and the OTS
     been explicitly and clearly      products are selected that fit the architecture. A collection of products with no
     described?                       architecture in mind is nothing but a collection of products.

2.   How was the software             The derivation of a software architecture is not a straightforward activity. If the
     architecture derived? Did        architecture development was not the product of some appropriate level of
     peers review the architecture?   analysis and reviewed by peers, it is probably not a good architecture.

3.   Is this a new architecture to    If the architecture or infrastructure of the system is new to the team, how did they
     the developing organization?     gain the knowledge to develop the architecture? Did the knowledge stay in-
                                      house?

4.   Is the software architecture     If only one or two people understand the architecture, there needs to be a plan to
     well understood by the           impart the knowledge of the architecture to the team.
     development team?

5.   How have the OTS                 The decision of which functionalities in the architecture are to be supported by
     components been allocated to     OTS products should not be made in an ad hoc fashion. There should be some
     the architecture?                type of a formal or semiformal analysis of the allocation of OTS products to
                                      support an architecture.

6.   Are all the interfaces in the    No interface within the architecture should be ignored or the knowledge of it
     architecture understood?         taken for granted. The impact and operation of all interfaces within an
                                      architecture must be understood.

7.   Is there a controlling           Middleware products typically form the backbone of an architecture; ensuring
     middleware layer? How much       that there is enough knowledge and experience on the team is critical. If the
     experience does the              middleware is new to the team, they should be getting explicit training.
     development team have in the
     middleware products?




                                                   Page 28 of 52
               OTS Software                                                                    SQAS30.01.00 - 2005


                             Table 4-2b. Evaluation Process Checklist: System Integration
            Questions                                                           Remarks

1.   Is the product new to the     If the product is new to the organization, obtain as much information as you can afford (e.g.,
     organization?                 training, supplier support/cooperation, test bed operations).

2.   Is the team qualified and     It is not enough to just have experience and training with a product operation; the integration
     trained in the integration    capabilities of a product can be quite different than the operation of the product. For example,
     capabilities of the           the graphical user interface may provide an intuitive operational control while the application
     product?                      programming interface may be fully object-oriented.

3.   Have the impacts on the       Each product has impacts on the available system resources; while some impacts are
     system resources of the       straightforward, some impacts are hidden or vary over time. For example, the impact of
     product been properly         temporary disk files varies over time, the use of them may not be documented, and the files
     analyzed?                     may or may not be deleted after their use. Product impacts on network utilization are typically
                                   not documented as well.

4.   Does the product have an      There are many different methods that products use for providing integration capabilities. An
     Application Programming       API is generally accepted as one of the most effective methods. If the product has a different
     Interface (API)?              integration method, some form of translator will most likely need to be developed.

5.   Does the development          The capabilities and complexities of APIs will vary widely across products. Simply knowing
     team understand the           that a product has an API is insufficient information to make critical decisions on estimating
     capabilities and              the integration effort. Some product APIs will have hundreds of routines and may use a
     complexities of the API?      paradigm that is quite different from the operational paradigm. If the team has no experience
                                   with such an API, some form of training is needed.

6.   Has the integration of the    Operational or control integration is only part of the problem. Analyzing the product’s data
     product’s data with the       formats and determining how the data is to be integrated in the system’s data scheme is
     system data been fully        essential. If a product uses a commercial DBMS, beware of attempts to directly access the
     analyzed?                     physical database, especially directly writing to a product’s database. It is very easy to destroy
                                   the internal integrity of the product’s database.

7.   Has the size of all the       Ensure that ALL the integration efforts are analyzed and the integration effort is estimated.
     integration effort for the    Each application integration effort in the development of a system may be different from the
     product been estimated?       others. Site corporate OTS software integration and subsequent updates may be very time-
     How confident are the         consuming, costly, and disruptive to organizations, particularly if done very frequently.
     estimations? Is there a
     measure of confidence?

8.   Does the product have any     While this situation initially appears to be an “ideal” situation (no integration development is
     built-in integration with     required), the situation is in need of further investigation. How cooperative are the suppliers
     other components of the       with respect to the upgrade schedules of the other supplier’s products? What is the history of
     system?                       past upgrades? Do both suppliers have continuing plans to continue to support the integration?

9.   Is the product upgrade        The product upgrade cycle for every product designed to be in the system should be known,
     cycle known? How will         and the impacts on the development and deployment of the products should be reflected in the
     the project react to the      support planning. Is there a problem reporting & corrective action capability provided by the
     upgrade?                      supplier as part of the upgrade approach?

10. What is the history of         As the products have been upgraded, determine how well the supplier has maintained
    forward and backward           compatibility in the past because this is a reasonable indication for the future.
    compatibility of product
    upgrades?



                                                            Page 29 of 52
          OTS Software                                                                     SQAS30.01.00 - 2005



                            Table 4-2c. Evaluation Process Checklist: Functionality
             Questions                                                         Remarks

1.   Have system/software             The system and software requirements should be allocated to components that are
     requirements been allocated      candidates for a OTS product. You should not select a set of products first and then
     to the products?                 try to “force” the requirements to fit the products.

2.   How are requirements that        For any large system that uses a significant number of OTS products, there WILL be
     are not met by the product       some requirements that were intended to be satisfied by a product but are not. How
     being handled?                   has the development team proposed to handle these situations?

3.   Has functionality that is        For any large system that uses a significant number of OTS products, there WILL be
     beyond the allocated             functionality that the products have that goes beyond the requirements. While this
     requirements been addressed?     may seem to be that the customer is “getting something for free,” there are several
                                      issues that must be considered. If you as the contractor must provide a system
                                      warranty, how do you plan to respond to a situation where the use of this extra
                                      functionally causes a system failure. Who is responsible—the user for using the extra
                                      functionality or the developer for delivering a system that can be grossly affected by
                                      the use of this extra functionality? The answer is by no means clear. Understanding
                                      the impact of the use of this extra functionality is critical.

4.   How easy is the product to       Basic and simple.
     use?

5.   How consistent is the look       A system that contains a collection of products that have different looks and feel can
     and feel of the product with     be very annoying to the user.
     the rest of the system?

6.   Were components on which         Dependencies such as shared operating environments, databases, and network
     the product is dependent         resources must be analyzed.
     properly considered?

7.   Is the product’s overall         Proceeding with system development with products whose operation is not fully
     operation well known?            understood is risky.

8.   How have known problems          All products have some problems, some simple and innocent and some that may
     that exist in the product been   significantly impact system operations.
     handled?

9.   What standards do the            Obtain certifications from the supplier for the standards that the products support if
     products support?                proof is needed by the customer; this should save testing for standards compliance.

10. How mature is the product?        Does the product have a proven track record, or is the product a new one? If the
                                      product is new, as much information as possible about its use should be obtained,
                                      such as references from current users.

11. How was the product               The evaluation and selection process should use a method that provides a
    evaluated and selected? Ad        comprehensive form of comparison. Simple checklists and weighted averages may not
    hoc or with a defined             be sufficient. Ad hoc selection is clearly insufficient.
    process?

12. Were the selection criteria       If selection criteria were used in the process, they should contain criteria that address
    sufficient?                       architectural and management issues in addition to product functionality criteria.



                                                       Page 30 of 52
            OTS Software                                                                      SQAS30.01.00 - 2005



                                            Table 4-2d. Supplier Checklist
              Questions                                                           Remarks

1.   What relationship does the          Working with knowledgeable suppliers whenever possible is best but not always
     company have with the               possible. The organization should work hard to understand the capabilities and
     supplier?                           philosophies of new suppliers.

2.   How long has the supplier been      Purchasing a new product from an upstart supplier could be risky.
     in business?

3.   How financially stable is the       If you are considering a product from a young supplier, perform due diligence on the
     supplier?                           supplier.

4.   What is the supplier’s market       This is a reasonable indication of the future viability of the product.
     share?

5.   Have data rights been properly      The data rights issue can vary widely with different suppliers. These rights may also
     negotiated with the suppliers?      involve execution of an escrow agreement (see #8 below).

6.   Have cost-effective licensing       Also try to negotiate a licensing agreement that is better than listed pricing. The larger
     agreements been worked out          the system’s use of a product, the more likely it is to strike a deal.
     with the suppliers?

7.   Has the organization executed       This is an absolute requirement, especially for suppliers of products that are critical to
     mutual nondisclosure                the system.
     agreements with all OTS
     suppliers?

8.   Is the supplier willing to modify   This is a rare but viable circumstance. Requesting a modification to a supplier’s
     the product to meet                 product requires a strong relationship with the supplier. Is the supplier simply a
     requirements currently outside      distributor with a third party being the actual developer or a component piece
     the scope of the product            developer? If such modification cannot be arranged, are there other mechanisms in
     capability?                         place (such as an escrow arrangement for the purchase of the software source for the
                                         purpose of transferring support rights)?

9.   Is there an escrow arrangement      An escrow arrangement allows for the potential of purchase rights to the software
     in place in case the supplier       product in case the business is not willing to continue product support or is unable to
     goes out of business or decides     continue product support due to business bankruptcy or buyout by another company.
     not to support the product.         Although there are many legal issues associated with such an arrangement, it may be
                                         useful if the OTS software product is part of the customer’s essential business.




                                                          Page 31 of 52
OTS Software                                                         SQAS30.01.00 - 2005


4.3   Acquisition

The acquisition process consists of those actions taken to acquire and provide sustaining
support for the software product(s) chosen through the evaluation process. These actions
include the purchase or transfer of the software product(s) from the supplier to the
customer through formal and/or informal methods. Most of the concerns described
below should already have been considered during the Selection and Evaluation steps,
but in some cases the acquisition mechanisms for purchase, transfer, acceptance, and
future support coordination have not been completely defined.

The purchase and/or transfer of OTS software should include concerns such as licensing,
maintenance, intellectual property rights, and escrow. The establishment of a support
concept for the OTS software, which might include supplier maintenance concepts as
well as organization-specific migration concepts, should be in place or defined. Specific
actions to be performed by the organization after the OTS software has been delivered
should be defined prior to delivery, including appropriate verification, validation,
acceptance testing, or other such efforts by the supplier/customer. An on-going
configuration management approach and qualification for application use should be in
place. If the software includes transfer from one NWC site/organization to another NWC
site/organization, then methods and techniques such as described in Appendix D should
be considered.

The acquisition process should include the following activities:
       a. establish the customer-supplier contractual (or other) mechanism for obtaining
          the OTS software product(s) as accepted for acquisition through the
          evaluation process;
       b. ensure all required artifacts that have been identified as a required part of the
          software product delivery are itemized in the contractual mechanism;
       c. ensure any acceptance criteria are defined;
       d. ensure adequate funding is available;
       e. consider possible software transfer mechanisms as identified in Appendix E,
          as appropriate; for weapon/weapon-related product, ensure it is qualified in
          accordance with TBP-306;
       f. transfer the OTS software product from the supplier to the customer with
          appropriate payment and record keeping as required by organizational
          procedures of both the supplier and customer;
       g. perform any customer checkout/verification activities as required by the
          Plan/Case upon receipt of the OTS software; and
       h. coordinate for any future updates/deliveries/training as defined by the
          acquisition contract.
A checklist of questions and concerns related to these acquisition activities is provided in
Table 4-3.




                                        Page 32 of 52
          OTS Software                                                                       SQAS30.01.00 - 2005


                                       Table 4-3. Acquisition Process Checklist
          Questions                                                            Remarks

Have the technical evaluation      The checklists for the Application System/Software Architecture, System Integration,
results been completed,            Functionality, and Supplier OTS software product technical requirements provide important
documented, and understood?        evaluation results. These results should be the basis for initiating the acquisition process.
                                   These results are the basis for establishing the customer-supplier contractual mechanism and
                                   on-going through-life support for the OTS software product.

Has the customer-supplier          The customer-supplier contractual mechanism may be as simple as a purchase order or as
contractual mechanism been         complex as a multi-year maintenance agreement with support clauses for on-site support,
documented?                        correction of defects, integration of improvements, training courses, and escrow/intellectual
                                   property considerations. As appropriate, the NWC site legal department should be involved in
                                   any reasonably complex contract.

Does the contractual               By definition, software includes not just the executable program, but source code, development
mechanism clearly identify         documentation, verification/validation test suites, user guides, theory manuals, configuration
what the software product          management and quality assurance information, operational and support environment
delivery includes?                 constraints and so forth. The contract should clearly identify what part of the full software
                                   package is to be delivered – including what part of the full software package is included in any
                                   escrow/intellectual property agreement.

Is there any acceptance            Prior to any delivery (initial or subsequent updates) of the OTS software product, there may be
criteria that either the           certain acceptance activities that are required to be conducted and documented. Such activities
supplier will provide as part      may include simple function/physical audits of the delivery, identification of changes from one
of the delivery, or the            version to the next, execution of a standard (perhaps updated) integration test suite for the
customer that the delivery is      customer’s application(s), and so forth. These activities may involve either or both customer
dependent upon?                    and supplier. Such acceptance criteria should be clearly understood by both the customer and
                                   supplier, including specific roles and responsibilities by both.

Is adequate funding planned        It is particularly important to ensure that the through-life support concept for the OTS software
for the initial delivery as well   product within the intended application(s) has been defined, including how such support is to be
as for the intended through-       funded. Purchase of an OTS software product (particularly COTS) has many hidden costs over
life support?                      and above the initial procurement cost. If an OTS software product is integrated within a
                                   critical application, any updates to that OTS software product will require appropriate updates
                                   to the application – including possible software support updates for any “wrapper” software that
                                   has been custom developed. These “costs” can be very high. NWC site updates to a corporate
                                   OTS software product capability (e.g., financial management database which is typically a
                                   COTS product) can cause a tremendous ripple effect throughout all the organizations at the site.

Has the contractual                Software transfer mechanisms as identified in Appendix E may be appropriate, particularly for
mechanism considered the           site to site transfer of weapon/weapon-related product, which should be qualified in accordance
transfer mechanisms that           with TBP-306. Electronic transfer, physical transfer, and other such mechanisms such as Web-
might be most efficient?           based transfers may be economical and efficient. Initial delivery as well as delivery of updates
                                   should be a consideration.

Has the transfer mechanism         For NWC site to site transfers, a no-cost ICO is appropriate. If the software product is weapon
included appropriate funding,      or weapon-related, then it should be a qualified product with documented qualification
checkout/verification,             information so no additional verification activities specific to the software should be required.
training considerations?           For non-site suppliers, evidence that the software satisfies an acceptance test suite may be
                                   required – as conducted by the customer and/or the supplier depending on the acquisition
                                   contract. On-site or off-site training schedules should be understood as part of the transfer
                                   process.




                                                          Page 33 of 52
OTS Software                                                         SQAS30.01.00 - 2005


4.4   Management

The management process includes all through life activities by responsible managers and
engineers that support the strategy for selection, evaluation, and acquisition of OTS
software products, including possible iteration of the activities as necessary to maintain
requisite capabilities. The management process is an overarching activity that must be
integrated within the general software management strategy of each NWC site, and site
application projects.

The management process for OTS software must include or be integrated with the NWC
site management of software quality, software configuration management, software
qualification for application use, and software support. Where there are OTS software
safety and/or security concerns the management process must integrate with the site
safety and security management activities. In short, it must be the case that the
management process for OTS software must be an integrated part of site-specific
software management and any associated application-specific system management.
Constraints, limitations, pitfalls, and other issues related specifically to the management
of OTS software are summarized in this section.

The management process should include the following activities:
       a. ensure the adequacy of the software quality plan for the OTS software product
          in its integrated application;
       b. ensure the adequacy of the software configuration management approach for
          the OTS software product in its integrated application;
       c. ensure the adequacy of the Plan/Case framework selection, evaluation, and
          acquisition processes for the OTS software product in its integrated
          application;
       d. ensure the adequacy of the software migration approach for the OTS software
          product in its integrated application;
       e. ensure risks associated with the OTS software product in its integrated
          application are identified and managed; and,
       f. ensure management of OTS software products is integrated with the
          management of the other products that comprise the integrated application.
A checklist of questions and concerns related to these management activities is provided
in Table 4-4, and to a large extent overlaps and integrates with the checklists of Tables 4-
1 through 4-3.




                                        Page 34 of 52
      OTS Software                                                                        SQAS30.01.00 - 2005


                                    Table 4-4. Management Process Checklist
            Questions                                                         Remarks

1.   Has the software quality          The software quality plan for OTS software products should not be assumed to be a
     plan for the products been        straightforward issue. This is a key aspect of the Software Quality Plan/Case
     properly documented?              framework.

2.   Has the approach for              The CM of OTS products should not be assumed to be a straightforward issue. The
     configuration management          initial version and subsequent versions will need to be controlled by the supplier
     (CM) of the products been         and the customer so there is no confusion as to what product is being used. Since
     properly planned?                 suppliers typically support no more that a couple of past versions, it is critical that
                                       the customer migration/support concept address how new versions will be migrated
                                       as they become available.

3.   Have the supplier’s               Straightforward – if supplier surveys as part of the selection, evaluation, and
     technical support                 acquisition process have been adequately performed.
     capabilities been fully
     evaluated?

4.   Is there a plan to negotiate      If the customer has required the use of OTS products and those products do not
     unsupported requirements?         support all the requirements, there needs to be a negotiated agreement between the
                                       customer and supplier to either remove some of the requirements or incorporate the
                                       unsupported requirements in future software releases.

5.   Has the integration testing       In a system that contains several OTS products, integration testing of these products
     of the products been              should begin as early in the development cycle as possible. Also, as new products
     thoroughly planned?               are added to the integration test bed, retesting of previously integrated products
                                       may be required. Since you can never have a full understanding of the behavior of
                                       the products, those products that are critical and tightly coupled may need to be
                                       retested as the integration proceeds.

6.   Have the risks related to         This aspect should be a normal part of a sound development process, the use of
     using each product been           OTS developed products is no different. When including the product into a system,
     identified and managed?           the contractor (not the supplier) assumes whatever risks that are associated with the
                                       product.

7.   Have all the integration          The integration of each product has many varying issues. All integration efforts
     costs been properly               should not be treated in the same fashion. In particular, if custom “wrapper”
     estimated?                        software is to be developed in order to adequately integrate the OTS software
                                       product in the specified applications, then the cost of this custom software should
                                       be included in the integration cost.

8.   Have all the related product      Do not forget to consider supplier technical support and maintenance costs during
     costs been considered?            development as well as deployment. Include potential training costs as well.




                                                      Page 35 of 52
OTS Software                                                                SQAS30.01.00 - 2005



APPENDIX A.             References, definitions, and Acronyms
A.1.            DOE / NNSA / NWC References
[DOE G 241.1-1] DOE Guide 241.1-1, Guide to the Management of Scientific and Technical
                Information
[DOE O 241.1A] DOE Order 241.1A, Scientific and Technical Information Management
[EP401422]         EP410422, Quality Program Requirements for Contractor- Furnished
                   Software
[QC-1]             DOE/NNSA Weapon Quality Criteria (QC-1)
[TBP-DEF]          Technical Business Practice, Definitions
[TBP-306]          Technical Business Practice, Software Product Processes
[TBP-307]          Technical Business Practice, Use of Models in the Product Realization
                   Process
A.2.            Model-Based Product Acceptance Example References
[CE1A0852F]        Technical Characteristics, MBPA Software Quality Assurance Strategy
[PQ1A0852]         Qualification Plan, MBPA Process
[SQ1A0852A]        Software Quality Plan, MBPA Accepted Software List
[SQ1A0852B]        Software Quality Plan, MBPA Pro/Engineer Product Family
A.3.            Other Sources
[CMMI]           “Capability Maturity Model Integration (CMMI) for Systems Engineering/Software
                 Engineering Integrated Product and Process Development,” Software Engineering
                 Institute Version 1.1, CMU/SEI-2002-TR-011 and CMU/SEI-2002-TR-012, Pittsburgh,
                 PA., March 2002.
[EPRI NP-5652] EPRI NP-5652, “Guidelines for the Utilization of Commercial Grade Items in Nuclear
               Safety-Related Applications.”
[IEC1226]        IEC 1226, “The Classification of Instrumentation and Control Systems Important to
                 Safety for Nuclear Power Plants,” International Electrotechnical Commission, February
                 6, 1993.
[JA1005]         JA 1005, “Software Supportability Program Implementation Guide,” Society of
                 Automotive Engineers, January 2004.
[NATO96]         NATO (Draft), “COTS Software Acquisition Guidelines and COTS Policy Issues – 1st
                 Revision,” NATO Communications and Information Systems Agency, January 12,
                 1996.
[NATO97]         NATO (Draft), “NATO Guidelines for the Integration of Off-The-Shelf Software,”
                 Working Paper AC/322(SC/5)WP/4, NATO C3 Board Information Systems Sub-
                 Committee, June 30, 1997.
[NUREG-6421]     NUREG/CR-6421, “A Proposed Acceptance Process for Commercial Off-the-Shelf
                 (COTS) Software in Reactor Applications,” Lawrence Livermore National Laboratory,
                 UCRL-ID-122526, March 1996.
[SPC-CEP]        Polen, Susan, Rose, Louis C, and Phillips, Barbara, “Component Evaluation Process,”
                 SPC-98091-CMC, Version 01.00.02, Software Productivity Consortium, Herndon,Va.,
                 1999.



                                           Page 36 of 52
OTS Software                                                                     SQAS30.01.00 - 2005

[SPC-CHK]         Rose, Louis and Alexander, Roger, “COTS Integration Questions and Checklist,” SPC-
                  2000018-CMC, Version 01.00.01, Software Productivity Consortium, Herndon,Va.,
                  May 2000.
[SPC-COTS]        Phillips, Barbara and Polen, Susan, “Add Decision Analysis to Your COTS Selection
                  Process,” Software Technology Support Center, CrossTalk, April 2002, pp 21-25.
[SRS-5.05]        Conduct of Engineering and Technical Support Procedure Manual, “Software
                  Classification” Manual E7, Procedure 5.05, Rev 0, Savannah River Site, May 1, 2001.
[SRS-5.07]        Conduct of Engineering and Technical Support Procedure Manual, “Evaluation of
                  Existing and Acquired Software,” Manual E7, Procedure 5.07, Rev 0, Savannah River
                  Site, May 1, 2001.
[VOAS98-1]        Voas, Jeffrey, “COTS Software: The Economical Choice?,” IEEE Software,
                  March/April 1998, pp 16-19.
[VOAS98-2]        Voas, Jeffrey, “An Approach to Certifying Off-the-Shelf Software Components,” IEEE
                  Software, May-June 1998.
[VOAS98-3]        Voas, Jeffrey, “The Challenges of Using COTS Software in Component-Based
                  Development,” IEEE Computer, June 1998, pp 44-45.
A.4.             Definitions
The definitions below are specifically for the application to OTS software products for use by an NWC
organization/site within specified project applications as described in this report. The terms may have
alternate meaning in other contexts.

OTS Software         Any software that is acquired by a customer from a supplier with the intent to use the
                     software as part of an application without modification to the software.

                     Note: when the supplier is a commercial company, the term COTS is typically used;
                     when the supplier is a government agency, the term GOTS is typically used.

Selection            Identification of criteria concerning a required OTS software product capability and
                     potential OTS software products and suppliers that may satisfy the criteria through
                     an appropriate evaluation process.

Evaluation           Analysis to determine how well the selected OTS software product(s) and OTS
                     software supplier(s) satisfy the specified evaluation criteria, and as appropriate which
                     OTS software product/supplier is most qualified for subsequent acquisition.

Acquisition          Actions taken to acquire and support the software product(s) chosen through the
                     evaluation process for specific application use. These actions include the purchase
                     or transfer of the software product(s) from the supplier to the customer through
                     formal and/or informal methods.

Management           The through life activities by responsible managers and engineers that support the
                     strategy for selection, evaluation, and acquisition of OTS software products,
                     including possible iteration of the activities as necessary to maintain requisite
                     capabilities.




                                              Page 37 of 52
OTS Software                                                    SQAS30.01.00 - 2005


A.5.        Acronyms
API            Application Program Interface
CAE            Computer Aided Engineering
CEP            Comparative Evaluation Process
CMMI           Capability Maturity Model Integration
COTS           Commercial Off-The-Shelf
DA             Design Agency
DAR            Decision Analysis and Resolution
DOE            Department of Energy
GOTS           Government Off-The-Shelf
GA             Government Agency
ICO            Integrated Contractor Order
IEC            International Electrotechnical Commission
IEEE           Institute of Electrical and Electronics Engineers
ISO            International Organization for Standardization
MBPA           Model Based Product Acceptance
NATO           North Atlantic Treaty Organization
NNSA           National Nuclear Security Administration
NWC            Nuclear Weapons Complex
OSTI           Office of Scientific and Technical Information
OTS            Off-The-Shelf
PA             Production Agency
PRP            Product Realization Process
PRT            Product Realization Team
QAIP           Quality Assurance Inspection Procedure
QAP            Quality Assurance Plan
QC-1           Weapon Quality Criteria document by DOE/NNSA
RSICC          Radiation Safety Information Computational Center
SPC            Software Productivity Consortium
SQ             Software Quality document prefix (used before part number)
SQA            Software Quality Assurance
SQAP           Software Quality Assurance Plan
SQAS           Software Quality Assurance Subcommittee
SRS            Savannah River Site
STI            Scientific and Technical Information
SW             Software
TBP            Technical Business Practice




                                   Page 38 of 52
OTS Software                                                     SQAS30.01.00 - 2005



APPENDIX B.           SRS OTS Evaluation and Acceptance Procedure
This appendix provides a summary of the key aspects of the reference [SRS-5.07]. The
pertinent information of interest for this document concerns "acquired" software. The
procedure provides requirements and responsibilities for the evaluation of acquired
software to determine the acceptability of the software for its intended use. Specific
organizations and responsibilities unique to SRS are identified, but what is of most
interest is the procedural steps for the evaluation of "acquired" software.

Acquired Software is defined as "Commercial grade software that will be qualified by
SRS or developed and qualified by the supplier prior to being placed in service for its
intended use in accordance with Manual E7, Section 5.0 or QAP 20-1.” Key to the
evaluation process is a Software Evaluation Package (SEP), which is the set of
documents that is utilized to demonstrate adequate confidence that the existing or
acquired software is acceptable for its intended end use. The "SEP" corresponds to the
SQA Case of evidence.

B.1.           References

Quality Assurance Manual 1Q
1. QAP 17-1, Quality Assurance Records Management
2. QAP 20-1, Software Quality Assurance
Conduct of Engineering and Technical Support Manual E7
1. Procedure 1.20, Engineering Document Numbering System
2. Procedure 2.14, Specifications
3. Procedure 3.10, Determination of Quality Requirements for Procured Items
4. Procedure 3.46, Replacement Item Evaluation/Commercial Grade Item Dedication
5. Procedure 5.01, Software Engineering & Control Overview
6. Procedure 5.03, Software Quality Assurance Plan
7. Procedure 5.05, Software Classification
8. Procedure 5.07, Evaluation of Existing and Acquired Software
9. Procedure 5.10, Software Requirements
10. Procedure 5.20, Software Design and Implementation
11. Procedure 5.40, Testing, Acceptance and Turnover
B.2.           Responsibility roles
1.   Design Authority: responsible for performance of software evaluations
2.   Technical Manager: assigns Responsible Engineer to evaluate and prepare SEP
3.   Responsible Engineer: conducts the evaluation and documents results in SEP
4.   Independent Reviewer: conducts independent review of the SEP
B.3.           Procedure Steps

There are several procedural steps described in [SRS-5.07], most of which apply to
"existing" software. The steps that pertains to "Acquisition of Commercial Grade


                                      Page 39 of 52
OTS Software                                                         SQAS30.01.00 - 2005


Software" are essentially as follows, although the steps may be conducted in some other
order if appropriate.

Step 1: OTS Grade Definition

The Responsible Engineer determines that the software to be acquired from a supplier
will meet functional requirements and critical characteristics, and will be adequate for its
intended end-use.

Step 2: Requirements and Adequacy for Intended Use

The Responsible Engineer determines that the software to be acquired meets commercial
grade definition criteria in accordance with Manual E7, procedure 3.46.

Step 3: Acceptance Criteria

The Responsible Engineer establishes acceptance criteria in accordance with the software
classification necessary to qualify the software for use at SRS in its intended end-use.

Step 4: Acceptance Methods

The Responsible Engineer determines the method(s) that will be used to accept
commercial grade software. The method will prove that the software meets the
acceptance criteria. The methods that can be used to accept commercial grade software
are:
Method 1 – Special Test Cases. This may be supplemented by one or more of the
      following methods:
Method 2 – Commercial Grade Survey of the Supplier
Method 3 – Acceptable Supplier Performance Record.
These acceptance methods will provide a means to ensure that the commercial grade
software meets the acceptance criteria. The selection of an acceptance method for
commercial grade software is based on the following: acceptance criteria, available
supplier information, quality history, and degree of standardization. Refer to EPRI NP-
5652 for requirements associated with Methods 1, 2 or 3.

Step 5: Procurement Order and Support Considerations

Based on the requirements established in the previous steps, the Responsible Engineer
prepares a procurement specification per Manual E7, procedure 2.14 and/or purchase
order in accordance with Manual 7B. The purchase order identifies the method that the
supplier should use to report software errors to SRS and the method that SRS should use
to report software errors to the supplier. Software is procured in accordance with the
SQAP and established Procurement methodology. (See Manual E7, procedure 3.10 for
procurement options and governing procedures.)




                                        Page 40 of 52
OTS Software                                                          SQAS30.01.00 - 2005


B.4.           Requirements Based on Software Classification

The reference [SRS-5.05] provides a specific software classification scheme for graded
formality from high to low as follows:

Safety Class or A: Software that has a direct effect on nuclear safety protection systems
that keep exposure to the general public below the offsite regulatory or evaluation
guidelines as determined in safety analyses.

Safety Significant or B: Software that has an indirect effect on nuclear safety protection
systems or toxic materials hazard systems that keep nuclear or toxic material hazard
exposure to the public and workers below regulatory or evaluation guidelines as
determined in accident analyses. This level also includes software whose results may be
used to make decisions that could result in death or serious injury as determined by
evaluation in accident analyses. Software whose output is for defense in depth is not
classified as Level B. This level also does not include software used to perform tasks
related to hazards that are routinely encountered in general industry and construction, and
for which national consensus codes, standards, or site programs exist to guide safe design
and operation.

Production Support or C: Software important to continued function of the business,
including environmental and other monitoring and compliance related activities. This
includes software whose output information, if incorrect, may result in losses exceeding
$1,800,000 (1999) or 6 months program loss. This level also includes software used in
support of the SRS Emergency Plan for communications with local, state, and Federal
Government Agencies.

General Services or D: Software important to the day to day function of the business but
whose output, if incorrect, will not result in losses exceeding $1.8 million dollars (1999)
or 6 months outage.

The reference [SRS-5.07] then provides for each of the general steps (as well as the
primary "acquisition" software step) specific guidance depending on the software
classification. That is, all of the general steps are required for software classification of
"A" or "B", whereas for classification of software as "D", no specific evaluation is
required. There are some differences for a software classification of "C".

B.5.           Software Evaluation Package Template

The reference [SRS-5.07] includes a template for completing the evidence required by
the procedure steps. In general this template includes paragraphs for each section plus a
section for pertinent references and a section for SEP approval.




                                        Page 41 of 52
OTS Software                                                                 SQAS30.01.00 - 2005



APPENDIX C.             Summary of NUREG/CR-6421
This appendix provides a summary of the key aspects of the reference [NUREG-6421].
Although this reference is focused on COTS software and safety-specific applications,
the approach is equally valid for any OTS software for use in high integrity applications
critical to the business of the organization. As such, the term “OTS” is used.
NUREG/CR-6421 bases its proposed OTS software acceptance process on a rather
comprehensive review of IEEE, ISO, and IEC SQA standards and guidelines. The
primary standard used for classification guidance is IEC 1226, "The Classification of
Instrumentation and Control Systems Important to Safety for Nuclear Power Plants,"
Dated February 6, 1993. Consistency with Nuclear Regulatory Commission practices
was a guiding principle. The OTS software of interest typically includes compilers,
operating systems, software supplied in Programmable Logic Controllers or Application
Specific Integrated Circuits, and software in commercial industrial digital control systems
– primarily as used in high integrity/safety-related systems.

The proposed acceptance process establishes an initial set of four criteria for OTS
software product identification and its safety category. Based on the safety category,
three sets of additional criteria, graded in rigor, are applied to approve (or disapprove) the
product. These criteria fall roughly into three areas: product assurance, verification of
safety function and safety impact, and examination of usage experience of the OTS
product in circumstances similar to the proposed application.

The proposed acceptance process is broken into two phases, preliminary qualification
phase and detailed qualification phase, based on the following classification scheme in
Table C-1.
               Table C-1. NUREG OTS Software Classification Scheme
  IEC 1226 Category                                   Example Systems
                        Reactor Protection System; Engineered Safety Features Actuation System;
  A
                        Instrumentation essential for operator action.
                        Reactor automatic control system; Control room data processing system; Fire
  B
                        suppression system; Refueling system interlocks and circuits.
                        Alarms, annunciators; Radwaste and area monitoring; Access control system;
  C
                        Emergency communications system.
  OTS Usage Category                                       Description
  Indirect              Directly produces executable modules that are used in A, B, or C applications.
                        CASE systems, or other support systems that indirectly assist in the
  Support               production of A, B, or C applications, or software that runs as an independent
                        background surveillance system of A, B, or C applications.
  Unrelated             Software that has no impact on A, B, or C applications.
  OTS Safety Category                                       Criteria
                        OTS software used directly in a system important to safety; safety category
  1
                        determined by the criteria of IEC 1226



                                           Page 42 of 52
OTS Software                                                                      SQAS30.01.00 - 2005

                           OTS software used to control the configuration of an executable software
 2                         product that is used in a system important to safety; safety category is the
                           same as that of its output.
                           OTS software that supports production of category A, B, or C software, but
 3                         does not directly produce or control the configuration of such software; safety
                           category is "unclassified".
                           OTS software that has no impact on category A, B, or C software or systems;
 4
                           safety category is "unclassified".

The preliminary qualification phase activities and acceptance criteria are indicated in
Table C-2. Additional qualification criteria for OTS Software in Categories A, B, and C
are summarized in Tables C-3, C-4, and C-5. Additional details can be found in
[NUREG-6421].
              Table C-2. Preliminary Qualification Phase for OTS Software
                Criterion                                         Acceptance Criteria
  1. Risk and Hazards Analysis               Risks and hazards analyses shall be used to identify system-
                                             level safety functions required.
  2. Identification of Safety Functions      The safety functions (if any) that the OTS software will
                                             perform shall be identified.
  3. Configuration Management                The OTS software shall be under configuration and change
                                             control.
  4. Determination of Safety Category        The safety category of the OTS software shall be determined.
                                             Further tables for acceptance criteria depend on the category
                                             A, B, or C.



     Table C-3. Preliminary Qualification Phase for Category A OTS Software
  Criterion                                       Acceptance Criteria
  5.A          The OTS product shall have been developed under a rigorous SQAP.
  6.A          Documentation shall be available for review that demonstrate good software engineering
               practices were used.
  7.A          It shall be demonstrated that the OTS product meets the requirements identified in criterion
               2 (Table C-1).
  8.A          It shall be demonstrated that the OTS product does not violate system safety requirements
               or constraints.
  9.A          The interfaces between the OTS product and other systems or software shall be identified,
               clearly defined, and under configuration management.
  10.A         The OTS product shall have significant (< 1 year) operating time, with severe-error-free
               operating experience. At least two independent operating locations shall have used a
               product of identical version, release, and operating platform encompassing the same or
               nearly the same usage as the proposed usage. Any adverse reports shall be considered.
               The configuration of the products in the experience data base shall closely match that of
               the proposed OTS product.
  11.A         All errors, severe or otherwise, shall be reported to and analyzed by the OTS supplier.
               Procedures and incentives shall be in place to ensure a high level of demonstrated
               compliance, or the OTS supplier shall demonstrate with statistical certainty that the error
               reporting system achieves this compliance. An error tracking, documentation, and
               resolution procedure shall document each error from report to resolution.
  12.A         Additional validation and testing shall be performed if needed to compensate for a small
               amount of missing documentation or alterations in configuration.




                                              Page 43 of 52
OTS Software                                                                       SQAS30.01.00 - 2005



   Table C-4. Preliminary Qualification Phase for Category B OTS Software
  Criterion                                        Acceptance Criteria
 5.B           The OTS product shall have been developed under a quality assurance plan and a
               systematic software development process.
 6.B           Documentation shall demonstrate criterion 5.B.
 7.B           It shall be demonstrated that the OTS product meets the requirements identified in
               criterion 2 (Table C-1), and that its reliability is sufficiently high that it does not present a
               high frequency of challenges to category A systems.
 8.B           The OTS product shall be consistent with system safety requirements.
 9.B           The OTS product shall have operated satisfactorily in similar applications. The version
               and release of reported experience may not be identical to the proposed OTS product, but
               a consistent configuration management program and well-managed update program
               provide traceability and change control.
 10.A          Error reporting, tracking, and resolution shall be consistent and correctly attributable to
               version and release, and procedures and incentives are in place that ensure demonstrated
               compliance during the first year after a version is released. The version and release
               proposed have no major unresolved problems. A current bug list shall be available to
               OTS product customers as a support option.



   Table C-5. Preliminary Qualification Phase for Category C OTS Software
  Criterion                                        Acceptance Criteria
 5.C           The OTS product shall have been developed according to good software engineering
               practices. Minimum documentation shall be available or reconstructable. Minimum
               V&V tasks shall have been performed.
 6.C           Minimum documentation described in criterion 5.C, including V&V task documentation,
               shall be available for inspection.
 7.C           The product’s performance of its intended effect shall be verified. The OTS product may
               enhance safety by improving surveillance, improving operators’ grasp of application
               operations, assisting in maintenance activities, reducing demands on category A or B
               systems, monitoring or reducing the effects of radiation releases, or similar purposes.
 8.C           It shall be demonstrated that the OTS product cannot adversely affect the safety functions
               of category A or B systems or software and that it will not seriously mislead operators.
 9.C           The OTS product must be shown to operate without serious malfunction in the targeted
               application.
 10.C          An error reporting scheme shall be planned or in place that tracks malfunctions of this
               OTS product in applications controlled by this product. Documentation and records
               retention allow error histories of 5 years or length of service, whichever is shorter.




                                              Page 44 of 52
OTS Software                                                        SQAS30.01.00 - 2005



APPENDIX D.               SPC CEP for Software Selection & Evaluation
The Software Productivity Consortium (SPC) has developed the Comparative Evaluation
Process (CEP) in response to its members’ need for a repeatable and systematic process
for evaluating and selecting components and COTS software. A description of this
process is presented in reference [SPC-COTS] and more extensive information is
provided in the report [SPC-CEP] and on the SPC website: http://www.software.org.
This appendix summarizes this process as an example of a method that supports many
aspects of the SQA strategy framework presented in Sections 3 and 4.

The CEP is an instance of the Decision Analysis and Resolution (DAR) process area of
the Capability Maturity Model Integration (CMMI) and a structured decision-making
process. The relationship between CMMI’s DAR and CEP is discussed in [SPC-COTS].
The process provides detailed guidance to alleviate the typical problems that occur during
an evaluation.

The CEP is made up of five top-level activities, which are typical of any trade-off study:
(1) Scope Evaluation Effort; (2) Search and Screen Evaluation Candidates; (3) Define
[detailed] Evaluation Criteria; (4) Evaluate Alternatives; and (5) Analyze Evaluation
Results. These activities are summarized below and depicted in Figure D-1 with a
mapping to the process strategy steps described in Section 4. Each activity has three to
five sub-activities, which are explained in detail in the technical report [SPC-CEP].



                                         1. Scope Evaluation




                                                                   2. Search and Screen
                                                                   Evaluation Candidates
        5. Analyze Evaluation
               Results




                                                                 3. Define [Detailed]
              4. Evaluate Alternatives                           Evaluation Criteria




 Figure D-1. Integration of the Comparative Evaluation Process and SQA Strategy


                                         Page 45 of 52
OTS Software                                                        SQAS30.01.00 - 2005


D.1.           Scope Evaluation Effort

This activity sets the expectations for the level of effort and schedule for the remaining
activities within CEP. It provides the expected number of OTS software products to
search, screen, and evaluate. This activity is part of the Selection strategy described in
Section 4.1 and typically involves iterative feedback from future activities that requires
redefining the scope for one or more of the activities because there were too many or too
few possible candidate components located during the search or that the addition of
criteria should change the scope. A high level understanding of the technical
requirements the OTS software product is to support level and the level of formality
appropriate for the intended application will assist in the scoping effort. This activity
should be sufficient to establish expectations for required resources and a schedule to
accomplish the activities. This activity can also be considered to establish the draft
“Plan” as described in Section 3.

D.2.           Search and Screen Candidate Products

The search for candidates first requires that the initial high priority search criteria and
thresholds be defined. The high level understanding of the technical requirements and
level of formality can be used to establish the search criteria and thresholds for required
functionality and key constraints. The checklists in Section 4 can be used to establish the
initial search criteria. The criteria should be reasonably broad so that the search is not
limited by too many constraints, since detailed evaluation criteria will be defined in
Activity 3 once a set of potential OTS candidates have been identified. Using the
established search criteria, identify possible candidates from sources both internal and
external to the project or organization, and screen them by applying the minimum
thresholds of the search criteria to each candidate. The most promising candidates are
selected for a full evaluation during Activity 4.

D.3.           Define Evaluation Criteria

This activity produces the detailed criteria necessary to support a repeatable and
systematic evaluation. The definition of criteria refines, formalizes, and expands on the
search criteria and addresses functional, architectural, management, strategic,
performance, and financial characteristics of the candidates. Weights are established for
all of the evaluation criteria with respect to each project’s importance. The evaluation
selection is based on criteria priority. The references [SPC-COTS] and [SPC-CEP]
provide details of ways to build the decision model and represent (e.g., through decision
matrices) the evaluation information for comparison across candidates.

D.4.           Evaluate Alternatives

This activity is conducted to assess how well the alternatives meet the defined criteria.
Evaluation scenarios such as demonstrated use to satisfy one or more functional
requirements specific to the intended application are developed to evaluate the
alternatives. Results are documented for analysis and form an important part of the
“Case” described in Section 3. Alternative candidates may not be evaluated in the same


                                       Page 46 of 52
OTS Software                                                          SQAS30.01.00 - 2005


manner since evaluation results are based on the available data. The evaluation criteria
may also include a weighting scheme that establishes a level of importance for each
criterion. The available data may be from hands-on experience, witnessing supplier
demonstrations, observing a user, and reading third-party literature or supplier’s
literature. Each type of data is given a rating value. Credibility – rating the confidence in
what the evaluator knows about an alternative – may then incorporated in a simple
weighted average.

D.5.           Analyze Evaluation Results

The evaluation produces data on how well each alternative candidate meets the defined
criteria. The analysis consists of activities to compare and contrast rankings of
alternatives based on the priorities. Sensitivity analysis, using a decision-support method,
is performed to determine the impact of criteria or groupings of criteria on the ranking of
alternatives. More confident decisions may be made when the impact of the criteria is
analyzed. More specifics on a possible Design Model is described in [SPC-COTS] and
[SPC-CEP].




                                        Page 47 of 52
OTS Software                                                       SQAS30.01.00 - 2005



APPENDIX E.           Software NWC Site Transfer Methods
E.1.          Background

At the July 2002 Quality Managers Meeting, the Quality Managers received a proposal
for a software transfer method and mechanism and after due consideration, passed this
proposal to the Software Quality Assurance Subcommittee (SQAS) for action. The
action was to provide guidance to the Quality Managers on how software should be
transferred from and between the various agencies and contractors working within the
Nuclear Weapons Complex (NWC). The proposal resulted from problems identified in
implementing software received for use at Y-12 that was Design Agency/Government
Agency (DA/GA) developed. Since the transfer of software between sites is a special
case of the acquisition of Off-The-Shelf (OTS) software, the resolution of this proposal
has been subsumed within this report considering the selection, evaluation, transfer, and
management of OTS software. The concerns in this appendix were derived specific to
addressing the issue of transferring existing software from one NWC site to one or more
other NWC sites, and what considerations were important to resolve in order to facilitate
the transfer and effective operational use of the software.

There is a Department of Energy/National Nuclear Security Administration
(DOE/NNSA) mechanism from the Development and Production (D&P) Manual that
outlines an existing method for “official” transfer of software and its validation and
verification (V&V) related information. This mechanism is called an “Integrated
Contractor Order,” or “ICO”. In addition, there are two organizations currently charged
with managing the distribution of software for common utilization across DOE/NNSA.

The Office of Scientific and Technical Information (OSTI), established by DOE Order
241.1A [DOE O 241.1A], is authorized to distribute information, documents, and
software to other Federal Agencies and to the international community. Additional
information concerning OSTI may be found at URL http://www.osti.gov. The Radiation
Safety Information Computational Center (RSICC) is a Specialized Information Analysis
Center authorized to collect, analyze, maintain, and distribute computer software and data
sets in the areas of radiation transport and safety. Additional information concerning
RSICC may be found at URL http://www-rsicc.ornl.gov. Guidelines for operation of
these organizations are contained in DOE Guidance 241.1-1A (Guide, 11/23/2001, SC),
“Guide to the Management of Scientific and Technical Information.”

At the present time, there is not a consistent software methodology, engineering process,
or Software Quality Assurance (SQA) process in place among the NWC sites. Similarly,
there is not a standard for the software transfer process or an overall umbrella under
which DOE Software Quality Assurance (SQA) is managed and controlled, other than for
software that is qualified in accordance with Department of Energy (DOE) Quality
Assurance Inspection Procedure (QAIP) and the Technical Business Practices (TBPs)
procedures. SQA covers the entire life cycle of software; whereas, the transfer process
includes receipt, testing, and implementation of software across the NWC. Therefore,



                                       Page 48 of 52
OTS Software                                                               SQAS30.01.00 - 2005


standards are needed for Quality and Software Engineering as related to software
development, implementation, and transfer.

E.2.           Concern

The Quality Managers received a proposal for a software transfer method and mechanism
at their July 2002 meeting. After consideration the proposal was passed to the SQAS
with the action to provide guidance to the Quality Managers on software transfer. The
proposal resulted from the problems encountered at Y-12 in implementing software
received for use that was DA/GA developed. In many cases there was not a
“qualification” process in place to accept and utilize this software. The use of software at
Y-12, whether it is internally developed, externally developed, DA/GA developed, or is
Commercial Off-The-Shelf (COTS) requires compliance to Y80-101INS, “Software
Management Instruction.” In particular, this compliance includes the test results
providing evidence that the requirements were functionally implemented and the
acceptance testing provided evidence that the expected results were received with known
inputs.

E.3.           Analysis

There are three basic methods used to transfer software between NWC sites. These
existing methods are briefly outlined in the following paragraphs. A simplified view of
the software transfer approach is illustrated in Figure E-1.

                               Software Transfer

                                     Standards
            Site #1                                                     Site #2
            Recipient          Name-G                                  Generator
                               *Generator
                               *Recipient                             *Software Contact
                               ID-G                                   *Technical Contact
                               Version # & Date-G
                               Function-G
                               Transfer Date-R/G
                               Documentation
                               Verification Evidence - G (Attested)
                               Validation Evidence - G/R
                               Problem Resolution - G/R                Documentation
                               QA Program-G                            User’s Manual
                               Environment-G                           Functions
                                                                       Special Functions
                                                                        (security)
                                                                       V&V



                   Figure E-1. NWC Software Transfer Approach




                                        Page 49 of 52
OTS Software                                                       SQAS30.01.00 - 2005


E.3.1.        Integrated Contractor Order
Existing methods for "official" transfer of software (and its V&V related information)
from one site to another are outlined in two cases. Case 1: An Integrated Contractor
Order (ICO) generally can be used to transfer software that is qualified in accordance
with Department of Energy (DOE) Quality Assurance Inspection Procedure (QAIP) and
the Technical Business Practices (TBPs) procedures. Case 2: An ICO can generally be
used to transfer software of any pedigree from one site to another site. The ICO can
specify what software artifacts (documentation, source code, and so forth) is to be
transferred, based on a negotiated agreement between the sites. In both cases, the ICO
serves as a “contractual” mechanism where deliverable product, support training, and
cost requirements, if any, can be specified.
E.3.2.        Office of Scientific and Technical Information
The Office of Scientific and Technical Information (OSTI) leads DOE government
initiatives for disseminating R&D information. Located within the Office of Advanced
Scientific Computing Research in the DOE Office of Science, it is responsible for leading
the Department's Technical Information Program and for providing direction and
coordination for the dissemination of Scientific and Technical Information (STI)
including software and related documents resulting from DOE research and development
programs. Guidance for the submittal of STI to OSTI is provided through the document
clearance process at the NWC sites. At Y-12 the Technical Information Office
determines whether a document contains STI that must be submitted to the OSTI.

DOE Order 241.1A [DOE O 241.1A] establishes requirements and responsibilities to
ensure that STI is identified, processed, disseminated, and preserved in a manner that (a)
enables the scientific community and the public to locate and use the unclassified and
unlimited STI resulting from DOE's research and related endeavors and (b) ensures
access to classified and sensitive unclassified STI is protected according to legal or
Departmental requirements. DOE Guide 241.1-1 [DOE G 241.1-1] provides non-
mandatory guidelines for implementing the objectives, requirements, and responsibilities
of DOE STI Management.
E.3.3.         Radiation Safety Information Computational Center
The Radiation Safety Information Computational Center (RSICC) is a Specialized
Information Analysis Center authorized to collect, analyze, maintain, and distribute
computer software and data sets in the areas of radiation transport and safety. RSICC
follows the policy and procedure directives defined in [DOE O 241.1A] and [DOE G
241.1-1].

RSICC is a computer code center located at the Oak Ridge National Laboratory within
the Nuclear Science and Technology Division of the Energy and Engineering Sciences
directorate. RSICC staff test each software code before its release, documents and
resolves difficulties or problem areas, and assists users with their questions. RSICC
provides scientific and operational consulting for codes and data that RSICC distributes.




                                       Page 50 of 52
OTS Software                                                          SQAS30.01.00 - 2005


E.3.4.         Proposed Standards
Generally ICO, OSTI, and RSICC methods are not available to users at all NWC sites,
therefore a need exists for the development and use of consistent Software Quality
standards at all the NWC sites. Software suppliers, i.e. developers, should provide
documentation to attest to the verification of the software at their site and the recipient of
software should perform validation through integration and acceptance testing within the
target application environment. To do this several elements are needed, such as the use
of a “common” software engineering methodology, establishment of the software
pedigree, established minimum requirements for software documentation, established
proper prerequisites for software qualification, and proper registration of any such OTS
software used at the recipient’s site. The following paragraphs further describe these
elements.
E.3.4.1.       Software Engineering Methodology Standard
At the present time software is being developed by several different sites within the NWC
with no specific software methodology being used, much less a methodology that
represents a “common” approach. This could potentially be a problem when software is
transferred between sites within the NWC, particularly concerning whether or not the
software meets its established requirements, has been properly tested, and is under
configuration control with consistent versions used at all sites. Whether or not a specific
methodology is used, the most important aspect of the development process is to ensure
that the software meets its requirements, has been properly tested and documented, and
the proper version is being used.
E.3.4.2.       Software Pedigree
Software pedigree is the documentation providing or pointing to evidence that software
meets or exceeds established requirements for qualified use in the developer’s site.
Additional testing and documentation may be required to meet the recipient’s site
application qualifications. Requirements do not exist which serve to establish standards
for a software pedigree, except in the case where the software is qualified as a Mark
Quality product through the Technical Business Practice system.

A pedigree allows the recipient to accept the software and, if the recipient’s environment
is the same, conduct simple verification activities that confirm the software performs as
desired in the recipient’s environment. Without the pedigree the recipient must perform
additional validation activity through more extensive Integration and Acceptance Testing
that typically incurs additional costs prior to placing the software into operational use.
E.3.4.3.       Software Documentation
Software documentation, as a minimum, provides the requirements, function, system
design, source and executable code, user instructions, test plan, and test results.
Verification documentation provides evidence of flow down of requirements through
functional design to programmed code and that implemented code addresses the
identified requirements. Validation documentation provides evidence that the software
when executed in a specified application environment performs the functions established,


                                        Page 51 of 52
OTS Software                                                         SQAS30.01.00 - 2005


implements the requirements as specified, and provides expected results when operating
with known standard inputs.
E.3.4.4.       Registration of Software
Each NWC site should have the responsibility to inventory all software used at their site,
including any OTS software. Establishing the type and use of all software developed,
procured, and used at a site as a part of this inventory would allow other sites to identify
software which may be of benefit to them and considered for reuse. Reuse of software
would greatly reduce the investment in repetitive software development at each site.
Software inventory details and "use" categorization should be established and
consistently implemented at all sites.
E.3.5.         Conclusions
Each site must ensure that site-specific and DOE/NNSA requirements are met. For
qualified software the supplier must provide the available evidence of qualification that
the software being transferred meets the supplier’s site SQA program. The recipient must
then review the supplied evidence and determine if there is a need for additional evidence
to meet the recipient’s site SQA program. This exchange of evidence is the heart of the
software transfer process.

It may be appropriate to ship software that has little if any SQA evidence or other such
qualification information. The receiving site must then make the determination of
whether such software may be used. It may be that such software is just for research or
investigation. It may be that such software is to be used for further development into a
product capable of being used to support weapon and/or weapon-related activities. It
may be that such software is intended to be used "as-is" to support weapon and/or
weapon-related activities. We would hope this last case is rejected by the receiving site
(and actually will not be considered for shipping by the source site, since a failure would
affect both sites).

This report provides some concepts that may be useful in supporting the transfer of
software from one NWC site to another NWC site. This report addresses the concerns of
software documentation, pedigree, and registration – somewhat independent of specific
software methodologies that might be used at a site. The more extensive description of
potential SQA evidence as well as the processes for OTS Software selection, evaluation,
acquisition, and management are presented in the body of this report, in particular,
Sections 3 and 4.




                                        Page 52 of 52

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:8
posted:8/18/2011
language:English
pages:52