Document Sample
paper1 Powered By Docstoc
					           Subsystem and System Integration Testing in Software
                Maintenance: A Case Study of the CF-188
        Alan W. Mack                                                                      +1 613 541 6000 x6031
   CrossKeys Systems Corp.                         Terry Shepard                    
      CrossKeys Center                       Department of Electrical and                     Margaret Lamb
     350 Terry Fox Drive                       Computer Engineering                     Department of Computing and
  Kanata, Ontario, K2K 2W5,                   Royal Military College of                     Information Science
           Canada                                     Canada                                 Queen‟s University
       +1 613 599 2300                        PO Box 17000, Stn Forces                   Kingston, Ontario, Canada                       Kingston, ON, K7K 3B4,                          +1 613 545 6050

ABSTRACT                                    Armed Forces is faced with the              fail to perform as required [8].
This paper presents the maintenance         continuing and expensive task of            Hard-real-time software is software
testing environment used for the            maintaining large amounts of                which must not only be logically
CF-188 Operational Flight Program           software.                                   correct, but must also satisfy
(OFP) as an example of the problems                                                     stringent timing constraints [7].
involved with the verification of such      This paper focuses on the CF-188            Verification that the software meets
software.     Among       the     special   Operational Flight Program (OFP) to         the functional, safety and timing
challenges in this testing environment      illustrate in some detail the problems      requirements is one of the major cost
are hardware interactions, and safety       associated with the verification of         drivers for the maintenance of
and timing issues. The difficulty of        modifications to such complex               weapon system software. To make
predicting what will be changed over        long-lived embedded software. It is         matters      still   more   complex,
the lifetime of the aircraft (at least 20   based on work that was completed            verification       during    software
years) adds to the challenge. As well,      over a year ago [9][10]. A brief            maintenance is often more difficult
verification      during        software    update on the current status is             than verification during software
maintenance differs from verification       provided in the Conclusion. The             development, for a number of reasons
during software development. Much           CF- 188 OFP is broken up into               which will be explored later in the
of the paper addresses issues               several subsystems, each of which           paper. The main issue is the risk of
involved in increasing the automation       runs on its own processor (or               introducing problems in parts of the
of testing of such a complex system         processors) and interacts with              software that have not been modified,
                                            different items of hardware on the          which could impact the operational
Keywords                                    aircraft, such as radar, weapons, pilot     effectiveness of the entire fleet of
system testing, regression testing,         input switches, and flight control          aircraft.
subsystem       testing,  real-time,        surfaces. The subsystems are
safety-critical                             connected by a series of buses, which       THE CF-188 SOFTWARE
                                            are multiplexed, and are therefore          MAINTENANCE PROCESS
INTRODUCTION                                referred to as MUX buses. The               The CF-188 software maintenance
With the increase in modern systems         CF-188 OFP testing environment              process consists of six phases:
such as the Tactical Command,               must provide suitable interfaces for        changes are proposed, validated,
Control      and    Communications          testing the integrated system and each      evaluated, approved, implemented,
System, the New Patrol Frigate, the         subsystem, providing either actual          and verified.
CP-140 Aurora long range patrol             hardware or a suitable emulator for
aircraft, the CF-188 Hornet fighter         each hardware device the system or          In the proposal phase, system and/or
aircraft, and the planned Sea King          subsystem interacts with. The CF-188        subsystem problems or changes to
                                                           OFP      presents      an    system level or subsystem level
                                                           additional     challenge     requirements are identified and
            (This space left blank for                     because        it       is   recorded on Software Trouble
             ACM copyright notice)                                    safety-critical   Reports (STRs).
                                                           hard-real-time software.
                                                           Safety-critical software     Validation activities applied to each
                                                           is software which could      STR depend on whether the STR is
helicopter replacement, the Canadian        result in injury or damage should it        based on a problem or a change in

requirements.      In the case of a                                                    echoes the intentions of the
problem report, a top-down testing         The approval phase of the CF-188            immediately preceding step” [3].
approach is followed in order to           software      maintenance     process       These definitions do not address the
duplicate and localize the source of       identifies SCRs which are to be             unique requirements of verification
the reported problem. Typically, test      implemented in future builds of a           during software maintenance in that
cases based on the operational profile     particular subsystem and allocates          they fail to recognize that the
of the system at the instant that the      resources to complete the next build.       maintenance phase starts at the end of
problem was first detected are                                                         the development phase, with an
executed on the system test rig in an      The implementation phase of the             existing system. Verification during
attempt to localize the problem to a       process includes all of those activities    software maintenance must not only
particular subsystem. Once the             that are usually associated with            answer the question “are we building
problem has been localized to a            software development, with some             the product right?”, it must also deal
subsystem, testing is carried out on       differences. The first difference is        with whether the product was built
the applicable subsystem test rig and      that modifications are designed and         right, and what has changed since. It
software development system in an          developed for only those units              must also deal with the fact that a
attempt to localize the problem            affected by the SCR. The second             copy of the software is in use on each
within the subsystem. In the case of       difference is the approach taken to         aircraft in the fleet, and modifications
a change in requirements, the process      incrementally build the new software        can impact that use in a negative
is simpler: the STR is assessed on the     baseline. First, the modifications          way.
basis of its operational validity.         made to incorporate each SCR are
                                           separately integrated into the baseline     Verification      during       software
Once the STR has been validated, it        software in accordance with a               maintenance,      especially     in    a
is converted to a Software Change          bottom-up approach. Second, the             subsystem/system             integration
Request (SCR), which is then               new software baseline is created by         environment such as that found in the
subjected      to     an    engineering    integrating the modifications of            case of the CF-188, is often more
evaluation to provide an estimate of       individual SCRs incrementally. As           difficult than during software
technical, operational, cost and           well, the implementation step is            development. The maintainers of the
schedule        impacts.      Technical    contracted out, so the details of how       software are usually not the
evaluation activities for those SCRs       it is conducted are usually hidden.         developers. The maintainers‟ only
which are problem based typically          The subsequent verification effort          visibility into the design decisions
follow a bottom-up approach starting       emphasizes       the      testing     of    made during original development is
from the problem source. They              requirements affected by the SCRs,          either via the requirements and
include structural testing at the unit     based on maximizing code coverage           design specifications or via the
level to determine which units require     of the modified software and on a           source code. Often, as is the case
modification to implement the              suitable level of regression testing.       with the CF-188 OFPs, the
change, and a combination of static        [11].                                       requirements          and         design
and dynamic testing and analysis at                                                    specifications are inadequate for
the unit, subsystem, and system            THE CF-188 OFP VERIFICATION                 verification purposes. In the case of
levels to identify potential integration   PROCESS                                     the CF-188, they do not decompose
impacts of the change. Technical           Verification must ensure that               the software in a clearly defined
evaluations for those SCRs which are       modifications      are    implemented       manner and are incomplete [11]. For
requirements based typically follow a      correctly, in accordance with               example, the timing constraint
top-down approach starting at the          modified specifications, and that the       specifications for the CF-188 Mission
level at which the requirement is          modifications      do not adversely         Computer       OFP       are     almost
changed.          For example, the         affect unmodified software. In the          non-existent. It appears that the
evaluation of a proposed change at         case     of    the     CF-188     OFP       designers‟ concern about the timing
the system level, such as the addition     maintenance, verification emphasizes        behaviour was minimal due to the
of a new weapon, commences with            test planning and test execution at the     fact that they could design the
system level testing on the system         subsystem and system levels [4].            original version of the OFP to fit the
test rig using a simulation of changed                                                 constraints imposed by the processors
subsystems to examine integration          Verification During Software                and memory [1]. For maintainers, this
issues at the system interfaces. It also   Maintenance                                 lack of timing specifications means
includes subsystem level testing           Verification is defined as the              that there is no way to predict
using stubs and/or drivers on the          “process of evaluating a system or          accurately if modifications will result
appropriate subsystem test rigs to         component to determine whether the          in timing problems.
examine integration issues at the          products of a given development
subsystem interfaces. Finally, the         phase satisfy the conditions imposed        The CF-188 OFP specifications are
appropriate        software      support   at the start of that phase” [6] or as the   written in a combination of natural
environment is used to assess              activity that “ensures that each step       language and flow-charts, for which
integration issues.                        of the development process correctly        the verification techniques are

limited.      The       benefits     of    factors.     First, the majority of        (PMV - referring to the aircraft itself)
re-engineering the specifications          software maintenance is now                flight testing, and ground based
using other notations are outweighed       contracted out. Second, the various        testing using the Integrated Test
by several factors, including the          subsystems have not been formally          Facility described later.
difficulty of contracting for software,    decomposed and documented. Third,
the required level of effort, and the      the different subsystems were              Test cases for both PMV flight tests
cost of training. Using formal             designed          using    different       and ITF system and subsystem
notations can make this situation          methodologies and are written in           testing (except as noted below in the
even worse, and there is dubious           different programming languages.           detailed descriptions of subsystem
benefit in attempting to prove                                                        testing)     include     design-based,
correctness, due to the questionable       The primary impact of the contractual      equivalence partitioned and boundary
validity of proofs in an environment       arrangements is that the contract          value functional tests. They also
as complex as the CF-188 [5]. As           imposes formal testing only at the         include regression tests, to ensure
another example, experience with the       subsystem and system levels. As a          that modifications have not impacted
SCR Methodology on the A-7 and             result, subsystem and system tests         the     implementation     of     those
Darlington      Nuclear     Generating     follow      contractually   specified      requirements which were not
System re-engineering exercises            procedures, while lower levels of          modified.      Flight test cases are
indicates that the level of effort         testing follow the contractor‟s            described in a set of flight test cards
required to generate and prove the         internal procedures.                       which provide the pilot with
formal specification was very high                                                    directions on how to execute the tests
and, while there was an improvement        Because the various subsystems have        manually. These flight test cards are
in the precision and readability of the    not been formally decomposed and           developed by hand, based on written
specifications, of questionable value      documented, unit specifications are        requirements. ITF system and
in terms of cost and product quality       either poor or non-existent, so testing    subsystem test case descriptions
[2]. Tool support for formal               of modified units in isolation is          provide the test engineer with explicit
specification notations improves this      difficult. As well, new and modified       directions on how to set up the test
situation, but tools are generally still   units are integrated into an existing      environment and how to execute the
at a research stage and not yet ready      baseline. These two factors make it        tests manually. As is the case for
for large-scale production use.            attractive to use integration testing to   flight testing, these test cases are
Furthermore,       many      of     the    achieve unit testing, provided that the    developed by hand based on
modifications to the OFP involve the       integration tests provide sufficient       published specifications. In both
integration of off-the-shelf (OTS)         unit test coverage and observability.      cases, the lack of a complete set of
software developed by the United                                                      system requirements in a testable
States Navy. The specifications for        The fact that the software within the      format makes the determination of
this OTS software would likely not         various     subsystems     has    been     test cases very difficult [11], but
be compatible with any one new             designed          using       different    improved requirements are gradually
specification method chosen.               methodologies and programmed in            being developed as changes are
                                           different languages means that             made.
There are advantages to testing in the     different       software       support
maintenance phase. One of the              environments are required for each
principal ones is that it can be           subsystem.         In general, the         Prime Mission Vehicle Flight
worthwhile      making      significant    programming support environment            Testing
investments      in     the     testing    that was used to develop the software      Final software testing is flight testing
environment. That is certainly the         for the subsystem was delivered with       on the actual aircraft, or PMV, using
case for the CF-188. Even so, funds        the subsystem. The result is that          dynamic,     functional       black-box
are limited, so one of the issues          unit, integration, and subsystem level     testing.
addressed in this paper is how to          testing uses testing tools which are
assess which areas of improvement in       unique to each subsystem.                  The primary advantage of flight
the testing environment are most                                                      testing is that the software is
worthy of investment. The paper does       CF-188 SOFTWARE TESTING TOOLS              subjected to the actual operating
not attempt to provide detailed            The testing tools employed in the          environment,      including     actual
cost-benefit analyses, but does            CF-188      software      maintenance      real-world inputs and a completely
present the factors and issues that are    environment include both static            integrated avionics suite.        The
important in the decision making           analysis and dynamic analysis tools.       primary disadvantages are safety
process.                                   In this section, some of the testing       issues, the high cost of flying hours,
                                           tools used to test the various CF-188      the limited controllability and
Factors Affecting CF-188 Testing           subsystems and the CF-188 system           observability of subsystems, and very
The     employment     of     testing      are described and assessed. The            limited      on-board      monitoring
techniques during the maintenance of       major distinction for testing purposes     capabilities, which do not permit
the CF-188 is influenced by several        is between Prime Mission Vehicle           thorough logging of test results.

Monitoring of flight tests primarily      largely not documented in the             (EWEG) is used to inject simulated
involves observations by the pilot,       original system, although progress        threat emitter signals into the Radar
though some aircraft instrumentation,     has been made in this direction over      Warning Receiver antennae and a
such as the Heads-Up-Display              the past several years.                   Weapons Simulator is used to inject
(HUD) camera and the Maintenance                                                    weapon signals into the Stores
Signal Data Recording System              Integrated Test Facility (ITF)            Management System.
(MSDRS), provides a limited               The Integrated Test Facility (ITF) is a
recording capability. Also, the           dynamic testing environment which         The ITF incorporates several
aircraft can be specially instrumented    is primarily used for system testing      capabilities       which       permit
with high-speed cameras, video            and subsystem testing of the Mission      non-intrusive      monitoring     and
cameras, strain gauges, and special       Computer (MC) subsystem. Test             recording of up to 100 input and
MUX data recording equipment.             stations for other subsystems can be      output       simulation     variables,
                                          switched in and out, depending on the     environmental variables, discrete
The PMV does not             have   an    mode of operation of the ITF. A           avionics variables and MUX bus
automated test capability.                functional duplication is provided of     words and variables as well as
                                          those CF-18 subsystems required to        non-intrusive monitoring of several
The CF-188 System Model                   perform testing of the MC subsystem,      discrete signals.       All recorded
This section contains a partial list of   along with a simulation of the            variable values are time-stamped
the CF-188 avionics subsystems that       real-world operational environment.       with a universal reference time. This
will help in understanding following      Certain avionics subsystems can be        can later be used for timing analysis.
sections. The top level CF-188            physically present as part of the ITF.    Several actual CF-188 avionics
Avionics System is decomposed into        Dynamic      and     static   software    displays and indicators permit the
a number of subsystems, including:        simulators are available to represent     monitoring of data presentations
           Mission Computer               the functionality of most subsystems.     intended for the pilot.
          (MC)                            Dynamic software simulators provide
 Radar Data Processor (RDP)               data in real-time while static            There is currently no acceptable tool
 Radar Signal Processor (RSP)             simulators simply respond to queries      for monitoring program execution
 Stores Management Set (SMS)              with an acknowledgment.            The    within the MC subsystem.
 Electronic Warfare (EW)                  “real-world”       environment       is
 Integrated Communications,               simulated using dynamic simulation        The monitoring capabilities of the
                                          models.                                   ITF have recently been upgraded to
          Navigation and
                                                                                    provide     a    more     user-friendly
          Identification (CNI)
                                          Avionics subsystems which can be          graphical user interface which
 Multipurpose Display Group (MDG)
                                          physically incorporated into the ITF      permits the tester to select various
 Maintenance Signal Data Recording
                                          include    actual    radar    receiver    simulation variables, MUX words,
          System (MSDRS)                  antennae, several devices for pilot       specified variables within the MUX
Each subsystem has its own OFP; the       input, and several displays. Avionics     words, and discrete signal variables
aggregate of all OFPs is also referred    subsystems which are only modeled         for analysis during the test execution.
to as the OFP. Most of these names        using dynamic software simulators         The values of these variables may be
are relatively self explanatory, but a    include communication radios and          displayed in alphanumeric format or
few need some explanation: The            the altimeter. Several other avionics     plotted on a graph against either the
MDG runs multiple displays for the        subsystems are only represented by        value of other variable values or the
pilot, and the SMS manages the            static software simulators which          value of the time-stamps generated
armament. Other subsystems relevant       provide only a simple Avionics MUX        by the universal time code generator.
to this paper include the Up-Front        bus interface, or which have limited      The capability to plot variable values
Control Panel (UFCP), which               dynamic functionality. These include      against each other permits some
provides the primary cockpit control      the Ground Proximity Warning              analysis of cause-effect relationships
of the CNI subsystems, the Ground         System,      the    Flight    Control     while the capability to plot variable
Proximity Warning System, which is        Computers, and the Maintenance            values against the universal time
an independent subsystem except for       Signal Data Recorder Set (MSDRS).         permits some analysis of timing
use of sensors and displays, and the                                                issues.
Flight Control Computers, which are       “Real-world” inputs to the ITF are
responsible for the fly-by-wire           simulated. Dynamic earth and              The ITF and its test stations have a
subsystem.                                atmospheric models are used to            limited automated test capability.
                                          simulate the “real-world” flight          Script files can be written in a special
Current software practice would           environment. A Radar Target               command language and automatically
document the further decomposition        Generator (RTG) is used to inject         executed. This capability is limited in
of each subsystem into smaller            simulated radar targets into the radar    that the command language is
modules. Unfortunately, the internal      when the actual radar is used. An         sequential and does not support basic
structure of the subsystems was           Electronic Warfare Emitter Generator      programming language features, such

as loops, required for more complete       is used to select regression tests by      updated under UCP. This tool is
automatic generation of test cases.        identifying where the impact of            used primarily to verify symbols and
Furthermore, there is no capability to     modified code is likely to be.             the positioning of symbols on the
compare      actual    test   outputs                                                 displays.
automatically with expected results.       The dynamic analysis capabilities
                                           within this support system includes a      Test cases for MC and MDG
The primary advantages of testing on       Function Simulation Program (FSP)          unit-level tests include structural path
the ITF versus the PMV are the             and a Multipurpose Display Group           coverage tests to verify the control
elimination of the flight safety issues,   Simulator (MDGSIM).           The FSP      flow within modified units.
and the lower cost. For example, test      consists of an emulator and a User
cases on the ITF can include stress        Control Program (UCP).             The     Data Reduction and Analysis Station
tests which exceed the safe limits of      emulator provides bit-by-bit results       The Data Reduction and Analysis
the flight envelope of the aircraft.       identical to those which would be          Station (DRAS) was intended to
Also, testing on the ITF provides a        obtained if the program were               provide a post-test analysis capability
much better monitoring and control         executed on AYK-14 computer that           of the MUX and discrete digital data
capability for the test cases. The         the MC OFP normally runs on. The           recorded during the execution of tests
primary disadvantage of testing on         UCP controls the FSP execution,            on the ITF and the Radar Software
the ITF is that the software is not        reads data in, schedules module            Test Station (RSTS). The DRAS
subjected to “real-world” stimuli. The     calculations, and writes test data on      also includes a Video Cassette
simulated environmental inputs and         files for subsequent printing.        It   Recorder (VCR) capability to permit
nearly complete integrated avionics        provides the user with the capability      the playback of the video recordings
suite on the ITF minimize this             to set values within the MC memory,        of two repeater Digital Display
disadvantage.                              to define the rate at which the MC         Indicators (DDIs) connected to the
                                           modules are to be executed, to             ITF. The event history of selected
Test Support Hardware and                  provide the logic to schedule the          recorded variables can be displayed,
Software in the ITF                        input of events when the elapsed time      synchronized with the time-stamps
                                           is reached and to define schedules to      inserted at the time of recording.
Mission Computer Support System            impose order on the individual MC          Additionally, it is possible to use the
The Mission Computer Support               modules. It also provides various          time-stamps to manually synchronize
System (MCSS) is hosted on an IBM          diagnostic capabilities to verify the      the video recordings with other data,
system and provides both static and        subsystem, including the capability to     although the limitations of the
dynamic analysis tools for the             compare computation results to             standard VCR make this difficult.
Mission Computer (MC) and the              user-defined expected results, to          The DRAS also can compare test
Multipurpose Display Group (MDG)           dump memory contents, to inspect an        results gathered during different
subsystems. These tools are used in        absolute memory address, to trace          executions of a test case, search for
combination to compile MC and              execution, and to observe changes in       variable values which are outside a
MDG source code, to debug MC               parameter values. The UCP also             prescribed upper and lower bound,
software, and for unit and integration     includes a pathfind capability to aid      and calculate the accuracy of some
testing of MC units.                       in the conduct of structural testing by    air-to-ground weapons.
                                           detecting and monitoring all paths
The static analysis capabilities within    taken by a set of test cases.              Unfortunately, this station is not
the MCSS include a compiler and a                                                     currently used because of its very
Data Base tool. The compiler has           The FSP is used for both unit and          poor user interface. Even if the
static verification capabilities which     integration testing.       Integration     interface is improved, it is very
are common to many other                   testing is conducted by executing the      difficult to correlate the program
compilers. For example, it checks          integrated units in a specified            execution traces from the special MC
the    usage      of   symbols      for    sequence for a specified number of         and radar test tools and the In Circuit
multi-defined symbols and symbols          iterations, forcing the units through      Emulator (ICE) with the data
not in the symbol table. The Mission       specified paths, and permitting the        playback on the DRAS This means
Computer Data Base tool is an Oracle       examination of full or partial results     that the analysis of much test data is
database application that provides         of the test.                               restricted to the real-time analysis by
information on MC internal and                                                        the tester of the monitored variables
external parameters, multiplex bus         The MDGSIM tool is used for unit           and the various CF-188 displays.
word list, and MC routines. It is also     testing of the MDG code.          It       The requirement to repeat tests to
used to provide cross referencing          provides a simulated graphical             verify that different events occurred
between these elements and to obtain       display of two displays used in the        correctly is costly, and there is no
the dynamic structure (calling tree)       aircraft. Unit testing involves the        verification of real-time events that
which represents the calling sequence      downloading of the background and          occur at rates higher than can be
of routines within the MC software.        first pass cyclic information after        analysed by a human tester.
This tool is important and valuable. It    which only cyclic parameters are

Test Stations Included in the ITF           associated    with.   They     have        subsystems are physically present on
                                            nonetheless been important, because        the CSCTS, while dynamic and static
A partial description of the test           they have been needed for testing          software simulators are used to
stations in the ITF follows.                modifications to other parts of the        represent the functionality of others.
                                            CF-188 software.
Stores Management Set Test Station                                                     The RSTS is the only one of
The Stores Management Set Test              The Radar Software Test Station            theseparts of the ITF that has unique
Station (SMSTS) is a dynamic testing        (RSTS)/Electronic               Warfare    non-intrusive       monitoring      and
tool for the SMS subsystem and for          Integrated Test Bench (EWITB) is a         recording capabilities can be applied
initial testing of the integration of the   dynamic testing tool which is              to virtually all variables.          All
SMS subsystem with the MC                   primarily used for Radar and EW            recorded      variable   values      are
subsystem. It includes a Stores             subsystem testing, though it is also       time-stamped with a universal
Management Processor (SMP), the             used for subsystem testing of the          reference time generated by a global
encoders/decoders which are used to         Mission Computer (MC). Its design          time code generator for timing
program the weapons on each                 and capabilities are virtually identical   analysis. Monitoring of program
weapon station, a weapon simulator          to those of the ITF. Depending on the      execution path coverage within the
which is used to replicate the signals      mode of the ITF, it runs as part of the    Radar Data Processor is possible
generated by the weapons, and a             ITF, or the ITF can run without it, in     through the use of the special
semi-dynamic simulation of the              standalone mode. The primary               Programming Support Environment
mission computers.                          difference between them is that the        (PSE) tool; however, this capability
                                            actual radar and the actual Electronic     is severely limited by the fact that the
The SMSTS incorporates several              Warfare (EW) subsystems are                PSE is only capable of recording data
capabilities       which        permit      integrated into RSTS and EWITB             at a one millisecond rate.
non-intrusive      monitoring      and      respectively (and hence into the ITF
recording of input and output               when it is operating in the mode in        Test cases for the Radar and CSC
simulation variables, environmental         which it is integrated with the RSTS       subsystems have only started to be
variables, discrete avionics variables      or EWITB). Other differences are           developed very recently, since the
and MUX bus words. In addition, a           that there is no actual Heads-Up           first applicable SCRs have been
test point panel is provided to permit      Display (HUD), and programmable            approved for implementation in the
attachment of multimeters, logic            touch panels are used in place of          past year. Test cases for the EW
analysers and a strip chart recorder.       avionics switches.                         subsystem level tests have been
All recorded variable values are                                                       developed recently, with the EWITB
time-stamped with a universal               The Radar Support System (RSS)             portion of the test station having been
reference time. Monitoring of               provides both static and dynamic           recently released for use. The lack of
program execution within the Stores         analysis tools for the Radar Data          experience with test cases for these
Management Processor (SMP) is               Processor (RDP) subsystem. The             subsystems makes assessment of
possible through the use of an Intel        static analysis tools include a            their potential for automation
8080 ICE which provides the                 propriety Hughes Assembly Program          difficult.
capability to insert breakpoints and        (HAP). The dynamic analysis
examine memory and register                 capabilities include a Programming         AUTOMATION OF TESTING
contents.                                   Support Environment (PSE) which
                                            provides capabilities comparable to        Advantages of Automation
SMS subsystem test cases describe           those of an In-Circuit Emulator. It        System and subsystem testing in a
subsystem stimuli and expected              can be used to control the execution       software maintenance environment is
responses, but not subsystem                of the program in the RDP, to insert       needed      both    to    verify   the
performance requirements.                   breakpoints, to examine memory and         functionality of modifications, and to
                                            register contents, and to write values     ensure that they have not impacted
Radar Software Test                         to memory and registers. There is          the     implementation     of    those
Station/Electronic Warfare                  currently no support for the Radar         requirements which were not
Integrated Test Bench; Radar                Signal Processor (RSP) subsystem.          modified. Automating these tests
Support Environment;                                                                   would have many advantages:
Communications System Controller            The       Communication        System         It would serve to improve the
Test Station                                Controller Test Station (CSCTS) is a          repeatability of the tests by
These parts of the ITF are grouped          dynamic testing tool which is used            minimizing the possibility of
here because they have similar              for testing and debugging of the CSC          human errors during test input
characteristics, including the fact that    subsystem, and for initial integration        injection. This is essential for
test cases have only been developed         testing with the MC subsystem. It             regression testing, which involves
specifically for them very recently,        also provides partial coverage of the         repeating past tests to ensure that
because changes are just now starting       navigation      and      identification       the modifications to the software
to be made to the subsystems they are       subsystems.      Certain      avionics        did not introduce faults into

   previously operational software. It                                              aiming of the radar antenna, and the
   would also be beneficial for the       Automating ITF Test Initialization        selection of radar targets, to name
   tests    used     to    verify   the   or Reset                                  only a few.      The ITF provides
   functionality of modifications         The ITF must be re-initialized            software overrides for all significant
   during a particular integration        between      tests    under     certain   HOTAS inputs in system and
   effort, as they are often used as      conditions: to load different code into   subsystem testing; but in some test
   regression tests during subsequent     the processors, to reset the simulation   cases, actions must respond to
   integration testing.                   environment, or to change a selection     external stimuli in ways that are
   The elimination of delays caused       of emulators versus real subsystems.      difficult or expensive to coordinate
   by human interaction would permit      The initialization process is highly      automatically.
   the execution of more tests within     automated, but still requires some
   what is typically a limited amount     operator intervention. Complete           For example, the tester may define
   of time allocated for testing.         automation of the initialization and      initial flight simulation conditions
   An automated capability would          reset procedures is not practical.        such as the initial altitude, position,
   reduce the amount of post-test                                                   speed, etc of the CF-188 system, and
   analysis of test data required by      Automating the Injection of Test          then “fly” the system within the flight
   providing a capability to monitor      Inputs                                    simulation using the actual aircraft
   and analyse more variables during      Testing of real-time software at the      flight controls.     Some tests will
   the execution of a single test case    subsystem and system levels requires      require flight control inputs to
   than a human tester could. This        that the software be subjected to         compensate for inputs from the
   can be especially important in a       “real-world” test inputs at the           environmental simulation models,
   real-time situation where test cases   appropriate time. In the case of the      such as the varying wind conditions
   may not be repeatable, since           CF-188, these “real-world” inputs         generated by the Atmospheric Model.
   environmental variables cannot be      include environmental inputs, such as     Currently, the tester is required to
   completely controlled.                 atmospheric conditions, radar targets,    interpret the effects of these
   Automated testing tends to             EW emitters, various radio stations       atmospheric conditions and provide
   improve the organization of testing    and navigation aids, and inputs made      the appropriate control inputs. Such
   by imposing discipline on the          by the pilot. The ITF provides for        control inputs cannot currently be
   conduct of test planning and           the automatic injection of many, but      controlled or are not easily controlled
   execution, and provides better         not all of these inputs. In particular,   by the ITF command language
   measurement of testing coverage.       environmental inputs, radar target        because it does not support loops;
The goal of automated testing is to       return signals, EW emitter signals,       thus, there can be no such
reduce all aspects of the work of the     weapons signals, communications           control-loop compensation. The full
human tester while increasing the         signals from radio stations, and          provision of this capability in the ITF
understanding of the operation of the     navigation aids signals are all           test software would be the equivalent
software under test. The ideal, but       automatically injected in real-time by    of providing an automatic test pilot.
impossible,      automatic     software   the various dynamic simulation            This would mean test software as
testing device would be a black-box       models and/or simulators. Inputs          complex and hard to verify as the
into which the software is fed and out    normally injected by the pilot are        software under test.
of which would come a test report         injected either manually via actual
and a statement of correctness or         CF-188 flight controls in the ITF or,     As another example, The Target
incorrectness [12].       This perfect    in some cases, semi-automatically         Designator Controller (TDC) switch
device cannot exist, since it is not      using the ITF override capabilities.      is used to position the TDC box over
possible to do enough testing to                                                    a radar target visible in the radar
demonstrate correctness. Also, this       The following sections examine the        display or an actual target on the
device would require a test oracle        potential   for     increasing     the    Heads-Up-Display (HUD), and to
which could determine the expected        automation of those inputs on the ITF     activate some of the Multipurpose
result. Thus, the goal of automated       which are currently injected manually     Display Group (MDG) menu
testing is to amplify the human           or semi-automatically.                    functions from the HOTAS. While
capability to plan and conduct                                                      the position of the TDC indicator and
testing.                                  Automated HOTAS Control Issues            the activation of the MDG functions
                                          The Hands-On-Throttle And Stick           using the HOTAS inputs can be
In the following sections, possible       (HOTAS) is the primary flight             controlled using the ITF command
improvements to the automated             control in the CF-188. It provides the    language, this capability is not useful
testing capability are investigated for   pilot with control of the throttles and   in practice. The tester interprets the
the CF-188 testing setup. Some of         flight control surfaces. It contains      displayed video images and decides
these improvements have now been          switches which permit the pilot to        what actions to take. This decision
implemented. A brief status report is     control other functions, such as the      process and control is very difficult
given in the Conclusion.                  selection of radios, the selection of     to automate, since it would require a
                                          weapons, the release of weapons, the      capability to monitor the position of

the targets on the appropriate display   Automated      Switch    and     Data      although the amount of time saved
and a capability to coordinate the       Overrides: The addition of overrides       would be highly dependant on the
position of the TDC indicator and the    for the various switches and data          speed of the tester‟s response to the
target. In essence, this would require   entry devices would provide the            prompts. Third, it would not require
an additional processor and complex      capability to automatically inject the     any changes to the current ITF
calculations linking parts of the ITF    test inputs; but it would require a        configuration.
which          currently       operate   hardware modification to the various       There are also several disadvantages
independently.                           avionics control panels. This is           to the implementation of a
                                         difficult, because of the physical         semi-automated test capability. First,
Automated Avionics Subsystem             design of the various control panels,      the inclusion of the prompts to the
Control Issues                           which are actual CF-188 avionics           tester, and any error handling
There are several cockpit controls       components. As well, the modified          required to handle an incorrect or
and displays which are used by the       control panels would be unique in the      untimely input by the tester would
pilot to control the operation of        fleet. This, in turn, would make           increase the complexity of the test
various avionics subsystems within       their maintenance expensive and            software. Second, there may be
the CF-188. The MDG and the              unreliable.                                insufficient time for the tester to
UFCP are the primary avionics                                                       respond to a prompt during a
control panels in the CF-188, while      Emulators: Another option is to            real-time scenario.
the ALR-67 Radar Warning Receiver        replace the various control panels
(RWR) Control/Indicator is used to       with emulators. In the case of the         Analysis of Test Results
control the Electronic Warfare (EW)      UFCP and the ALR-67 RWR control            The analysis of test results during test
subsystem. The MDG includes three        panels, this option would provide the      execution requires that the behaviour
multifunction displays which each        capability to control the injection of     of the system/subsystem(s) be
have twenty programmable switches        test inputs automatically into the         monitored and analyzed with respect
around their display periphery, and a    Communications         Set    Controller   to the expected results. For system
Heads-Up-Display (HUD). The ITF          (CSC) and the EW suite subsystem           level and subsystem level testing of
has a current capability to monitor      respectively. In the case of the           real-time systems, this means that the
but not override the ALR-67 and          MDG, however, this option would            system and/or subsystem outputs
MDG programmable switches, and           impact the integrity of the system         must be compared to the correct
does not have a capability to enter      level and MC subsystem level tests.        value and the correct timing must be
data into the UFCP.                      In particular, the MDG display and         verified. In addition, there is a
                                         control functions are controlled by        requirement to measure the coverage
Automated Test Input Injection           processors within the MDG, whose           of the test cases to help determine
Requirements                             activity is in turn coordinated by code    when sufficient testing has been
The capability to control MDG            that runs on the MC. One of the goals      done. Analysis can be conducted
switches automatically is required for   of system level and MC subsystem           either in real-time during the
fully automated system level and MC      testing is to verify the real-time         execution of the test, or post-test
subsystem level testing of most          processing capability of the MDG           using a record and playback
avionics functions. The capability to    code, both on its own processors, and      capability.
automatically inject data into the       on the MC. Emulation of the MDG
UFCP is required for fully automated     would limit the ability to verify the      The ITF has no capability to
system level and MC subsystem            real-time processing of the MDG            automatically analyze test data during
testing of the navigation and            code. Furthermore, there is reluctance     the execution of a test. All real-time
communication functions.          The    among pilots to accept a system            analysis must currently be carried out
capability to automatically control      which has been verified using an           manually by the tester.
ALR-67 input data via the ALR-67         MDG emulator, since the MDG is the
switches is required for fully           primary avionics control for the           The following sections examine
automated system level testing with      pilots.                                    issues which impact the ability to
the EW subsystem.                                                                   provide an automated real-time
                                         Semi-Automated Testing: There are          analysis capability on the ITF and
There are two ways to provide the        several      advantages     to     the     issues which impact the ability to
automatic control needed: an override    implementation of a semi-automated         automate post-test analysis on the
capability for the switches and data     test capability, in which the operator     DRAS.
entry devices, or the replacement of     would be prompted for those inputs
the control panel with an emulator.      which cannot be automated. First, it       Monitoring and Recording Issues
In addition, given that fully            would improve the repeatability of         Limiting the recording of simulation,
automated testing is not practical for   tests by automating most test inputs       environmental, digital discrete signal
some functions, the possibility of       and controlling the injection of the       and MUX bus word variables to a
semi-automated testing is discussed.     others. Second, it has the potential       maximum of one hundred total
                                         to reduce the test execution time,         monitored variables has little

practical    adverse      impact    on    measure path coverage, although they       subsystem test is dependant on a
automated real-time analysis, since       can generate program execution             combination of variable values
the analysis of more than one             traces which can later be analyzed to      pertaining to air speed and the
hundred variable values in real-time      determine path coverage. In-Circuit        attitude of the aircraft. In this case,
would require very complex test           Emulators (ICE) can be used to             the test software could analyze the
software. Furthermore, the ITF has        monitor the execution of the CSC and       values of the pertinent variables and
the capability to record all MUX bus      SMS code, although they may cause          determine if the stall warning was
words and up to one hundred other         system level faults, such as timing        correct at the appropriate time.
variables; so additional analysis can     faults, that may not exist with the use    Given that the test software in this
be done post-test. On the other           of the actual processors.                  example could be of similar
hand, there are times when test cases                                                complexity to the actual stall warning
must be redone, with a different          Automated Real-Time Analysis               function software, this example also
selection of variables to be recorded.    Requirements                               shows how the overall complexity of
                                          While the ITF provides some                the software can be doubled, with
Real-time analysis of analog discrete     capabilities to automatically monitor      resulting additional verification costs.
signals cannot easily be automated,       and record test data, it was not
since these signals are monitored         designed to support automated              One requirement that is clear from
using        equipment     such     as    real-time analysis. Fully automated        the above is that the ITF command
oscilloscopes, power meters, and          real-time     analysis    requires     a   language      and     its    sequential
strip chart recorders. This limitation    capability     to    compare      actual   interpreter must be either upgraded or
has little impact on the automation of    monitored values to expected results.      replaced with a capability which can
system level or MC subsystem testing      Assuming        that    the     CF-188     provide real-time analysis.
since most test cases do not require      specifications     were     sufficiently
this       type     of     monitoring.    precise to permit either the               New DRAS
Furthermore, these outputs can be         determination of expected results for      Given that the current DRAS is based
verified either in real-time using        inclusion in a test program or the         on late 1970's technology, an upgrade
non-automated tests or they can be        derivation of algorithms to calculate      to its capabilities it is not considered
recorded for post-test analysis.          the expected values, the test              practical. Instead, a new DRAS is
                                          command language and the test              needed to provide the post-test
Real-time analysis of data displayed      controller need several capabilities.      analysis capability of the test data
on the various actual CF-188              First, they require a capability to        that was recorded during test
displays, such as the MDG displays,       calculate the expected results,            execution on the ITF. This new
the UFCP, and the ALR-67                  possibly based on the values of one        DRAS should provide a capability to
control/indicator, cannot easily be       or more monitor variables, in              display the test data in different
automated since the test software in      real-time. Second, they require the        formats, including a scrollable list of
the ITF cannot monitor the display        capability to compare the monitored        user-specified      variables    in    a
data. For example, the test software      values with the expected results.          text-oriented tabular format which is
cannot easily be modified to analyze      Third, they require a capability to        keyed on the global reference time
the correlation of the TDC indicator      select or skip test procedures based       generated by the time code generator,
and a radar target on the radar display   on the results of the analysis, and        a graphical plot format in which the
page on the MDG. The impact of            finally, they require a capability to      values of user-selected variables are
this limitation is minimized by the       report the analysis results. Included      plotted versus the global time, and an
ability to monitor and record the data    in the latter is a requirement to be       event history format in which the
sent to the displays. Furthermore,        able to indicate which tests were and      changes in values of user-selected
the actual CF-188 displays can be         were not executed.                         variables are listed against the global
recorded for post-test analysis using a                                              reference time at which they
Video Cassette Recorder (VCR),            The provision of the capability in the     changed. The new DRAS should
although there are problems in            test software to compute the expected      have an interactive query and
making use of these recordings, due       result increases the complexity of the     reporting capability which supports
to the difficulty of coordinating them    test software and imposes a                queries based on such criteria as the
with other test outputs on playback.      requirement to verify that the test        magnitude of the minimum and
                                          software computes the expected             maximum excursions, the average
Special test harnesses and tools          value correctly. In this context, it is    value of the variables, and the total
external to the ITF provide a             possible, especially in a real-time        number of transitions of the
capability to monitor the execution of    system, that a measured variable may       user-selected variables. It should
the MC and RDP subsystem code on          be correct either within a given           provide a capability to analyze and
their      respective      processors.    tolerance of the expected value or         display the program execution traces
However, they cannot be controlled        within a given range of values. For        generated by the various ICEs and
remotely by the ITF. Furthermore,         example, a test of the “stall warning”     the special tools used for monitoring.
these tools are cannot directly           function in a CF-188 system or MC          The use of the VCR to record the

CF-18 displays should be replaced        state at the commencement of the test      and analyzing test results, both
with a digital record and playback       execution.                                 during execution and post-test.
capability which uses the global
reference time. This would permit        CONCLUSION                                 It was concluded that the automation
the correlation of the displayed data                                               of tests designed to verify exact
with other recorded data. Finally,       The CF-188 system and its                  timing     requirements      and   to
the DRAS should provide a                subsystems        are     hard-real-time   coordinate inputs with the exact
capability to analyze analog data that   software systems. To perform               presentation of data on the displays
was recorded.                            properly, they must be logically           and indicators would not be practical.
                                         correct, and must satisfy timing           The problem is the high present and
An Automated Test Capability for         constraints. The maintenance of this       future cost of modifications to the
the ITF                                  software requires extensive use of         avionics used in the ITF and of a
While fully automated testing is not     testing, to verify the functionality and   capability to interpret and interact
feasible for all test cases, such as     timing of the modified software, and       with displayed data to enable
those which require the acquisition of   to ensure that the modifications have      selection of subsequent test inputs.
a target using the TDC and the radar     not impacted other parts of the            On the other hand, it is practical to
antenna elevation controls, there are    software. The goal of this paper was       automate the majority of test cases
many test cases which do not require     to examine how the degree of               executed on the ITF by implementing
the acquisition of targets. These test   automated system and subsystem             emulators for the MDG, UFCP and
cases can and should be automated.       testing capability could be increased,     the ALR-67 controls and displays,
Automation of these test cases           based on the current CF-188                and improving the test control
requires the implementation of           Integrated Test Facility (ITF).            language so it would be capable of
emulators for the MDG, the UFCP,                                                    controlling and logging execution of
and the ALR-67 RWR control panels,       Automation of testing on the PMV is        a wider range of test cases.
and the implementation of a test         out of the question, as it would
language which is capable of             require extensive modifications to at      Development of initial versions of
controlling and logging the control      least one aircraft. This creates a         MDG, UFCP and ALR-67 control
flow of the test activities based on     number of problems, and leads to           and      display      emulators     for
results while tests are in progress.     high expenses for marginal returns.        incorporation in the ITF is now
The implementation of the emulators      By the time PMV testing is used, the       complete. These efforts commenced
will provide both a means of             software has already achieved a high       with the development of a digital
automatically injecting test inputs      degree of reliability, and problems        record and playback capability for the
and a means of automatically             that do appear during PMV testing          control panels and display data of
monitoring and/or recording the          can generally be found using the ITF.      each of these control and display
display outputs for analysis of the      As well, there is no guarantee that        system. Subsequent phases of these
functional correctness of the system     more automated PMV testing would           projects will develop a more
behaviour.        Also, the global       significantly   improve     diagnosis      complete emulation of these control
time-stamp capability within the ITF     capabilities.                              and display systems. A new version
should be used to correlate the                                                     of the DRAS has been delivered,
display data on the emulators with       The paper has examined the CF-188          which can record pilot inputs and
the MUX and digital discrete data        software       maintenance    process,     replay the flight on the ITF. There
captured by the current ITF              organized around the various testing       continue to be plans to investigate an
monitoring and recording capability;     tools within the CF-188 software           improved test control scripting
however, this data cannot be used to     maintenance              environment.      language. It is also of some interest
analyse the correctness of the timing    Limitations in the application of          that, while this paper has focused on
behaviour of the system, since the       testing techniques in the current          testing in the traditional sense, there
emulators may not exactly emulate        CF-188        software    maintenance      has been a software process
the timing behaviour of the actual       environment were identified, one of        improvement initiative underway at
system components. For those tests       which is the limited automation of         the same time, and code inspections
which are designed either to verify      system and subsystem testing               are now part of the maintenance
the exact timing behaviour or the        capabilities.                              process.
exact MDG display presentation of
the system, either manual or             Regression tests at the system and         Further research could be conducted
semi-automated testing using the         subsystem levels are candidates for        to answer the following questions:
actual aircraft control panels will be   further automation. Current ITF            How can the current CF-188 system
required.      Automated test cases      capabilities were examined to              and software specifications be
should         ideally      commence     determine the extent to which further      improved from a testability point of
immediately after the power-on           automation is feasible. This included      view? How can test cases be
initialization or reset to ensure that   an     examination     of     possible     automatically generated from existing
the processors are in a known initial    improvements in automating inputs          documentation? What improvements

are    possible    in   non-intrusive          Equipment, CF-18 Software
monitoring of timing and program               Support         Re-engineering
execution within target processors?            Technology         Improvement
How can automation of test coverage            Team       Report,      Ottawa,
measurement and dynamic coverage               May 1995
control best be provided?                 [10] Mack, A.W., Subsystem and
                                               System Integration Testing in
BIBLIOGRAPHY                                   Software Maintenance: A Case
                                               Study of the CF-188", M.Eng.
[1]   D.W.       Campbell,      Timing         Thesis, Royal Military College
      Analysis in Hard-Real-Time               of Canada, May 1996
      Systems with Application to the     [11] Working Group Report, „A
      CF-188,      (M.Eng.     Thesis),        Testing Philosophy for the
      Royal Military College of                CF-188 Operational Flight
      Canada, May 1991.                        Programs‟, CF18 Weapon
[2]   Craigen, D. Gerhart, S., and             System Software Unit, Cold
      Ralston, T., “Case Study:                Lake, Alberta, Canada, 3 May
      Darlington Nuclear Generating            1990
      Station‟, IEEE Software, 30-32,     [12] Young, Dr N.J.B, „Automating
      Jan 1994.                                the Testing of Software‟,
[3]   Deutsch,      M.S.,     Software         AGARD                Conference
      Verification and Validation -            Proceedings No.343, 41-1 -
      Realistic Project Approaches,            41-13, Oct 1983
      Prentice-Hall Inc., Englewood
      Cliffs, NJ, 1982.
[4]   Falardeau, Capt J.D.G., “A
      Study of Computer Aided
      Software Engineering (CASE)
      Tools used in the CF-18
      Mission      Computer      (MC)
      Software            Development
      Process”,       4      Software
      Engineering Squadron Report,
      May 1995
[5]   Fetzer,      J.H.,     „Program
      Verification: The Very Idea‟,
      Communications of the ACM,
      Vol 31, No. 9, 1048-1063, Sep
[6]   IEEE Standard Glossary of
      Software            Engineering
      Terminology,         ANSI/IEEE
      Standard 610.12-1990, IEEE
      Press, New York, 1990.
[7]   Jahanian, F. and Mok, A.K.,
      "Safety Analysis of Timing
      Properties     in     Real-Time
      Systems", IEEE Transactions
      on Software Engineering, Vol
      12, No. 9, 890-904, Sep 1986.
[8]   Leveson, N.B. and Harvey,
      P.R., "Analyzing Software
      Safety", IEEE Transactions on
      Software Engineering, Vol
      SE-9, No. 9, pp. 569-579, Sep
[9]   Mack, A.W., A Study of an
      Automated Test Capability for
      the        CF-18        Avionics
      Operational              Support


Shared By: