Automating Traceability of Requirements to Test Case & Defect Logs - a case study


One of the major problems in software development stems from constant changes to software
requirements. Keeping track of such changes and ensuring that the test suite is also kept updated, is not a
trivial task. The success of testing efforts depends on the ability to track and trace requirements as they
evolve and to validate that they have been incorporated into the software.

In any test project, traceability should be considered as an important activity. In a world of changing

requirements and reduced time to market, a test practitioner’s tool is traceability. Traceability helps to

ensure test coverage and provides evidence that each requirement has been tested using a planned test


What is traceability?

In a software development life cycle, requirements are translated into specifications based on which code

is developed. Once the construction process is complete, the product is put through various stages of

testing before it finally goes live.

Requirement: A requirement describes a condition or capability to which a system must conform; either

derived directly from user needs, or stated in a contract, standard, specification, or other formally

imposed document.

Specification: A declarative description of what something is or does.1

Traceability is the common thread that ensures that the verification and validation of the product is

complete. Requirements tracing is the process of documenting the links between the user requirements

for the system you’re building and the work products developed to implement and verify those

requirements. These work products include software requirements, design specifications, software code,

test plans and other artifacts of the systems development process.        Requirements tracing helps the

project team to understand which parts of the design and code implement the user’s requirements, and

which tests are necessary to verify that the user’s requirements have been implemented correctly.2 Table

1 illustrates the concept of a trace matrix.

Table 1 – Comprehensive Trace Matrix

Requirement          Functional         Design          Code section      Test Condition     Defect ID
                     Specification      Specification
1                    1.1                3.1             Line 300-350      12                 D2
                                                                          13                 -
                                                                          14                 D6
                     1.2                3.2             Line 450-600      21                 -
                                                                          22                 -
                                                                          23                 -

                                                 Page 1                                       3/28/2012
Table 1 assumes that the trace matrix includes development activities as well. However, this is not

always possible as data may not be readily available. In the above example, Requirement Number 1 has

resulted in two defects (D2 and D6). These can be traced back to Conditions 12 and 14 respectively. By

drilling down further, it is possible to identify the impacted specifications and the relevant code section.

Where the trace excludes construction activity, the trace matrix should include other elements of the test

suite as given in Table 2.

Table 2 – Trace Matrix from Requirements to Test Suite

Requirement         Functional         Test           Test Case                 Test Script     Defect ID
                    Specification      Condition
1                   1.1                12             12.1/12.2                 T5              D2
                                       13             13.1/13.2/13.3            T6              -
                                       14             14.1                      T6              D6
                    1.2                21             21.1                      T8              -
                                       22             22.1/22.2                 T8              -
                                       23             23.1/23.2/23.3            T8              -

Table 2 drills further into the actual test case that failed. This gives additional information to the

developer to be able to zoom into the cause of the defect as well as assist the users in assigning a severity

classification to the defect. For example, the defect may not relate to a critical functionality or the user

could be trained to avoid the defect. These decisions become imperative when the time to fix is high and

the go-live date is non negotiable.

For small projects, maintaining a paper-based traceability or using spreadsheets would suffice in preparing

a trace matrix. However, in large sized business critical projects where the number of requirements is

voluminous and change is inevitable, it is vital to use automated tools to verify traceability relationships.

Automation helps in:

             Managing versions and changes

             Storing requirements attributes

             Linking requirements to other system elements

             Tracking status

It also helps the test practitioner easily identify the tests that are to be executed at the time of testing,

tests that are to be deferred and tests that are invalid. Traceability, thus, helps in the following areas:

        Requirements Management

                 Assessing the project impact of a change in a requirement

                 Managing the scope of the project

                 Verifying that all requirements of the system are fulfilled by the testing

        Change Management

                 Establishing a baseline

                                                   Page 2                                         3/28/2012
                 Determining critical dependencies to trace

                 Establishing traceability between related items

                 Managing Change

        Defect Management

                 Assessing the impact of a failure on a tested requirement

                 Verifying that the application does only what it was intended to do

                 Tracing the defect back to the requirement

Case Study

We share with you our experiments in traceability using an automated tool during User Acceptance

Testing for a leading Multi National Company in the Fortune 50 list. The project was part of a release for

an implemented product having nine enhancements. There were 43 requirements, 170 conditions, 1168

test cases, and 150 test scripts resulting in 107 defects. In all, there were a total of 45 documents

consisting of multiple versions of documents like Business Requirements, Gap Analysis, Functional

Specifications, Test Conditions, Test Cases, Test Scripts and Defect Logs. As the release was fairly large

and our involvement was from the requirements stage, we decided to use an automated tool for

traceability. The tool used was Rational Requisite Pro from Rational Corporation.

The study uses a real life situation of constant change in User Requirements during a typical Software

testing project. For this purpose, five different stages of change throughout the Test Planning and

Execution phases are identified. The five stages are Requirements, Analysis, Design, Development and

Test Execution stage. How a Trace Matrix was used during these stages and how automation helped in

achieving the purpose of Traceability is detailed in the following paragraphs.

Stage 1 – Requirements Stage

The Test Planning phase started with the understanding of the Application by the Test Team. This was

done using the pre-signed off Business Requirements documents provided by the Client. These documents

were imported into the test tool as a base line. The Test Team read through these Requirements

document and prepared Test Conditions (i.e. Conditions Version 1) that were again imported into the test


Test Condition: A Test Condition is derived from a requirement or specification. It includes all possible

combinations and validations that can be attributed to that requirement/specification.

Using the tool, the test conditions were linked to the User Requirements given in the pre-signed off

Business Requirements (BR) document. The activity in this stage and the trace output from the

automation tool is represented in Figure 1 below:

                                                  Page 3                                      3/28/2012
                                       Requirements Stage

                                        Pre-Signed Off
                                         BR Version

                                            Version 1
                                             Figure 1
Stage 2 – Analysis Stage

Requirements were frozen and the Signed Off Business Requirements were published to the Test team and

the Product Assurance Group of the client. Changes to requirements from the initial state occurred. The

changes were in the following categories:

              New Requirements

              Modified Requirements

              Deleted Requirements

The Product Assurance Group of the client prepared a Gap Analysis document based on the differences

between the existing application and the new user requirements. The task of the test team was to identify

the accepted changes to the requirements and update the baseline document within the automated

environment. These updations were in the form of enrichments and eliminations as per the signed off

business requirements. Based on the modifications to the baseline document, the impacted conditions

were highlighted as suspect links (links which need to be either modified, added or deleted) by the

automated tool and these suspect links were reviewed manually. After the review, decisions were taken

whether to remove the trace already established between the Requirement and Condition or to clear

suspect links as the trace already established might still be valid (even after the change).      This could

happen in situations where the modification of a requirement may result in the revision of few conditions

only. Revision to the Conditions version1 was prepared resulting in Conditions version2. The activity in this

stage and the trace output from the automation tool is represented in Figure 2 below:

         Analysis Stage
             Signed Off
             BR Version

             Version 2

                                            Figure 2 4                                          3/28/2012
Stage 3 – Design Stage

The Gap analysis document published showed the requirements that were to be dropped / deferred and

enhanced based on the analysis. Product assurance team of the client then finalized the Functional

Specifications (FS) based on the gap analysis. Revisions to the existing conditions document were done

based on the Functional Specifications. After the revision of the conditions (Conditions version3), Test

cases were designed for all valid test conditions.

Test Case: A Test Case gives values / qualifiers to the attributes that the test condition can have. Test

cases, typically, are dependant on data / standards. A Test Case is the end state of a test condition, i.e.,

it cannot be decomposed or broken down further.

A trace between the test condition and multiple cases for the conditions were established and linked within

the automated tool. As test cases were prepared, the test team raised clarifications to the Product

Assurance Group of the client. These clarifications were documented in a Clarifications document.

Clarifications at this stage, sometimes led to changes in the business requirements. Under these

circumstances, clients were expected to confirm that the changes suggested supercede the business

requirements. These changes are normally retrofitted into the business requirements at later phases of

the project to complete the documentation. It is important to mention here that it is beneficial to try and

trace risks identified throughout the project life cycle to the requirements. As a particular risk increases in

severity or is mitigated, the risk’s effects can be reviewed for its impact on the requirement.

Depending on the nature of the change in requirements, test conditions were modified and a new version

released. As this was a parallel activity, Test Conditions Version3 included the changes due to clarifications

received from clients at this stage. Trace matrix was then updated with Clarification document and related

client approvals that are imported into the automated tool. The activity in this stage and the trace output

from the automation tool is represented in Figure 3 below:

                                                     Page 5                                       3/28/2012
              Design Stage

               Signed Off
                BR / FS

               Conditions                                                        Clear Suspect
               Version 3
                                                                                  Remove Trace

               Test Cases

Stage 4 – Development Stage
                                                    Figure 3
At this stage, a trace between requirements, functional specifications and any other interim documents

authorizing requirements change had been traced to test conditions and test cases. The Test team started

preparing the test scripts based on the latest version of the test conditions and test cases. The test scripts

contained multiple test cases put together to form a logical user operation scenario. The trace from Test

Cases was a many to one relationship with Test Scripts.

Test Scripts: A Test Script contains the Navigation Steps, Instructions, Data and Expected Results

required to execute the test case(s).

The development teams had completed their design and were in the process of building the code. Changes

at this stage were minimal and mainly in the form of deletions or deferring certain requirements to the

next release. The Trace Matrix was the main document the test team referred to while preparing the test

scripts required for test execution.    Once the test scripts were completed, these were linked to the

respective test cases. The activity in this stage and the trace output from the automation tool is

represented in Figure 4 below:

                                                  Page 6                                         3/28/2012
           Development Stage
                Signed Off
                 BR / FS /
                Other docs

                Version 3

                Test Cases


                                            Figure 4
Stage 5 – Test execution Stage

Test Scripts were executed based on the Day-wise test execution plan. During the course of the execution,

depending on the way the system had been implemented and the way the User Requirements were

addressed in the application, the following happened:

              Invalid Scripts

              Additional Scripts

              Modifications to Scripts

Trace was also established from the Test Scripts to the Defects Log. The Test Plan was to conduct three

test iterations. The second and third iterations were to concentrate on the defects raised during the first

test iteration. In order to ensure all defects were retested and the impact points regressed, the trace

matrix was used to identify the test scripts that need to be run for the next iteration. By doing a reverse

trace from defects to requirements, it was possible to identify those requirements, which had defects and

accordingly schedule the scripts for the next iteration. E.g. In the example given in Table 1, two defects

D2 and D6 were detected by using test scripts T5/6. Since these defects impacted Requirement 1, we can

take a decision to execute all scripts belonging to that requirement, i.e., Test script T8, in the next

iteration. The activity in this stage and the trace output from the automation tool is represented in Figure

5 below:

                                                 Page 7                                        3/28/2012
Test Execution Stage

     Signed Off
      BR / FS /
     Other docs

     Version 3

     Test Cases

    Test Scripts


          Figure 5

            Page 8     3/28/2012

A Test Summary Report was prepared by the test team towards the closure of the project in order to

obtain sign off from the client. The Trace matrix was used to highlight the following:

         Test coverage

         Features not tested due to agreed reasons

         Test Scripts not executed

         Defects deferred and not fixed

         Defect analysis providing the broad categories of the cause of defects

         Validate Scripts to be a true representation of requirements

The decision to go live with the Application Under Test was made by the client through a clear

understanding of the residual risks and the impacted work products at the end of the test, given the above

information. Thus, using the Business Requirements as a base-line document, it was possible to:

         Ensure that each requirement has been traced to a specification in the Functional Specifications

          document and to a test case in the test script, thereby achieving one-to-one test coverage.

         Assimilate the impact of changes made to any of the base-line documents by automatically

          indicating “suspect requirements” when a linked requirement is modified or deleted.

         Assess Business impact of defects by tracing the defect id in the defect log to the business


Requisite Pro, the tool that was used, integrates with a familiar environment, Microsoft Word, and a

database to provide a framework for managing requirements. Requirements are itemized within

documents such as Business Requirements, Functional Specifications, Test cases and Defect Logs, and

managed using an integrated database.

Lessons learnt

         Traceability gives insight into functionality fine-tuning.

         Traceability acts as a strong foundation for resolution of Clients, Developers and Tester bias.

         Traceability is effective in test closure.

An automated tool will make the job easier, but is no substitute for a good process!


                                                       Page 9                                     3/28/2012

To top