Docstoc

Test Boundaries

Document Sample
Test Boundaries Powered By Docstoc
					Test Boundaries White Paper

9/4/2007

A Technical White Paper on Test Boundaries
Author: Mike Stoddard, Senior Engineer

Abstract
This article describes the one of the most intractable problems facing software development teams today; the cost of testing. According to most pure Systems Engineering Analysis experts the testing of a System requires at least as much effort as its development. In reality, the budget for testing a system is a great deal less than the budget created to develop it in the first place. In many instances, fully testing or retesting a system is simply not feasible. It would take too much time and cost too much to fully test the system. Therefore the challenge of the test team is to find a way to detect and fix most of the major problems in a system prior to its deployment. The problems of testing fall upon two actors in a system’s development, namely, the engineer and the integration tester. For the engineer the challenge is that as the system becomes better defined, its specification becomes more disconnected from the original requirements. For the integration tester the problem is that tests must be developed to test the requirements and threads of the system rather than insuring that all the developed features are exhaustively tested. This usually occurs despite an ever shrinking testing budget. In analyzing the problems faced by both engineers and testers a number of key facts emerge that could help the test engineer. Testing to requirements is only a valid test of the software if the requirements have been kept in synchronization with the code as it is developed. Reducing test personnel will not go far enough in reducing testing costs because most test teams are at their minimum staffing level already. The key is automatic generation of tests based on a static analysis of the system’s source code. In this paper we introduce the concept of a Test Boundary. A Test Boundary is a virtual wall around a portion of a software system that represents the portion of the code that is evaluated during a specific run of a test. It provides a conceptualization of the system not only along its programmatic details, but also around the elements that are executed during a particular test. Some of these Test Boundaries already identified include the “Unit Test Boundary”, the “Change Test Boundary”, and the “Change Impact Boundary”. The objective is to determine ways to generate these boundaries such that a minimum number of tests exercise the maximum amount of code in the software system. Trident Systems has introduced a tool called Safe Test Boundaries that is designed to help both developers and test engineers manage and generate test boundaries for a software system.

Testing Software in the Current Environment
Today software is becoming more complex as larger, componentized, and multithreaded/multi-processor systems are developed. This complexity is even more pronounced as the requirement for interoperability is demanded both from new systems and legacy applications. In many of these systems the ability to fully re-test when upgrades are made is simply not feasible. Budgets for integrated testing are constantly being cut while the time required to fully test/re-test a large system is simply not feasible. As such integrated test teams must make tradeoffs, hoping to detect the most pressing deficiencies while leaving the more obscure problems for a later maintenance upgrade.

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

1

Test Boundaries White Paper

9/4/2007

However in reality a maintenance upgrade often is bundled together with new functionality that often introduces more high priority deficiencies that have to be re-tested and dealt with. In many cases the low priority problems that are detected are never repaired. The question becomes how to test more while at the same time reduce the time and expense of testing complex applications as they become larger and more complex. In order to answer this question we’ll need to look at the current practices that make the testing of large complex systems so difficult and expensive. First we’ll look at the problem faced by many engineers. Next we’ll look at the problems faced by many testers. In this document Trident Systems will present a number of diagrams representing a portion of a software system. These diagrams are directed graphs in which the nodes represent some source code element such as a class or a function and the edges represent software dependencies between two nodes.

Figure 1 - In the directed graphs that follow, the direction of the arrow indicates dependency. Therefore in this example it is said that node A depends on B. If B changes, then A will need to be retested.

The Engineers Problem
Most large and complex software projects undergo a rigorous engineering process that defines the data elements and the program in every detail prior to the beginning of its development. This process begins with the definition of the requirements of the software. Then the process proceeds to the software architecture models, followed by the design and finally the detailed design. Once the detailed design documentation is completed the Systems Engineering is said to be complete and the software development can begin. The problem here is that the Systems Engineering evolves through a number of steps that creates an ever-increasing level of detail. Each step in the process becomes further removed from the original requirements definition. And it is from the requirements definitions that the tests are developed. The development of the architecture models may drive the creation of new requirements as should the design and detailed design phases of the project. However, for complex projects it is difficult to insure that all the issues determined during the detailed design get pushed back into the requirements specification. Once software development begins this “disconnect” between requirements, design and actual code developed becomes even more pronounced. Often individuals who have no experience with systems engineering at all are the ones who perform the software development. This means that these individuals will not be able to recognize when they,

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

2

Test Boundaries White Paper

9/4/2007

in solving a particular problem have created a situation that requires a change in the requirements. The key to solving this problem is to either insure that the development and maintenance of the program stays connected to the engineering documents or to regenerate the engineering documents based on analysis of the software. The industry has been unsuccessfully trying to develop a process to keep engineering documents up to date for the last twenty years. One method to keep these documents up to date is to perform software analysis through white-box dependency testing. That is, the development of a custom source code parser to produce a dependency graph so that additional analysis may be done on the actual code that was developed. This has certain advantages over manually keeping documentation up to date in that it relies on the product of the development process. It also has advantages over using exiting compilers in that the custom parser/analyzer may perform optimizations not available in a standard source code parser.

The Testers Problem
Once an application is developed an integration phase usually begins in which all the various pieces of the application are fused together into the final application. This phase generally exists for all large software development efforts including large upgrades and maintenance. Up to this point developers and development teams have been conducting ad-hoc unit testing of their assigned bit of the application using whatever data sets they could cobble together during their development. Integration is usually as complex as the application under development. In today’s environment, in which applications are being re-engineered to be componentized so that they may provide services to clients, this level of complexity is even greater. Components are being integrated that had no connection to each other when originally developed. Multi-threaded, multi-processor, and, and multi-computer applications also are difficult to test and debug because of the asynchronous nature of the environment. Processes that work just fine in a synchronous environment may fail unpredictably when placed in an asynchronous environment. During the Integration Testing Phase an independent test team usually conducts tests of the project based on the detailed requirements. Here is one point at which inaccuracies can crop up between software and testing. If the Engineering documentation is not kept up to date, then there is no way for the test team to accurately know what features to test for. Automatic Test software typically passes data through components or creates events that simulate/stimulate the selection of user interface controls to see how the applications behave under specific conditions. However, this method of testing often does not exercise all the code developed for an application. The big question facing business today is how to reduce the cost of testing while maintaining good quality of software produced. The easiest way to reduce the cost of testing is to reduce the number of personnel engaged in the testing process. However, in most cases, the test team is already at a minimum and reducing the size of the team may not be feasible or advisable. It may, in fact be a single individual developing and running tests for a large and complex application. The job of the test team is to develop the tests that will be performed on the application, insure that all branches of the developed system are tested adequately, develop the data that must be used to test the system, and

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

3

Test Boundaries White Paper finally, to run the tests on the system. intensive.

9/4/2007 This process is time consuming and labor

Automated generation of these artifacts is the key to reducing the cost of testing. If the tests and the test data can be generated automatically by some application to suite of applications, then a large part of the effort in creating the tests is reduced. However, in order to generate the tests that an application needs to perform, a detailed understanding of the software structure is required. We come, then to a common need for an analysis of the structure of the software application in order to regenerate our engineering documentation and to generate complex tests from simple ones. Some approaches for analyzing a software application involve reverse engineering the object code or reverse engineering the application itself. While these activities are useful when the need exists to discover the structure of an application when source code is not available, there is one primary deficiency with this approach in the area of test/retest generation. Reverse engineering is not able to distinguish between the portions of the software that were developed and those that were integrated. Developed software means the actual code that is what makes the application unique and solves the particular problem or performs the particular function that the application was designed to do. Integrated software means software that is part of the compiler environment, or the operating system, or third party libraries that were incorporated into the application to provide the infrastructure on which the application was developed. While some may argue that many of the operating system features or integrated third party libraries are provided as run-time libraries many of the features of the compiler environment are linked directly into the application making it impossible to distinguish where the functions of the software leaves off and the compiler environment begins. The logical next approach would be to analyze the application’s software directly. To do this requires the use of a source code parser. The source code is parsed into abstract syntax trees and then analyzed to determine dependencies and the scope of dependencies. In this approach the boundaries of the software application are clearly defined. Any source code element not directly defined by a source file read becomes part of the application’s “operating environment”. The elements described, while they may compile properly, will do so only because the operating system, compiler, or some Figure 2 - A notional dependency graph third party library provide the representing a simple software application functionality required and are considered “external” to the application. Figure 2 illustrates a software dependency graph, which is the output of this approach. Note in Figure 2 that we’ve also included the application’s data points. The triangles represent files, the black circles represent network TCP/IP sockets and the double bar represents a

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

4

Test Boundaries White Paper

9/4/2007

database table. A Key need exists to identify potential data points when representing the dependency graph because this will allow the tester to identify the threads that each of his or her tests execute. When developing tools to use to analyze software that will generate tests, two approaches may be taken to parse the application’s source code. • • Use a fully functional compiler to parse the application code. Develop a custom parser to parse the application code

The difficulty in working with a compiler’s parser is that the purpose of a compiler is to parse the source code into object files that represent machine instructions. Issues like dependencies and the scope of variable access are analyzed only in a cursory way. If the compiler cannot resolve a reference it defers action on it to the linker, which completes the task of creating an executable application. The tasks needed to generate and analyze tests for a software application involve a detailed knowledge of the dependencies between elements in the application and some indication of the scope of these dependencies. Most source code compilers are not designed with this task in mind. This would, therefore require a high degree of manipulation of the models produced by the compiler, which would actually impede the performance of the tool. Another limitation of the compiler is that many are created to optimize the compilation of source code developed in a single language. In order to work with another language the tool needs a separate compiler. And the dependency analysis capabilities of these compilers will not allow the models to analyze source code from multiple languages simultaneously.

Test Boundaries
The concept of a “Test Boundary” is that of a virtual wall around a certain portion of a software application. This wall represents the portion of code affected by a particular test. Initially the “Test Boundary” contains the entire developed application. The concept allows for the creation of smaller internal test boundaries around which an application may be divided. For example, Unit Tests are tests that are scoped to a single unit, which may be as small as a Class or a Function. The Unit Test is tightly coupled to the code and is usually developed on an ad-hoc basis. Another type of test that is well defined is known as (among other names) the Thread Test. A Thread Test is designed to test a particular thread of functionality across the entire application. Tools that test threads are tools that, for example, wrap or capture actions made from the User Interface and generate the events that propagate the thread. The end of a thread may be the display of another User Interface component, or the generation of an error message. Another kind of threaded test is a Data Test. A Data or Telemetry Test pushes a set of pregenerated data through some driver that interfaces with the application, allowing the application to respond as though a sensor or the network were streaming data into the application. This test data is usually designed specifically to stress the application in order to more completely test its capabilities. These are typically the kind of tests that need to be created in order to perform integrated testing or when testing the application to its requirements. All applications will have test sets, whether they are automatically or manually generated.

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

5

Test Boundaries White Paper

9/4/2007

Each time a test operates on a limited scope in an application it is said to be operating inside a “Test Boundary”. Therefore each individual test, including the Unit Tests form small Test Boundaries within a given application. Additionally other boundaries may be defined that would form logical groupings for individual tests. For example, a highly componentized application would have smaller groups of Test Boundaries for each individual component, and another set specifically for the entire application. A Suite of applications would need to have a set of tests for each individual application, and a larger set of tests to insure that the application interaction behaves as intended. This implies a “loose hierarchy” of test boundaries associated with an application. We call the hierarchy “loose” because higher-level tests will need to be defined across and through lower level boundaries. For example, take an application that provides access to a data store through a web-service and displays the results via web enabled client. In such an application the web client will need to have its own isolated Test Boundary to insure that the client operates properly, that it can discover the web-service, and that it knows what to do when it can’t find the network, or the web service. Also the web service will need its own Test Boundary to insure that it can access the database, accept and send data, and react correctly when a data request cannot find the data in the repository, or when the network or the client cannot be found. There will also need to be a third Test Boundary. This one will test the interactions between the web client and the web-service. In a sense the Test Boundaries for the client and service handle the “tests of failure”. That is, the tests that tell the application or service what to do in the case of a data or connection failure because these tests are limited only to the client or service. Logically, then the “higherlevel” Test boundary will handle the “tests of success”. That is, the tests that test the transfer of data between the web client and the web-service and back again. In the example described above, in which a hierarchy of Test Boundaries is created to identify individual components of a web-based application, it can be seen that the tests to be performed in the higher-level Test Boundary may involve retesting elements that may have already been tested in the lower level Test Boundaries. What is important to note is that the higher-level boundary may completely encompass lower-level Test Boundaries. In the instances in which the lower level tests did not adequately test all conditions of all the code, then retesting at the higher level continues to be productive and necessary. However, if the lower-level tests do completely test the component to which they are assigned, then re-running those tests may not be necessary. Tbd picture of the web app dependency graph This is an important distinction in terms of reducing the cost of testing. If both the higher-level test and the lower-level test perform basically the same function with respect to the element being tested, then one of these tests may be safely removed. In fact, the goal here should be to eliminate the need to develop two tests for any single element of the software application. It represents a duplication of work, which is inefficient and costly. For example, in the example described above the web client only defined tests for the functions it handled exclusively. For example, the web client Test Boundary contained tests for displaying a user interface, responding to user interface events, or handling errors. The web-service contained the tests that pertained to the access of data, and the handling of errors, and the higher-level application contained the tests that involved the transfer of data between client and service.

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

6

Test Boundaries White Paper

9/4/2007

Maximizing Unit Test Boundaries
How is this concept useful to the Engineer who is trying to reduce the costs involved with developing and running Unit Tests? Consider a small software application that is characterized by the dependency graph in Error! Reference source not found. above. The circles might represent classes of the system and the arrows would represent a dependency between two classes. The structure of the graph therefore is a static characterization of the call flow for the application. We introduce the concept of the Unit Test Boundary. This boundary represents the tests that will be used to individually test each element in the graph. The idea is to completely test each node and each arrow in the graph. We begin by creating a boundary around each class as pictured In Figure 3 below.

Figure 3 – The dependency graph with Unit Test Boundaries assigned to each class

In a large software development effort, in which there will most likely be hundreds of classes and many thousands of functions and dependencies it is not feasible to devise a plan that includes the development of unit tests for every class. Therefore a maximization step is required to determine the Test Boundaries that will encompass the largest area while still preserving the integrity of the Unit Test. For example, in Figure 4a we’ve isolated two classes in our dependency graph. If we expand each Unit Test Boundary to encompass the next level of dependency in the graph we get a pair of boundaries similar to that pictured in Figure 4-b. The red dependency grouping is entirely encomapssed by the yellow dependency grouping. Therefore it is more economical to develop only the Unit Test for the yellow grouping. When the pink group is removed we see the yellow group as pictured in Figure 4-c.

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

7

Test Boundaries White Paper

9/4/2007

a

b

c

Figure 4 - Expansion of two Unit Tests one level. The pink group is entirely encompassed by the yellow group in Figure 4-b and so may be removed as shown in Figure 4-c.

Extending the concept to the entire graph is fairly straightforward. However, it should be noted that expanding and combining each Unit Test past the first level will produce one Unit Test Boundary for the entire connected application. The goal is to determine which classes can best be combined to form the smallest number of Unit Test Boundaries for the application. Figure 5-a pictures the dependency graph with all the Unit Tests expanded by one level of dependency. Figure 5-b shows the small Test Boundaries that were removed as a result of the expansion. They are the pink group in the upper right, the small green group in the upper left and a small orange group in the lower right that itself did not get expanded. At this point what reamins will for the basis for all subsequent expansion operations. The reason for this is that when we expand to the next level, we will simply be subsuming an already existing Test Boundary. If we choose the largest Unit Test Boundaries and expand them one level we are able to reduce the number of Unit Tests to be developed to five as shown in Figure 5-c.

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

8

Test Boundaries White Paper

9/4/2007

a

b

c

Figure 5 - Expanding the initial Unit Test Boundaries yields a number of boundaries that may be removed as the pink group in the upper right corner, and the small green group in the upper left corner of Figure 5-b. Further expanding and reducing the largest groups yields the five Unit Test boundaries as shown in Figure 5-c.

When we started this section we asked a simple question. How is the concept of test boundaries useful to the engineer? Our answer is that providing a dependency graph of the software as it has been defined by the engineer can be a useful tool in his or her efforts to develop the Unit Test strategy for the application. Instead of building ad-hoc Unit Tests based on how he thinks the software should be running, he can use the concept of maximizing Unit Tests by expanding them along their dependencies. This information can be used to eliminate the development and execution of tests that are redundant.

The Thread Test Boundary
We can ask the same question of the Test Engineer. How is this concept useful to him and how can it be used to reduce the cost of testing an application? To answer this question we introduce the concept of the Thread Test Boundary. A “Thread Test Boundary” is a boundary around the portion of the code that is actually executed during the run of a particular test thread. There may be a number of different test teams involved with testing a system, especially if it is a large and complex one. Integration testing is only one phase of the entire verification procedure. There may be additional configuration and requirements testing that will take place. All of this testing occurs after the development of the system baseline is complete. How then doe a team with limited resources test as much of the code as is possible to find any critical faults which may delay the final delivery of the system? The key task employed by the Tester in this kind of environment is to pass some test data into the application and observe that the results are as expected. This implies the things the Tester needs to know before the test is developed. The Tester needs to know what actions are possible with the application, and what the expected results of those actions will be. This information is usually specified by the requirements. To the tester

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

9

Test Boundaries White Paper

9/4/2007

the application or system is a black box through which requirements are tested by exercising the software. As mentioned above this becomes problematic when the requirements are not kept up to date with the developed system. It is also problematic when there is no traceability between the requirements and the developed code. A key thing to note here is that when this testing begins, each test is designed to provide some input in order to produce some output. The input may be data passed through a file, or a network socket connection, or a gui component, or it may come through some external library function that works with a more raw form of the data. This is the reason that we represent the data points in the dependency graph. First, it allows the analysis process to be applied to code elements that are ony connected through these data points. It also allows the tester to easily identify the entry points for a given test. The tester evaluates the test by looking at what data is passed into the application. He or she can then look at the dependency model and determine which data points are used to initiate the test. Mapping a test to its input and output data points will allow the test to be mapped to the code elements it may touch. The dependency traceability from the input data point will be a superset of the actual code executed by the test. The expected output of the test indicates the specific activity that the test should activate. An integrated test should provide one kind of input and expect one or possibly two kinds of output (pass/fail). Now, in a real world environment there will probably be a number of input and output data points. However, it should be noted that the subset of the code elements that are executed during a test may be more precisely defined as the dependency path between the test's inputs and its output data points.

a

b
Figure 6 - Three examples of Thread Test Boundaries.

c

Analysis of the dependency graph can immediately answer a number of questions pertaining to the input and output of a test. Is there a software path between the given input and the given output? Are there multiple paths? What is the shortest path? From the answers to these questions comes the basis for the formation of a Thread Test Boundary that can represent the test in the code. There may be a number of thread test

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

10

Test Boundaries White Paper

9/4/2007

boundaries that are definable given a particular dependency structure. For example one test boundary might consist of the shortest path from an input to an output. Others may be defined that traverse each path to the outputs that exist in the graph. Yet another may be defined that include all paths from input to output. What we are describing here is a way in which Thread Test Boundaries may be automatically generated from the inputs and outputs detected by static analysis of the application's source code. This process should be fairly easy for an integration tester to enter. He or she need only know what inputs and expected outputs the test provides to be able to map the test to its executed code. Creating these traces automatically can help the tester determine the nature of the tests that need to be developed. It can also help him or her determine whether any of the developed tests are redundant, as well as whether any of the developed tests are leaving some code untested (see Figure 7). Providing this information to the tester will shorten the development time of tests and allow the tester to know what code is being tested and what code is not being tested without requiring subject matter expertise. The concept of the test boundary provides an organized and managed way to visualize tests in terms of the source code involved.

Figure 7 - Notice that the thread tests do not reach all classes in the diagram.

The Change Test Boundary
When changes are made to a system it invalidates all the Test Boundaries associated with that change. We call this the Change Impact Boundary. The Change Impact Boundary contains all the tests for all the elements that were changed from one baseline to another. Any higher-level Test Boundary that spans, intersects, is a subset or the union of a Change Test Boundary is invalidated as a result of the change and must be retested. However, those elements that do not span, intersect, are a subset or the union of the Change Test Boundary may be safely recertified. In other words, those tests do not need to be re-run.

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

11

Test Boundaries White Paper

9/4/2007

In Figure 8-a below the changed nodes are shown in yellow. The boundary defined includes all the changed nodes and all the nodes that have dependency traces to the changed nodes. Notice that there are a number of output data points that are included in the Change Impact Boundary. The yellow Thread Test Boundary pictured in Figure 8-b does not intersect the change test boundary therefore it may be safely recertified without running it. This is the key capability that can reduce the costs of testing. By determining which tests are not affected by software change the test team can reduce the amount of regression testing to be done to reliably test the changes. We also provide the tester with concrete information about the nature of the changes posted to a particular application or system and what elements are affected by that. This removes the guesswork involved with maintenance testing.

a

b

Figure 8 - A Representation of the Change Test Boundary. Changed Nodes are indicated in yellow. The yellow test boundary may be safely recertified without executing the test because it is not dependent on the changes made.

What we describe here is a capability which provides both software developers and testers with a way to define and collaborate in the test process. Test Boundaries developed by developers to perform unit testing may be able to be re-used when integration testers start developing their own Test Boundaries. In fact the integration testers may improve the tests, so that when the developer uses them again, for example when an upgrade is being worked on, the new test will be more accurate. The cost of testing is reduced both by reducing the number of tests that need to be developed, and by informing the test team when changes are made what precisely needs to be retested.

Safe Test Boundaries Software
Trident Systems has created a software application designed to allow both developers and testers to create and manage test boundaries for a given application. This tool is a first

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

12

Test Boundaries White Paper

9/4/2007

step toward a more comprehensive test generation and management capability as described above. The tool is built using pluggable parser technology that will allow mixed language systems to be analyzed as easily as single language systems. Using custom parsers allows us to have full control over the analysis and its performance. This also provides the user with a clear distinction between the code that belongs to the system under analysis and that which is part of the infrastructure. This application, which we call the “Safe Test Boundary System”, performs detailed dependency analysis on parsed source code, including the establishment of indirect associations between code elements, through data points such as common global variables, or files on the file system, or through common sockets or other network connections. The Safe Test Boundary tool forms complete and correct static associations between functions and classes in the software system. It then allows the user to create and manage the Test Boundaries that will be necessary to test the system. The purpose for the software is to provide both developers and testers with a common place to define and develop their tests. The tool provides an easy way to report when multiple Test Boundaries have been defined for a single function or class and can provide a way to automatically generate Test Boundaries for a given system. A number of techniques are used to generate test boundaries including the automatic creation of the Change Test Boundary and the Change Impact Boundary when a new baseline is introduced to the tool. The tool has the capability to differentiate the significant changes in an application’s source code and report those changes to the user. Changes in white space or in comments are ignored. Safe Test Boundaries then uses the detailed dependency analysis to determine which elements in the software system have not been changed, but are dependent on the elements that have changed and should therefore be included in the Change Impact Boundary. Once the Test Boundaries for a software system have been defined, then the tests that have already been developed may be applied and mapped to the test boundary. The tests that have not yet been developed will still need to be done; however, the test engineer will be able to reuse tests created by the developer to perform unit tests. He will be aware of these tests by looking at the areas of the program he wants to cover and see which portions have already had tests applied. The goal here is to reduce the number of tests that will need to be developed by reusing ones that already exist. After the application is deployed and the tests have been run, Safe Test Boundaries allows the tester to determine, from subsequent baselines, exactly what Test Boundaries are affected by the Change Test Boundary and the Change Impact Boundary. This provides additional savings by reducing the number of tests that need to be re-run in future baselines.

Conclusion
The testing of software systems will become more complex and more costly as the systems they test become more componentized, disconnected, and integrated with other applications. As this happens, the budget for testing is being reduced both in terms of time and in terms of money. Current engineering practices do not guarantee that the code developed meets the original specifications. This is primarily due to the fact that the

Copyright © 2006 by Trident Systems Inc. All Rights Reserved

13

Test Boundaries White Paper

9/4/2007

specifications do not tend to get updated as actual problems are solved during development. Many test teams are currently operating at the minimum staff level that is required to perform the tests. Fully testing or retesting a system is simply not feasible. Analysis of the application developed in order to generate tests is the key to the reduction of the cost of testing in the long term. Reverse Engineering involves the analysis of the executables or object code directly. However, this technique does not provide a clear distinction between the code that was developed for the system and the code that was provided by third party libraries or the compiler environment. Use of a commercial compiler to provide the analysis does allow the distinction between the developed source code and the compiler environment is still ambiguous in the products provided by the compiler. Also, the dependency analysis provided by the compiler is not optimized for the generation of tests. The ideal solution, therefore, is to generate a series of language parsers and dependency graph generators that will provide a detailed analysis of the static structure of the software system. We introduce the concept of a Test Boundary. This is a virtual wall that surrounds and is tied to a given portion of the source code. The entire software project represents one Test Boundary. Each individual code element also represents another set of Test Boundaries. The optimization of Test Boundaries involves the creation of the fewest number of Test Boundaries that will cover all the code in a given software system. It is an iterative process that can generate the tests required to test the system. Trident Systems has developed the first step toward this goal. This is the Safe Test Boundaries software analysis tool. This tool has the capability to parse existing source code structures and generate an initial set of Test Boundaries based on a few simple rules. The Safe Test Boundaries software also has the capability to load a second baseline, differentiate it from the first baseline, and report the results as a Change Test Boundary. The application also can indicate the Change Impact Boundary, which includes the Change Test Boundary and all the elements in the software system that did not change, but were dependent on the elements that were changed. Safe Test Boundaries provides both developers and testers with a tool to generate and manage Test Boundaries. As such the development of tests becomes a collaborative process over the course of the development of the software. Unit Tests developed by developers to test their particular portion of a project can be described using Safe Test Boundaries. Test Engineers can then determine which portions of the code have not had tests developed for them and begin to develop these tests accordingly. Unit Tests may be reused and/or improved in the development of integrated tests eliminating the tendency to duplicate the effort of developing tests. Safe Test Boundaries also has the ability to detect precise changes in later baselines of a software system, and apply those changes to the test boundaries developed. This allows the application to report which tests have been invalidated by the changes, and which tests do not need to be rerun as a result of a particular change. Automatically generating Test Boundaries, eliminating the duplication of effort, creating a specific plan for the development of tests, increasing collaboration between developer and tester, and reporting which tests need to be rerun as the software is updated all will provide a means to truly reduce the cost of testing in today’s complex software development environment. Copyright © 2006 by Trident Systems Inc. All Rights Reserved 14


				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:100
posted:8/18/2008
language:English
pages:14
Debra Orebaugh Debra Orebaugh
About