Test_Automation

Enabling a Converged World™









Keys to Automating

Regression Testing









915-2104-01 Rev B July 2011

Keys to Automating Regression Testing







Keys to Automating Regression

Testing

The Cost of Poor Testing



Network devices and Internet servers have evolved from single-function devices into highly complex,

intelligent components. Routers, switches, firewalls and web and other servers often need to handle triple-play

(voice, video and data) traffic along with associated web and media services. Network equipment

manufacturers (NEMs), service providers and enterprises are required to perform more and more testing in

order to ensure product quality and performance. The time and cost associated with proper testing using

manual and automation techniques often exceed the funds and time allocated to that function.



Time to market pressure often means making a choice between missing a market opportunity and shipping

products without adequate testing. Inadequate testing invariably results in poor product or service quality,

equating to increased support costs. As devices enter SOHO and home market spaces, the typical customer

is no longer the network/IT engineer able to perform troubleshooting on their own. Inexperienced consumers

often require multi hour support calls that can consume all the profits from the equipment sale.









Figure 1. Cost to fix bugs







26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 1 of 15

Keys to Automating Regression Testing









Poor quality products can cause brand damage and have substantial financial impact. As reported by

Newsweek, Microsoft® rushed its XBox 360 console to market a year ahead of its competition. As a result,

some boxes are completely malfunctioning; Microsoft said that it would take a charge of $1.05 to $1.15

billion to cover the failures.



The often used reason for skimping on testing is that quality assurance (QA) test cycles are too long – leading

to delays in bringing a product to market and lost opportunities. There’s more than a modicum of truth here –

proper testing should be performed at multiple stages of a product’s life cycle: during development, during

integration, and prior to deployment. What’s more, at each stage, testing should be performed for every new

software build, for every product upgrade, and for every operating system, security and product patch.



As shown in Figure 1, the costs associated with finding and fixing problems increases exponentially later in

the product life cycle. In other words, the earlier that testing is performed the greater the benefits.









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 2 of 15

Keys to Automating Regression Testing









The Need for Speed

Basic product requirements are simple to express:

 Quick time to market

 High quality

 Minimal expense

 Interoperability



Test automation is a key ingredient in satisfying these requirements. When implemented well and applied

consistently, test automation results in more tests, performed more often with better coverage



With test automation, it’s possible to perform far more tests than would be possible with manual testing. For

example, a router could be tested with many different combinations of services enabled at a time. Or,

functional tests can be run while the router is under varying levels of traffic load.



With test automation, it’s easy to run complete regression tests on a daily basis. It’s also easy to run

regressions, albeit on a larger scale, before deployment of updates, patches and network modifications.

Running regression tests early keeps us on the lower part of the curve in Figure 1, where it’s less expensive to

repair problems.



The payoff for using test automation is substantial. NEMs can use automation to:

 Automate regression tests for developers to ensure that nightly software builds haven’t created more

problems than they’ve solved.

 Combine tests from multiple vendors, platforms and applications during system integration.

 Automate regression tests for device and system quality assurance to ensure that integrated systems

continue to work together.

 Use regressions to ensure that patches and updates haven’t affected operation or performance.

 Track the result of regressions run over time to ensure that a project is making continuous progress

toward zero bugs and optimal performance.



Service Providers and Enterprises can use automation to:

 Automate device qualification tests both for initial deployment and upgrades.

 Ensure that patches and updates haven’t affected operation or performance.

 Ensure interoperability between current and prospective devices.

 Run daily tests to ensure that operation and performance haven’t been affected by network, file

system, or hacker activity.

 Reduce the effort required to create complex, real-world test cases.









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 3 of 15

Keys to Automating Regression Testing









Automation Tool Requirements



A typical work flow for developing a single automated test is shown in Figure 2. Multiple steps are required

to set up a device under test (DUT) or system under test (SUT), and the test bed(s) that are used to exercise the

DUT/SUT, program the test application, run the test, and analyze the results. A perfected test can then be

transformed into an automated test.









Figure 2. Individual test automation

This process is repeated by those responsible for each unit test. Multiple tests are held in the automation

repository and consist of:

 DUT/SUT setup and monitoring scripts

 Test bed setup

 Test scripts

 Regression test suites









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 4 of 15

Keys to Automating Regression Testing









Figure 3. Test automation workflow





As pictured in Figure 3, these items are reused by QA groups responsible for product testing and by

automation groups responsible for overall test automation. Individual tests are often reworked to enhance

their usefulness in a larger scope. This same flow is repeated multiple times in a product line’s life cycle, as

shown in Figure 4.









Figure 4. Test automation life cycle









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 5 of 15

Keys to Automating Regression Testing









With such extensive use of test automation, it’s important that automation tools be very intuitive and easy to

use, without the need for extensive training or programming experience.



Automation tools should:

 Be organized. Large numbers of DUT configurations, test bed definitions, test scripts, and test and

regression runs will be used by a QA community. It should be possible to easily filter, find and

compare all types of items.

 Perform DUT configuration and monitoring. Although device testing will predominantly be

accomplished through high-performance test ports, each DUT must be properly configured to

participate in each test. DUTs also contribute to the successful execution of the test – providing

statistics to be used to guide test operation and to make pass/fail determinations.

 Be vendor agnostic. Complex devices may require test equipment from multiple vendors to

provide complete test coverage. Automation tools must provide an integration mechanism so that tests

can be coordinated and run on different devices.

 Manage DUT and test equipment usage. This can be done through the use of scheduling and

device management. A shared environment requires device/port reservation and scheduled

execution.

 Provide the long view. The results of regression runs must be compared over time, allowing

management to track progress.









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 6 of 15

Keys to Automating Regression Testing









Ixia Automation Solutions



Ixia makes automation on many levels. Ixia provides:

 An all-in-one platform that runs all of its test applications in a multiuser environment.

 Powerful, interactive test applications that include or instantly generate test scripts.

 A Tcl-based set of compatible APIs.

 Test Conductor – a complete test automation framework that integrates Ixia and third-party test

applications and equipment.



IxAutomate is a complete automation tool, with over 90 prebuilt tests that cover routing and switching,

Carrier Ethernet, triple-play, firewall, and multicast tests. IxAutomate even includes industry standard tests,

such as RFC2544 throughput and performance tests. IxAutomate’s batch scheduler allows any number of its

tests to be scheduled for execution. IxAutomate includes sophisticated display and reporting functions, as

shown in Figure 5.



Most Ixia applications include a Tcl- or C-based API that allows automation of any test. Each test application

uses an interactive GUI to facilitate quick test development. When a test is perfected, an equivalent Tcl or C

script can be generated and run immediately.



With only simple modifications, scripts can be expanded to:

 Iterate over different combinations of parameters so as to cover all DUT modes.

 Iterate through multiple combinations of limits to characterize mixes of DUT usage.

 Run sequences of tests.

 Include pass/fail criteria.

 Perform special logging.

 Generate special reports.



Test Conductor is a complete automation and regression framework for Ixia and third-party test

applications. Tests can be organized into arbitrary test groupings and used for development, QA and

regression testing. Generating a baseline set of run results is critical to establishing consistency across many

versions of test configurations. Some configurations do not intuitively define a test case’s expected results,

only the method required to achieve those goals.





Test Conductor is a powerful tool, with:

 An intuitive, easy-to-learn GUI.

 Vendor-agnostic operation, able to include any test from any test vendor.

 A powerful DUT configuration module that can record and replay scripts from a DUT connected

through any communications medium and protocol, eliminating the need to write scripts. Reusable

scripts are maintained for device setup, device validation, cleanup and background monitoring.

 Run scheduling, using an Outlook®-like calendar. Tests may be run serially or in parallel to optimize

equipment usage.



26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 7 of 15

Keys to Automating Regression Testing









 Extensive use of real-time statistics from multiple sources, including the DUT. Using optional

calculation functions, statistics are used to determine pass/fail status.

 Unattended operation with automatic e-mail test notification.

 Configurable test and regression reports.

 Regression trend analysis. As shown in Figure 6, trends over time can be easily monitored. Baseline

data may be selected from individual runs or averaged over several.

 Test bed management. Assignment of test ports to tests may be quickly mapped to another set of

compatible ports.

 Central repository for all DUT scripts, test bed configurations, tests, and test and regression run results.

 Mercury Quality Center integration. This eliminates the need to segregate network testing plans from

other test plans and provides a tight integration between general-purpose test management processes

and Ixia testing solutions.









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 8 of 15

Keys to Automating Regression Testing









Test Repository

An organization’s domain experts manually construct test configurations as part of the normal QA process.

As they work, these experts also create a centralized repository for all configuration files generated. The

repository’s revision control system is accessed by means of graphical editors that enable other test engineers

to reuse those files. With graphical tools, no programming or scripting is required to group and edit initial

configurations into a regression scenario.



As the number of tests and configurations grow, it essential that additional grouping mechanisms be used to

organize data. Test Conductor includes the ability to assign custom attributes at both test and the regression

level, enabling test engineers to filter configurations according to specific criteria. For example, an engineer

may need to filter a large set of tests to find the tests created at a particular moment in time or created for a

specific revision of a DUT’s software.



As the number of test configurations scales ever larger, test engineers will need to define extra levels of

classification beyond the standard attributes. The better an engineer can identify the attributes of the original

test, the more reusable it will be. For example, a test might be marked as belonging to OSPF and VLAN

groups. One regression could be created and scheduled that includes all OSPF test configurations as part of

a routing protocol compliance effort. Another regression might include the same test in order to verify that a

change made to a VLAN protocol did not adversely affect any configurations.









Figure 7. RFC 2544 latency tests filtered using groups and attributes





26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 9 of 15

Keys to Automating Regression Testing









DUT Configuration Management

Test Conductor’s DUT configuration management (DCM) is a critical component of an end-to-end automation

solution. Procedures that configure and monitor a DUT can be generated from a variety of sources: the

response behavior of a DUT in a live session, predefined templates, or scripting languages. A common

practice is to automate communications between the test scheduler and the DUT’s management interface.



DUT configuration procedures often follow common patterns. Whether performing a specific test or an entire

regression, a DUT first must be configured into a particular testing state, and then returned to its default state.

These procedures are often device specific and a test engineer may not completely understand the syntax. To

be useful, procedures need to be broken down into the smallest steps possible and associated with a

particular action — for example, configuring an interface or enabling a protocol module.



Test Conductor maintains four procedures for use during a test.

 A “setup event” triggers a procedure that initializes the DUT with a default configuration and

prepares it to execute the test case.

 A “validate event’ triggers a procedure that verifies that a test case body executed and that the DUT

responded correctly. This procedure may involve automated inspection of DUT parameters or statistics

such as timers, interfaces states, etc.

 A “cleanup event” triggers a procedure that resets the DUT to its default state.

 In addition to event-driven procedures, the test engineer often has to manually inspect the DUT to

check counters, operational logs, and error states. This information happens asynchronously, so an

automated test scheduler provides custom procedures that execute in the background during

tests/regressions runs. A “background monitoring” procedure serves this purpose.



Good practice dictates that setup and cleanup procedures be defined so that any particular test or set of tests

does not alter the steady state of the DUT, since this could adversely affect its response to other test cases.









Figure 8. RFC 2544 latency test DUT configuration events and corresponding scripts





26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 10 of 15

Keys to Automating Regression Testing









Capture/Replay



A “capture/replay” feature is critically important in the generation of DUT configuration procedures.

Experienced test engineers may be comfortable enough with the configuration syntax of a particular vendor’s

DUT to manually configure a test scenario, but doing so does not record their knowledge for other engineers.

Capture/replay permits any test engineer to configure commands remotely, in real time, using graphical

editing tools. No programming or script writing is necessary. In a capture session, commands recorded for

the duration of a session can be saved directly into a procedure. Once the capture is complete, the tester

adds logic for flow and comments. Configuration steps are saved in a particular DUT’s syntax. As a result,

they can be replayed back to the DUT when the appropriate trigger occurs.









Figure 9. DUT output capture session





For complex, long-running configurations, control logic is directly dependent on a DUT’s configuration state

at key points during the test. The test will execute different control paths and even reboot the DUT in order to

continue. Determining whether a regression run yielded a passed result often requires inspection of the DUT.



Custom, reusable DUT response templates enable a test engineer to graphically mark up a DUT command-

line response in order to manually enter input during a capture session. Custom templates can also be used

to mark up output files generated by a DUT or by a test tool application. Comma separated value (CSV)

format is often used in test applications to log the volumes of traffic and error statistics generated during a



26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 11 of 15

Keys to Automating Regression Testing









regression run. A typical CSV file containing a set of expected values can be marked up to indicate the

statistics of interest in a specific test scenario. This file will then become the template for use in similar

scenarios.



Once the statistics have been marked up, they become globally available for use at both test and regression

levels. In order to have meaning, a test must have an indicated result — pass, fail, or inconclusive. The test

engineer needs to define a formal pass/fail expression based on run statistics, operators, and reference

values or an acceptable level of tolerance.









Figure 10. Custom response template for OSPF timers









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 12 of 15

Keys to Automating Regression Testing









Trend Analysis



Trend analysis report templates contain comprehensive reporting capabilities that enable the display of

regression-run and test-run attributes and statistics in table and chart formats. These templates specify which

data is to be included in a report and how it is rendered. A test engineer can then easily create reports

based on a template's settings.









Figure 11. Test bed catalog









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 13 of 15

Keys to Automating Regression Testing









Version Independence for Better Maintainability



The key to building a flexible and reusable automation system is the ability to generate a series of catalogs

that store the logical and physical settings for all test equipment. These catalogs can then map a logical test

scenario to the physical limitations of available hardware.



A test bed catalog defines a logical view of the test platforms and ports used in tests. A ”client machines”

catalog defines all the automation machines available to execute a test configuration. The test engineer can

automatically reserve hardware to enable parallel execution of tests, thus reducing overall test cycle time.



A “hardware mapping” feature permits a test engineer to substitute or remap the hardware used for a

particular test to another set of equivalent hardware. The engineer can then remove hardware during a test

run or migrate a test from one physical configuration to another without having to modify the test

configuration.









Figure 12. Hardware mapping between two test beds









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 14 of 15

Keys to Automating Regression Testing









Conclusion

An effective automated testing system requires modular, reusable components; easily accessible central

repositories; and graphical editing tools. With these in place, engineers of all skill levels will be able to

consistently apply a test organization’s “best practices” and achieve quality results in less time.









26601 W. Agoura Road · Calabasas, CA 91302 USA · Tel + 1-818-871-1800 · Fax + 1-818-871-1805 · www.ixiacom.com

P/N 915-2104-01 Rev A. 8 November 2007 – Page 15 of 15