Effective Test Management

Document Sample
Effective Test Management Powered By Docstoc
T E S T- M A N A G E M E N T P R O C E S S

Ta b l e o f C o n t e n t s

What is Test Management?          3   Running Tests                              6          Planning Tests                             9

Do it Right or Do it Over —           Managing Issues                            7          Running Tests                             10
the Reasons for Test Management   3
                                      Mercury Interactive TestDirector           8          Managing Issues                           12
Requirements Management           4
                                      Global Web-based Test Management           8          Summary                                   13
Planning Tests                    5
                                      Requirements Management                    8










                                      TESTING EFFORTS.




                                      TEST MANAGEMENT.

Quite often in the world of application development, testing is not seriously considered until program-   project. Without a standard foundation for test

ming has been almost completed. Clearly, this approach to testing is inadequate in light of the           planning, execution and defect tracking, testing
                                                                                                          efforts are non-repeatable, non-reusable and
increasingly high demands for software quality and short release cycles. As a result, the place of
                                                                                                          difficult to measure.
testing in the application lifecycle is beginning to change.
                                                                                                          Better, Faster, Cheaper
                                                                                                          Testing without following any planning and
Fueled by competitive pressures and high costs       The aim of a test-management process is to           design standards can result in the creation of
of downtime, application testing is evolving into    create one central point of control that is acces-   tests, designs and plans that are not repeatable
a multi-stage process with its own methodology,      sible to all members of the testing team. This       and therefore unable to be reused for future
structure, organization and documentation.           central point houses all testing assets and          iterations of the test. Organizations that think a
                                                     provides a clear foundation for the entire testing   testing process is difficult to implement, and
Increasingly more organizations are starting to
                                                     process — from deciding what needs to be             even more challenging to enforce and maintain,
adopt the process where testing takes place in
                                                     tested, to building tests, running scenarios and     should consider the cost of redundant work and
parallel with the application development,
                                                     tracking defects. The test-management method-        accidentally lost and overwritten testing assets.
starting as soon as the project has commenced.
                                                     ology also supports the analysis of test data and    Without a central point of control and clear,
Like much of the development process, the            coverage statistics, to provide a clear picture of   repeatable methodologies, it’s difficult to keep
testing process needs a methodical building-         the application’s accuracy and quality at each       the testing project on track and deliver quality
block approach that includes requirements            point in its lifecycle.                              applications on time with limited resources.
definition, planning, design, execution and
                                                                                                          Daily Builds and Smoke Tests
analysis — to ensure coverage, consistency and       WHAT IS TEST MANAGEMENT?
                                                                                                          The process of releasing a new build every day
reusability of testing assets.                       Test management is a method of organizing
                                                                                                          and then checking it for consistency, function-
                                                     application test assets and artifacts — such as
As soon as the project requirements have been                                                             ality and compatibility is becoming increasingly
                                                     test requirements, test plans, test documenta-
defined, test planning can get underway. The                                                              popular not only in the Web environment, but
                                                     tion, test scripts and test results — to enable
advantage of doing it in parallel with develop-                                                           in any organization that builds complex,
                                                     easy accessibility and reusability. Its aim is to
ment is that the knowledge that was put into                                                              dynamic applications. While smoke tests are
                                                     deliver quality applications in less time. Test
the application design is not lost and can                                                                fairly simple, the sheer number of tests and
                                                     management is firmly rooted in the concepts of
applied to designing the testing strategy.                                                                versions can make the testing process over-
                                                     better organization, collaboration and informa-
                                                                                                          whelming and nearly impossible to manage.
Based on the project objectives, test managers       tion sharing. Planning, designing and running
                                                                                                          A well-defined, systematic approach to testing
can build a master test plan, which will             tests represent a considerable amount of work,
                                                                                                          and a centralized repository for tests, plans and
communicate the testing strategy, priorities and     but these efforts are well rewarded when all
                                                                                                          execution results can significantly increase the
objectives to the rest of the team.                  testing assets are shared by testers, developers
                                                                                                          accuracy of smoke tests and add value to
                                                     and managers alike; preserved when a tester
Based on the master plan, testers can define                                                              having frequent builds.
                                                     leaves the team; and reused throughout the
test requirements, or specific testing goals.
                                                     application’s lifecycle.                             Managing Changing Requirements
Requirements define WHAT needs to be tested
                                                                                                          Complete requirements-based testing is the way
and what objectives — such as performance
                                                     DO IT RIGHT OR DO IT OVER — THE                      to ensure that the finished system meets the
goals — testers have in front of them. Once the      REASONS FOR TEST MANAGEMENT                          user’s needs. In an ideal world, each require-
requirements have been defined, testers can          Despite the claim that test project management
                                                                                                          ment would be tested at least once, and some
start developing the testing procedures              is widely accepted as a common practice, most
                                                                                                          requirements would be tested several times. But
designed to meet/validate those requirements         organizations don’t have a standard process for
                                                                                                          constant time and resource constraints make it
and run the tests.                                   organizing, managing and documenting their
                                                     testing efforts. Often, testing is conducted as an
                                                     ad-hoc activity, which changes with every new

extremely difficult to conduct comprehensive          on a particular product. Still, there are multiple   designing and developing tests; running the
testing of all requirements. As a result, testers     functional areas, builds and baselines; different    tests — both manual and automated, functional
must prioritize the requirements they wish to         platforms and environments; and an endless           and load — and analyzing application defects.
test. And like any other area of application          array of integrations that need to be validated.
                                                                                                           The testing process is not linear, and naturally
design and development, requirements undergo          To keep track of multiple test cases, testers
                                                                                                           it differs depending on each organization’s prac-
frequent revisions and changes, which have to         need a process that will allow them to manage
                                                                                                           tices and methodologies. The underlying
be reflected in the tests.                            multiple projects and clearly define the objec-
                                                                                                           principles of every testing process, however, are
                                                      tives of each one.
Without a test-management process that ties test                                                           the same. This section of the white paper
plans to application functional requirements and      More than Bug Tracking                               delves deeper into each phase of the test-
allows organizations to track requirements            In some IT shops, testing is still regarded          management process and explores what it
changes to test cases (and vice versa), it’s nearly   mostly as a bug-tracking activity. It’s true —       means to have an effective method for organ-
impossible to design tests to verify that the         finding defects is the testing team’s primary        izing, documenting, prioritizing and analyzing
system contains specified functionality.              responsibility. However, there’s much more to        an organization’s entire testing effort.
                                                      the testing process than recording bugs and
Global Testing
                                                      passing them over to R&D. Testing today              REQUIREMENTS MANAGEMENT
Testing, like most other IT departments, is
                                                      focuses on verifying that the application’s          The testing process begins with defining clear,
seeing the effects of globalization. Even small
                                                      design and functionality meet the business           complete requirements that can be tested.
companies often have their testing teams
                                                      constraints and user requirements. To success-       Requirements development and management
distributed throughout the country, or even the
                                                      fully achieve those goals, a testing process         play a critical role in both the development and
world. Cost considerations, higher availability of
                                                      requires clearly defined system specifications       testing of software applications. After all, engi-
qualified testers and better retention of skilled
                                                      and application business rules.                      neers cannot build, and testers will not test,
employees force companies to turn to a “distrib-
                                                                                                           what business analysts were not able to define.
uted” testing model. It’s not uncommon to have        Early in the Lifecycle
multiple development and testing teams, spread        As mentioned earlier, testing no longer fits         To use a simple metaphor: If a team of archi-
around the world, working on the same project.        between the end of development and the begin-        tects builds a house without first consulting the
True, the Internet is making communication            ning of implementation. Early problem                homeowner and thoroughly documenting his/her
easier, but how does a distributed team like this     detection (which is also up to 10 times cheaper      requirements, they may end up with something
ensure that its members are not accidentally          than finding issues at the end of the lifecycle)     completely different than what the owner had
deleting each other’s files or working on dupli-      requires that testing and development begin          envisioned. Building a house without require-
cate defects? Without a clearly defined testing       simultaneously. However, for testing to be           ments is difficult enough, but the problem
process and an easy-to-use, intuitive collabora-      implemented earlier in the lifecycle, testers        becomes greater when dealing with software
tion method, any attempt to set up a                  need a clearly defined set of goals and priorities   applications, which are intangible, complex and
geographically distributed “virtual” testing team     that would help them better understand system        constantly changing.
will likely bring more problems than benefits.        design, requirements and testing objectives.
                                                                                                           Research shows that more than 30 percent of
                                                      Only a well-defined, structured and intuitive
Many Tests, Many Projects                                                                                  all software projects fail before ever being
                                                      test-management process can help ensure that
Effective testing requires a systematic process.                                                           completed, and that five out of eight main
                                                      the testing process meets its goals and
When a testing team is handling multiple proj-                                                             reasons for failure are ambiguous, poorly
                                                      contributes to enhancing application quality.
ects, each with its own cycles and priorities, it’s
easy to lose focus. Almost no organization has        The Test-Management Process
the luxury of dedicating a testing team to one        No matter what the system does, how it’s
development project. At best, a testing team is       written or what platform it’s running on, the
part of the development group that is working         basic principles of test management are the
                                                      same. It begins with gathering and documenting
                                                      the testing requirements; continues through

worded and badly defined requirements. All too      realistic given time-to-market and other             PLANNING TESTS
often, requirements are neglected in the testing    pressures. Requirements-based testing is one         Comprehensive planning is critical, even with
effort, leading to a chaotic process in which       way to help you prioritize the testing effort.       short testing cycles. Planning tests and
testers fix what they can and accept that certain   Requirements can be grouped according to how         designing applications simultaneously will
functionality will not be verified.                 critical they are to mission success. The ones       ensure a complete set of tests that covers each
                                                    that affect the core functionality of the system     function the system is designed to perform. If
Quality can be achieved only by testing the
                                                    must be extensively tested, while the not-so-        test planning is not addressed until later in the
system against its requirements. Although unit
                                                    critical requirements can be covered by minimal      application lifecycle, much of the design knowl-
testing of individual components may be suffi-
                                                    testing effort or be tested later in a lifecycle.    edge will be lost, and testers will have to return
cient during the development process, only
                                                                                                         to the analysis stage in order to try to recreate
testing the entire system against its require-      For example, if the application under test is a
                                                                                                         what has already been done before.
ments will ensure that the application functions    new release of a mature product with only a
as expected.                                        few functional changes, the testing effort           During the test planning phase, the testing
                                                    should be concentrated on the modified or            team identifies test-creation standards and
Of the many types of requirements, functional
                                                    enhanced areas. These changes are usually            guidelines; selects hardware and software for
and performance requirements are the ones that
                                                    well-documented in the development and/or            the test environment; assigns roles and respon-
most often apply to the testing process.
                                                    marketing specification documents, and testing       sibilities; defines the test schedule; and sets
Functional requirements help validate that the
                                                    requirements can be derived directly from them.      procedures for running, controlling and meas-
system contains specified functionality and
                                                                                                         uring the testing process.
should be derived directly from the use cases       Recognizing the importance of requirements-
that developers use to build the system.            based development and testing, the testing           The test plan itself contains test designs, test
                                                    industry began marketing a variety of tools          procedures, naming conventions and other test
Performance requirements cover any perform-
                                                    designed to aid the creation, traceability and       attributes. The planning phase is used to define
ance standards and specifications identified by
                                                    management of application requirements. Some         which tests need to be performed, how these
the project team, such as transaction response
                                                    of these enterprise requirements tools are           tests must be executed, and which risks and
times or maximum numbers of users. These
                                                    sophisticated and complex. Others offer the          contingencies require planning.
requirements — also referred to as service-level
                                                    basic functionality designed to assist in
objectives (SLOs) — are aimed at ensuring that                                                           Set the Ground Rules:
                                                    creating, storing and documenting require-
the system can scale to the expected number of                                                           Testers don’t need overly bureaucratic processes
                                                    ments. The key to selecting the appropriate tool
users and provide a positive user experience.                                                            to slow them down, but ambiguities and lack of
                                                    is the functionality. Most organizations are still
                                                                                                         clearly defined procedures can make testing a
Both functional and performance requirements        using word-processing tools or spreadsheets to
                                                                                                         bottleneck for product deliverables. This stage
are designed to give the testing team a clear,      track requirements; however, commercial tools
                                                                                                         can be used for setting the ground rules for
concise and functional blueprint with which to      designed for requirements management are a
                                                                                                         keeping test logs and documentation, assigning
develop test cases. Most software testing           better choice for organizations that want to
                                                                                                         roles within the team, agreeing on the naming
managers will agree that it's impossible to test    create a solid, flexible requirements-based
                                                                                                         convention for tests and defects, and defining
everything — even in a rather simple system.        testing process.
                                                                                                         the procedure for tracking progress against the
To test each feature and functional permutation,
                                                    Requirements-based testing is the way to             project goals.
you would potentially need to develop thou-
                                                    keep the testing effort on track — even when
sands if not millions of test cases, which is not
                                                    priorities are shifting, resources become tight or
                                                    time for testing runs out. Requirements-based
                                                    testing is the way to measure quality against the
                                                    end-user needs.

Set Up Test Environment:                              Communicate the Test Plan:                           procedure that specifies the tasks that need to
A test environment needs to be set up to              Once the test cases have been created, it’s          be performed at each step of manual testing.
support all testing activities. All issues that       helpful to create a master test plan and             For both manual and automated tests, testers
have long lead-times need to be considered as         communicate it to the rest of the organization       need to keep a complete history of all test runs,
early in the testing process as possible. The         — specifically project leaders, developers and       creating audit trails to help trace history of tests
planning phase is the time to get done with any       marketing — letting them know the areas on           and test runs.
hardware and software procurements, network           which the testing group will be working. This
                                                                                                           Create Test Sets:
installation, test environment design and             way, more people in the organization have the
                                                                                                           A popular way to manage multiple tests is
making any special arrangements that may be           visibility into the project and can add their
                                                                                                           grouping them into test sets, according to the
necessary during the actual testing phase.            input, questions or comments before the actual
                                                                                                           business process, environment or feature. For
                                                      testing begins.
Define Test Procedures:                                                                                    example, all tests designed to verify the login
During the planning stage, testers determine if                                                            process can be grouped together under the
                                                      RUNNING TESTS
tests are going to be manual or automated. If                                                              “Login” test set. Individual tests (both manual
                                                      After the test design and development issues
the test is automated, testers need to under-                                                              and automated) can be assigned to test sets to
                                                      have been addressed, the testing team is ready
stand which tools will be used for automation,                                                             help testers ensure thorough coverage of each
                                                      to start running tests. To test the system as a
and estimate what techniques and skills will be                                                            functional area of the application.
                                                      whole, testers need to perform various types of
needed to effectively use these tools.
                                                      testing — functional, regression, load, unit and     Set Execution Logic:
Develop Test Design:                                  integration — each with its own set of require-      In order to verify application functionality and
Test design may be represented as a sequence          ments, schedules and procedures.                     usability, tests have to realistically emulate the
of steps that need to be performed to execute a                                                            end-user behavior. To achieve this, test execu-
                                                      The test environment has been set up back in
test (in case of a manual test), or a collection of                                                        tion should follow the predefined logic, such as
                                                      the planning phase, so at this point testers can
algorithms and code to support more sophisti-                                                              running certain tests after other tests have
                                                      view all available hardware and choose which
cated, complex tests. During test planning,                                                                passed, failed or have completed. Consider this
                                                      tests will run on which machines. Most applica-
testers create a detailed description of each                                                              example: A user logs into the system, enters a
                                                      tions must be tested on different operating
test, as well as identify what kind of data or                                                             new order and then exits the system. To
                                                      systems, different browser versions or other
ranges of data are required for the tests.                                                                 emulate this simple business process, it makes
                                                      configurations. During the test-run phase,
                                                                                                           sense to run the tests following the exact same
Map Test Data:                                        testers can set up groups of machines to most
                                                                                                           sequence: login, insert order, logout. The execu-
Test-data requirements need to be mapped              efficiently use their lab resources.
                                                                                                           tion logic rules should be set prior to executing
against test procedures. The testing team must
                                                      Scheduling automated tests is another way to         the actual tests.
understand what types of data need to be
                                                      optimize the use of lab resources, as well as save
obtained to support each type of test, and how                                                             Manage Test Resources:
                                                      testers time by running multiple tests at the
this data could be obtained or generated.                                                                  After the test environment has been set up and
                                                      same time across multiple machines on the
                                                                                                           configured, testers define what tests or groups
Design Test Architecture:                             network. Tests can be scheduled to run unat-
                                                                                                           of tests should be run on which machines.
Test architecture helps testers get a clear picture   tended, overnight or when the system is in least
                                                                                                           Hardware can be set up to help test the on a
of the test building blocks and assists in devel-     demand for other tasks. It also makes the scripts
                                                                                                           particular operating system, platform or browser.
oping the actual tests. Test architecture helps       much more reusable and easier to maintain by
                                                                                                           Additionally, managing test lab resources by
plan for data dependencies, map the workflow          allowing the ability to create more modular tests
                                                                                                           assigning tests to individual machines helps
between tests and identify common scripts that        and schedule them in certain sequence.
                                                                                                           ensure that hardware and network resources are
can potentially be reused for future testing.
                                                      Having an organized, documented process does         being used most efficiently and effectively.
                                                      not only help with automated tests, it also helps
                                                      make manual test runs more accurate by
                                                      providing testers with the clearly defined

Run Manual Tests:                                       The key to making a good defect-reporting and          Cosmetic and inconsistency defects are the
Running manual tests involves manually inter-           resolution process is setting up the defect work-      easiest to report and handle. Although they
acting with an application, then recording              flow and assigning permission rules. This way, a       make it more difficult to use the system, they
actual results against expected outcomes. While         tester can clearly define how a lifecycle of a         don’t affect system functionality. The defects
some manual tests may be routine, they are an           defect should progress and who has the                 that result in loss of functionality are more
essential part of the testing procedure, allowing       authority to open a new defect, who can change         severe and therefore more urgently need to be
the tester to verify functionality and conditions       its status to fixed, and under what conditions         fixed. Defects that cause the system to crash, or
that automated tools are unable to handle.              the defect can be officially closed. It’s also crit-   worse — lose data — are commonly referred to
                                                        ical to maintain a complete history and audit          as “show-stoppers.” These must be documented
Schedule Automated Test Runs:
                                                        trail throughout the defect lifecycle. Extra time      as thoroughly as possible, and must be fixed
After the decision to automate has been made
                                                        spent on documenting the defect and its history        before the system goes live.
and tests have been developed and assigned to
                                                        is often well rewarded by easier analysis, shorter
the host machines, the testing team needs to                                                                   Establish the Reporting Procedure:
                                                        resolution times and better application quality.
create an execution schedule. Scheduling is                                                                    The key to making a good defect report is
another way to avoid conflicts with hardware            Analyzing defects is what essentially helps            supplying developers with as much information
and system resources — tests can be scheduled           managers make the go/no-go decisions about             as they will need to reproduce and fix the
to run unattended, overnight or when the                application deployment. By analyzing the defect        problem. A defect report generally includes a
system is in least demand for other tasks.              statistics, a tester can take a snapshot of the        summary and detailed description of the
                                                        application under test and tell exactly how            problem, version and system configuration infor-
Analyze Test-Run Results:
                                                        many defects it currently has, their status,           mation, list of steps needed to reproduce the
During the test-execution phase, testers will
                                                        severity, priority, age, etc. By giving different      problem, and any relevant information and
uncover application inconsistencies, broken func-
                                                        team members access to defect information, a           attachments that will help illustrate the issue. It
tionality, missing features and other problems
                                                        tester can greatly improve communication in the        is critical that a defect report not only provides
commonly referred to as “bugs” or “defects.”
                                                        organization and get everyone on the same page         information describing the bug, but also includes
The next step is to view the list of all failed tests
                                                        as to the current status of the application.           all data necessary to help fix the problem.
and determine what caused the test to fail. If the
tester determines that the test failed due to an        Agree on the Naming Convention:                        Set the Permission Rules:
application defect, this defect has to be reported      The key to effective defect management is              For each step in the test-management process,
into the defect tracking system for further investi-    communication among different parties                  there must be different access privileges. This
gation, correction and re-test.                         involved in the process. Before reporting mech-        is especially true in the defect-management
                                                        anisms can be put into place, the testing team         stage, since the defect information helps the
MANAGING ISSUES                                         needs to set the ground rules, such as define          management make a “go/no go” decision on the
Managing or “tracking” issues and defects is a          the severity of the bugs and agree on what             application release. Before the testing begins,
critical step in the testing process. As today’s        information must be included in the defect             testers must decide on which members of the
systems become more complex and mission-                report. For example, many testing organizations        team will have permission to report, open or
critical, so does the severity of the defects. A        categorize defects as follows:                         re-open defects, who can adjust the status, and
well-defined method for defect management will                                                                 who has the authority to determine that the bug
                                                        • Cosmetic/UI
benefit not just the testing team. Developers,                                                                 has been fixed and is now closed.
managers, customer support, QA and even Beta            • Inconsistency

customers can effectively contribute to the             • Loss of functionality
testing process by having access to an open,
                                                        • System crash
easy-to-use, functional defect-tracking system.
                                                        • Loss of data

Establish the Process Flow:                          MERCURY INTERACTIVE TESTDIRECTOR                     REQUIREMENTS MANAGEMENT
After the defect has been found, it is submitted     The test-management process is the main prin-        TestDirector’s Requirements Manager links
into the defect repository. The next step is to      ciple behind Mercury Interactive’s TestDirector      test cases to testing requirements, ensuring
have it reviewed by a developer who determines       — the industry’s first Web-enabled test-manage-      traceability throughout the testing process.
if a reported issue is indeed a defect, and if it    ment tool that combines the entire                   TestDirector enables the user to easily see what
is — assigns it a “new” status. Since few devel-     test-management process in one powerful, scal-       percentage of the application functional require-
opment organizations have the bandwidth to           able and flexible solution. TestDirector’s four      ments are covered by tests, how many of these
repair all known defects, a developer or project     modules — Requirements Management, Test              tests have been run, and how many have passed
manager must prioritize. If R&D is under pres-       Planning, Test Lab and Defect Manager — are          or failed.
sure to release the next version of the              designed with the testing process flow in mind.
                                                                                                          The process of developing testing requirements
application, a project manager may decide to         The seamless integration between these
                                                                                                          starts with reviewing all available documenta-
only fix the “show-stopper” level defects, and       modules enables smooth information flow
                                                                                                          tion on the application under test, such as
leave the rest for after the release cycle.          among different stages of the testing process.
                                                                                                          marketing and business requirements docu-
Re-Test Repairs:                                                                                          ments, system requirements specifications and
                                                     GLOBAL WEB-BASED TEST
Whatever fixes or changes have been made to          MANAGEMENT                                           design documents. These documents are used
repair a known defect, the application needs to      TestDirector is the global test-management tool.     to obtain a thorough understanding of the appli-
be re-tested to verify that the changes have         By being completely Web-enabled, TestDirector        cation under test and determine the testing
taken effect and that the fix did not introduce      supports communication and collaboration             scope — test goals, objectives and strategies.
additional problems and unexpected “side-            among distributed testing teams — whether            Based on the testing scope and goals, QA
effects.” If the defect does not appear during       they are located in different parts of the world     managers can start developing requirements,
the re-testing phase, its status can be changed      or separated by organizational boundaries.           log them into the TestDirector requirements
to “closed.” If the problem persists and/or the
                                                     In today’s organizations, different groups are       tree and assign responsibilities for specific
fix has introduced additional problems, the
                                                     involved in the testing process — managers,          areas. TestDirector’s requirements tree is a
defect is reopened and the cycle is repeated.
                                                     developers, customer support and even                graphical representation of the organization’s
Analyze Defects:                                     customers. By simply using a browser, all these      specific requirements and displays the hierar-
Analyzing defects is the most critical part of the   groups can easily access testing information.        chical relationship between the different
defect-tracking process. It allows testers to take   In addition, TestDirector’s ability to configure     requirements. Each requirement in the tree is
a snapshot of the application under test and         user groups and set up permissions helps             described in detail, such as the reviewed status,
view the number of known defects, their status,      maintain access privileges and preserve infor-       priority and creation date, and includes any
severity, priority, age, etc. Based on defect        mation integrity.                                    relevant attachments.
analysis, management is able to make an
                                                     Another tremendous benefit of a Web-based tool       TestDirector’s Requirements Manager provides
informed decision as to whether the application
                                                     is its ability to keep everyone in synch by          testers with two ways to link their requirements
is ready to be deployed.
                                                     instantly upgrading the version of the tool or       to tests. Testers can automatically generate tests

                                                     installing new modules. As a result, users no        based on application requirements. Or, if they

                                                     longer need to be taken off-line while their test-   have an existing test plan, they can link require-

                                                     management tool is being upgraded. They can          ments to tests that are in turn associated with

                                                     simply refresh a browser and automatically be        defects. In this way, testers can keep track of

                                                     in synch with the rest of the organization.

                                                                                                                       PLANNING TESTS
                                                                                                                       The test-plan tree in TestDirector is a graphical
                                                                                                                       representation of the organization’s test plan. It
                                                                                                                       is a hierarchical list of tests organized according
                                                                                                                       to topic and describes the set of tests that must
                                                                                                                       be implemented to meet the quality require-
                                                                                                                       ments defined in the previous steps. In
                                                                                                                       TestDirector, testers can also associate a test
                                                                                                                       with specific defects. This is useful, for
                                                                                                                       example, when a new test is created specifically
                                                                                                                       for a known defect. By creating an association,
                                                                                                                       testers can determine whether the test should
                                                                                                                       be run based on the status of the defect.

                                                                                                                       In TestDirector, testers can link each test in the

TestDirector’s Requirements Manager links test cases to testing requirements, ensuring traceability
                                                                                                                       test-plan tree with a requirement in the require-
throughout the testing process.                                                                                        ments tree. By defining requirements coverage
                                                                                                                       for a test, testers can keep track of the relation-
                                                                                                                       ship between the tests in their test plan and
their testing needs at all stages of the testing                 In TestDirector, all filters, reports and graphs      their original testing requirements.
process. If a testing requirement changes, they                  can be saved as “Favorites” — public or
                                                                                                                       For each test, TestDirector allows testers to
can immediately identify which tests and defects                 private. Only by the person who creates these
                                                                                                                       create test steps describing which operations
are affected, and who is responsible. They can                   private favorite views can access them. Public
                                                                                                                       need to be performed, what specific areas must
group and sort requirements in the tree, monitor                 favorites, however, can be saved for use by the
                                                                                                                       be checked, as well as the expected results.
task allocation and progress of requirements, and                entire team. For example, for the weekly
                                                                                                                       After testers define the test steps, they can
generate detailed reports and graphs.                            meeting, a manager needs to have updated
                                                                                                                       decide whether to perform the test manually or
                                                                 information on the status of all testing require-
At any stage of the testing process, TestDirector                                                                      to automate it.
                                                                 ments for a particular application. She only
will help testers generate quick graphs and
                                                                 needs to create the report once and save the          In addition to supporting both manual and auto-
reports to gain accurate status information on the
                                                                 filter under Favorites. TestDirector will automati-   mated tests, TestDirector supports the migration
application under test. For example, once the
                                                                 cally update her view with new information, and       from manual to automated tests. If testers
requirements-creation stage has been completed,
                                                                 the manager will have an updated status of all        choose to automate a certain test, TestDirector
testers can quickly extract information on what
                                                                 requirements every week for her meetings.             will create a template for the specific type of
percentage of requirements are covered by tests,
                                                                                                                       automated test based on the design steps.
what the status is of those tests, how many                      TestDirector also helps organizations maintain
                                                                                                                       Testers only need to use an automated testing
requirements were rejected in the review process                 an audit trail by providing the history of any
                                                                                                                       tool (such as WinRunner) to record the business
and have to be repaired, as well as which                        changes made to a particular requirement. This
                                                                                                                       process and complete the test.
requirements have “urgent” priority and need to                  helps preserve information integrity and ensure
be validated as soon as possible.                                that testers can trace every event throughout
                                                                 the requirement lifecycle — from initial creation
                                                                 through any changes to its status, priority or
                                                                 test coverage.

Before moving to the execution phase, it’s                          Favorites and ensure that this information is      RUNNING TESTS
essential to review the test plan to determine                      available at any time.                             TestDirector’s Test Lab helps the testing team
how well it meets the goals that testers defined                                                                       manage the scheduling and running of many
                                                                    If an organization already has planning informa-
at the start of the testing process. They can                                                                          different types of tests. To help organize the
                                                                    tion stored in a word-processing tool, such as
analyze their test plan by generating                                                                                  tests that need to be run, TestDirector supports
                                                                    Microsoft Word or Excel, testers can reuse this
TestDirector reports and graphs. For example,                                                                          the concept of a “test set.” A test set is a group
                                                                    information and import it into TestDirector,
they can create a report that displays design                                                                          of tests in a TestDirector project database that
                                                                    preserving their investment and eliminating the
step data for each test in a test-plan tree. They                                                                      is designed to achieve specific testing goals.
                                                                    need for redundant planning efforts.
could use this report to help determine their                                                                          Test sets may include all tests that validate
                                                                    Additionally, TestDirector allows testers to
test design priorities.                                                                                                specific functionality (such as the login process)
                                                                    create attachments so they can add any infor-
                                                                                                                       or that verify the application works on a partic-
As with reports and graphs in the requirements                      mation — such as design documents or feature
                                                                                                                       ular version of the browser. In TestDirector, tests
tab, testers can save their test plan views as                      specifications — to a specific test.
                                                                                                                       can be added to the test set directly from the
                                                                                                                       planning tree by simply dragging and dropping.
                                                                                                                       Testers also can create a filter inside the plan-
                                                                                                                       ning tree and export into a test set all tests that
                                                                                                                       satisfy the specified criteria.

                                                                                                                       In TestDirector, testers can define the execution
                                                                                                                       conditions. TestDirector enables testers to
                                                                                                                       control the execution of automated tests in a
                                                                                                                       test set. They can set conditions and schedule
                                                                                                                       the date and time for executing their automated
                                                                                                                       tests. They also can set the sequence in which
                                                                                                                       to execute the tests.

                                                                                                                       For example, to verify a simple business process
                                                                                                                       in which the user logs into the system, creates
                                                                                                                       a reservation, enters payment information and
                                                                                                                       logs out, it makes sense to create four modular
                                                                                                                       tests: “Login,” “Create Reservation,” “Insert
                                                                                                                       Payment” and “Logout.” To realistically emulate
                                                                                                                       this simple business process, testers can
                                                                                                                       arrange the four tests sequentially, beginning
                                                                                                                       with “Login,” followed by “Create Reservation,”
The test plan tree in TestDirector is a graphical representation of the organization’s test plan.                      “Insert Payment” and “Logout.” In TestDirector,
                                                                                                                       testers also can define the execution conditions.
                                                                                                                       In this example, the “Create Reservation” test
                                                                                                                       can be set to run only after the “Login” test has
                                                                                                                       passed. The “Logout” test, on the other hand,
                                                                                                                       can be run regardless of whether the previous
                                                                                                                       test passes or fails. So, the tester can set the
                                                                                                                       condition to run “Logout” after “Insert
                                                                                                                       Payment” has finished, regardless of the result.


• Supports the Entire Testing Process: TestDirector         • Integrates with Third-Party Applications: Whether a                reported into TestDirector’s central repository,
   incorporates all aspects of the testing process —           tester uses an industry-standard configuration                    creating an accurate audit trail for analysis.
   requirements management, planning, scheduling,              management solution, Microsoft Office or a home-
                                                                                                                              • Facilitates a Consistent, Repetitive Testing Process:
   running tests, issue management and project status          grown defect-management tool, any of these
                                                                                                                                 By providing a central repository for all testing
   analysis — into a single browser-based application.         applications can be integrated into TestDirector.
                                                                                                                                 assets, TestDirector facilitates the adoption of a
                                                               Through the open API, TestDirector preserves the
• Provides Anytime, Anywhere Access to Testing Assets:                                                                           more consistent testing process, which can be
                                                               users’ investment into their existing solutions and
   Using TestDirector’s Web interface, testers, devel-                                                                           repeated throughout the application lifecycle or
                                                               enables the users to create an end-to-end lifecycle-
   opers and business analysts can participate in and                                                                            shared across multiple applications or lines of
                                                               management solution.
   contribute to the testing process by collaborating                                                                            business.
   across geographic and organizational boundaries.         • Manages Manual and Automated Tests: TestDirector
                                                                                                                              • Provides Analysis and Decision-Support Tools:
                                                               stores and runs both manual and automated tests,
• Provides Traceability Throughout the Testing Process:                                                                          TestDirector’s integrated graphs and reports help
                                                               and can help jumpstart an automation project by
   TestDirector links requirements to test cases, and                                                                            analyze application readiness at any point in the
                                                               converting manual tests to automated test scripts.
   test cases to issues to ensure traceability                                                                                   testing process. Using information about require-
   throughout the testing cycle. When requirement           • Accelerates Testing Cycles: TestDirector’s TestLab                 ments coverage, planning progress, run schedules
   changes or the defect is fixed, the tester is notified      manager accelerates the test execution cycles by                  or defect statistics, managers are able to make
   of the change.                                              scheduling and running tests automatically —                      informed decisions on whether the application is
                                                               unattended, even overnight. The results are                       ready to go live.

In TestDirector, testers can run tests locally or
remotely, on any available machine on the
network. Through the Host Manager, they can
define which machines will be available for
running tests and group them by task (func-
tional or load testing), operating system
(Windows or UNIX), browser type and version
(Netscape or Internet Explorer), or by any other
specification or configuration.

TestDirector offers tight integration with
Mercury Interactive’s automated functional and
load-testing tools. It can also be configured
through its open API to run tests using a
third-party testing tool. With TestDirector,
testers can schedule their automated tests to
run unattended, either overnight or when the
test lab machines are in least demand for other
tasks. Using its scheduling mechanism,
TestDirector will invoke the automated testing
tool, run tests and report results back into the
                                                            In TestDirector, tests can be run locally or remotely, on any available machine on the network.
central repository.

MANAGING ISSUES                                                 and verifying the fix — as well as set permis-          TestDirector’s customizable analysis tools
Information about issues or defects is critical                 sion rules and access privileges for any member         can be shared across the organization.

for testers who must determine the status of an                 of the organization.                                    Based on one’s role in the organization, he/she

application and decide whether it is ready for                                                                          can be restricted from viewing certain records in
                                                                Using TestDirector’s many customization
deployment. TestDirector’s Defect Manager is a                                                                          the defect-tracking process. For example, some
                                                                options, organizations can set up the workflow
complete system for logging, tracking, managing                                                                         members of the development team can be
                                                                rules that are most appropriate for their
and analyzing application defects. It allows                                                                            restricted from viewing defects other than the
                                                                process. For example, a defect must start its
different types of users — testers, project                                                                             ones that have been assigned to them.
                                                                cycle with the status “new.” It could not be
managers, developers, beta customers and                                                                                Similarly, beta customers who have access to
                                                                closed without first being reviewed by the QA
many others — to contribute to the testing                                                                              the defect database are not able to view the
                                                                manager, after which it is transferred to R&D. A
process by entering defects directly into                                                                               “target fix date” for any defect.
                                                                member of the R&D team either rejects a defect
TestDirector’s database.                                                                                                Similar to using attachments in other modules,
                                                                or fixes the problem and assigns the new status
An effective defect-tracking process is firmly                  — “fixed.” But the defect cannot be closed              testers can attach information to a defect, such

rooted in the concepts of well-defined workflow                 until the project manager reviews the fixes and         as a description file or a snapshot of the appli-

and permission rules. In TestDirector, testers                  changes the status to “closed.” By setting the          cation under test, to help illustrate a problem.

can define exactly how the defect should                        workflow and permission rules for different             To further help R&D reproduce the issue, testers

progress through its lifecycle — from initial                   members of the team, testers can configure              can attach information on all system compo-

problem detection through solving the problem                   TestDirector to reflect their organization’s            nents, such as memory, operating system or

                                                                process flow and organizational requirements.           color settings. Attached as a text file, all this
                                                                                                                        data can help reproduce the problem and accel-
                                                                                                                        erate problem resolution.

                                                                                                                        The filters, reports, graphs and Favorites in the
                                                                                                                        defect-tracking module can be used to help
                                                                                                                        assess whether an application is ready to be
                                                                                                                        deployed. Are testers finding more defects than
                                                                                                                        they are fixing? Are developers overloaded with
                                                                                                                        too many urgent bugs assigned to them? Are
                                                                                                                        there more defects in the current version than
                                                                                                                        in the previous release? All this information is
                                                                                                                        available through TestDirector’s customizable
                                                                                                                        analysis tools, and can be shared across the

TestDirector’s Defect Manager is a complete system for logging, tracking, managing and analyzing application defects.

The application-testing process is unique for
every organization. However, many testing teams
follow a similar methodology that includes
identifying the functional requirements, creating
a test plan, running tests and tracking applica-
tion defects. This approach to testing requires a
powerful tool that can promote communication
and collaboration between teams, while adding
organization and structure to the testing

Mercury Interactive’s TestDirector pioneered
the concept of global test management. By
integrating requirements management with test
planning, test scheduling, test execution and
defect tracking in a single Web-based applica-
tion, TestDirector streamlines and accelerates
the testing process.

Mercury Interactive is the global leader in business technology optimization (BTO). Our Optane suite of testing, tuning and
performance-management solutions enables companies to unlock the value of their information technology (IT) invest-
ments by optimizing business and technology performance to meet business requirements. With Mercury Interactive,
customers can measure the quality of their IT-enabled business processes, maximize technology and business performance
at every stage of the application lifecycle, and manage their IT operations for continuous optimization throughout the
lifecycle. Our leading-edge BTO software and services are complemented by technologies and services from our global
business partners, and are used by more than 30,000 customers — including 75 percent of Fortune 500 companies — to
improve quality, reduce costs and align IT with business goals.

Optane, TestDirector, Mercury Interactive and the Mercury Interactive logo are registered trademarks of Mercury Interactive Corporation. All other company, brand
and product names are marks of their respective holders. © 2003 Mercury Interactive Corporation. Patents pending. All rights reserved. WP-0461-0203.

1325 Borregas Avenue, Sunnyvale, CA 94089 U.S.A. Phone: 408-822-5200 or 800-837-8911

Shared By: