Testcomplete 5 - PowerPoint by dwc20915


More Info
									                  Test Automation
                 Let’s talk business

Igor Gershovich
Connected Testing, Inc.
 Test Automation Myths
 Return on Investment (ROI)
 Frameworks
Test Automation Myths
Myth #1 - Test Automation is
simple, that every tester can do it
   This myth is promoted by the tool sales people.
    They are trying to promote the following test
    automation process:
     Record  the script
     Enhance the script by adding functions and data
     Run the scripts
     Report results
   Under the influence of this myth the QA
    manager can proudly report: All our testers are
    developing test automation.
Reality - Test automation is a
software development task
   Automation should be designed, developed and
   You need to have some kind of a
    programming background to implement test
    automation. Test Automation is not as complex
    as C++/C#/Java development.
   Test automation standards should be developed
   Automated test components are assets that
    should be treated like application source code
Myth #2 – Commercial test tools
are expensive
   Under the influence of this myth some
    companies, especially the small ones:
     Tryto develop their own test automation tools
     Use scripting languages like Perl and Ruby
     Use shareware test tools
     Do not consider test automation at all
Reality – Commercial tools are
   Per seat license for the most expensive automation tool
    is $8K
   This tool will be used for 5 years.
   Maintenance/Support fees are 20% of tool cost or
    $1,800 per year
   The cost of this tool is $8K/5+$1,800 = $3,100 per year
   The automation developer cost with overhead is $100K
    per year
   The cost of this tool is just 3% of the person who uses it,
    but productivity gain can be very significant
Commercial Tool Benefits:
   Customer support. Many of the open source tools come and go with
    little to no support
   Most commercial tools are constantly being updated as technologies
   Most commercial tools usually have more functionality (QTP can test
    various GUI applications: Web, .Net, Java, VB, C/C++,
    PowerBuilder, etc. vs. WATIR – Web only)
   Commercial tools usually have a large community of users, which
    translates into better availability of qualified resources
   Commercial tools require less advanced programming
   More test automation frameworks are available for commercial tools
   Commercial tools are integrated with Test Management tools which
    makes reporting and execution much simpler.
Over 300 Test Tools are available
 Load/Performance tools – 54
 Web Functional/Regression – 60
 Java Test tools - 48
 Other Web tools – 76

Which tool is right for you?
Absolutely Unscientific research:
Employer demand by tool expertise
           (May, 2008)
 Dice search results across US (30 days)
Tool           Search string                      Matches

QTP            Qtp OR quicktest OR "quick test“   613
Functional     "rational robot" OR "functional    118
Tester         tester”
SilkTest       silktest OR "silk test”            86
TestComplete   testcomplete OR "test complete”    26
TestPartner    testpartner OR "test partner“      19
VSTE           VSTE AND test                      45
WATIR          watir                              26
Selenium       Selenium                           108
                    Demand per tool
           Open Source
  Microsoft WATIR Open Source
   VSTE        2%        Selenium
     4%                    10%
                                           Functional Tester
                                  HP QTP   SilkTest
                                   60%     TestComplete
     2%                                    VSTE
    Borland                                WATIR
   SilkTest                                Selenium
      8%        Tester
Excerpt from IDC Report:

“Worldwide Automated
Software Quality 2007-2011
Forecast and 2006 Vendor
Think of yourself like a business
   Facts:
     Average IT job tenure is about 2 years
     Competitive intensity is rising
   What you should do:
     Investyour time/money in learning mainstream
     Create a niche for yourself - add Configuration
      Management, Performance Testing or Project
      Management to your skillset
     Market yourself
Return On Investment (ROI)
Classic ROI Calculation


Automation Cost = Price Of HW + Price of SW + Development Cost +
  Maintenance Cost + Execution Cost

Manual Testing Cost = Development Cost + Maintenance Cost +
  Execution Cost

ROI = (Manual Testing Cost - Automation Cost)/Automation Cost

Looks right, Doesn’t it?
Problems with Classic ROI Calculation

   You can’t compare Automated Testing
    and Manual Testing. They are not the
    same and they provide different
    information about the AUT.
   You can’t compare cost of multiple
    execution of automated tests vs. manual
    tests. You would never dream of
    executing that many test cases manually.
Automation Real ROI
 ROI   value IS NOT the value of Automation
  vs. Cost of executing these tests manually
 Automation ROI value IS the benefit of
  this type of testing, and it can be:
   Reducing Time to Market
   Increased Test Efficiency (Productivity)
   Increased Test Effectiveness
Reduced Time to Market

 Can get a greater market share
 Makes people available to work on
  other projects
 Higher margins, if no competitive
  products are currently available
Productivity and Effectiveness

   More testing gets done faster, increasing the odds of
    finding defects
   Defects found early have better chances of being fixed
   Manual Testers can concentrate on clever ways to
    finding defects, instead of typing test inputs and verify
   About 7% of bug fixes create new bugs, sometimes in
    already tested parts of the system. With automation you
    can rerun tests for those modules. This almost never
    happens when testing is done manually.
ROI summary
 Each project requires different types of
  automation - there is often no easy
  formula available to calculate ROI
 Performing ROI calculation can help to
  determine upfront what type of
  automation, what level of skills, what tools
  will be required.
Automation Frameworks
Test Automation Framework
 A Test Automation Framework is a set
  of assumptions, concepts and tools that
  provide support for Automated Software
 A correctly implemented Test Automation
  Framework can further improve ROI by
  reducing the development and
  maintenance costs.
Types of Test Automation
 Modular
 Data-Driven
 Keyword–Driven
 Model-Based
Modular framework
   The Modular framework is the natural
    progression derived from Record-and-Playback
   The modular framework seeks to minimize
    repetition of code by grouping similar actions
    into “modules” (e.g.: login)
   Test Data is in a script or in an internal table

This is what tool sales people are promoting.
Data-Driven/Keyword Driven
 These frameworks are similar in that the
  data is separated from the test script
 The script is just a "driver" or delivery
  mechanism for the data.
 The difference:
     In keyword-driven testing, the navigation data
      and test data are contained in the data source
     In data-driven testing, only test data is
      contained in the data source.
Model-Based testing

 Model-based testing is software testing
  in which test cases are derived in whole or
  in part from a model that describes some
  (usually functional) aspects of the system.
 Model-based testing for complex software
  systems is still an evolving field.
We’ll be concentrating on
Keyword-Driven testing since it
is the most beneficial framework
for large-scale test automation.
Advantages of Keyword-Driven
Test automation
   This Framework addresses the most common problem
    with test automation:
    Automation Engineers do not have domain knowledge
    and the End Users (Subject Matter Experts/Test
    Engineers) usually do not have automation expertise.
   When properly implemented and maintained, it presents
    a superior ROI because each business event is
    designed, automated and maintained as a discrete
   Keywords can then be used to design test cases, but the
    design and automation overhead for the keyword has
    already been paid.
Advantages of Keyword-Driven
Test automation
   Reduced the cost and time spent maintaining
    and updating tests
   The modular structure of keyword-driven testing
    means that new tests can easily be created from
    pre-existing modules
   The test team is capable of entirely automating
    tests, even without programming knowledge
   Can be easily modified to use with different test
   Reusability across different projects
Classic Keyword-Driven
Object                 Action       Data

Textfield (username)   Enter Text   <username>
Recent Success with Keyword-Driven
   Estimate using Modular Framework: 1 year
   Estimate with Keyword-Driven Framework: 6 months

Project was completed in 4 months by 1 person, 2 month
  ahead of schedule.
This was the first project for the automation developer
  utilizing the Keyword-Driven Framework.

170 components were developed (1 component per Web
109 Regression Test cases were automated.
Example of Test Data for Keyword-Driven test
automation developed by Connected Testing Inc.
for TestComplete and QTP
   “Everything you want to know about Test Automation…” by Brian Le
   “Manager’s Guide to GUI Test Automation” by Yury Makedonov
   “The ROI of Test Automation” by Michael Kelly
   “What is my ROI?” by mVerify Corporation
   SQA Forums
   “An Overview of Test Automation Frameworks” by Nathaniel
   “The benefits of keyword-based software test automation” by David
    W. Johnson
   Wikipedia
Igor Gershovich
Company: Connected Testing, Inc.
E-Mail: igershovich@connectedtesting.com
Phone: 720-933-9395
Website: www.connectedtesting.com

To top