Docstoc

Evolution Evolution

Document Sample
Evolution Evolution Powered By Docstoc
					An AutomAted testing institute Publication - www.automatedtestinginstitute.com




AutomAted .......
SoftwAre teSting
             .                                  AuguST 2009 $8.95
                                                                                   MAGAZINE




               Evolution
of        AutomAtEd SoftwArE
              tESting




     evolutionary                                   5 StepS to Framework
     roaD                                           Development
     inTroduCing your ProjeCT                       How To CreATe A FrAMework
     To new TeCHnology                              For MAinTAinAbiliTy



 August 2009                       www.automatedtestinginstitute.com   Automated Software Testing Magazine   1
Give me the key...worD: Building A Keyword driver Script in 4 StepS
Still Testing SAP Manually?




Worksoft Certify®   Get ahead of the curve and maximize the value of
                    your SAP investments with Worksoft Certify®, the
for SAP®            only code-free approach to end-to-end,
                    cross-platform business process validation and
Automate            automated testing.

Accelerate          Worksoft Certify® for SAP® is a specialized
                    lifecycle management solution for SAP that
Optimize            automates and optimizes SAP lifecycle tasks from
                    change impact analysis through functional testing,
                    performance testing and audit documentation.
                    Certify for SAP is unmatched in the industry for
                    accelerating time to value and reducing overall
                    project costs by up 40-80%.

                    Are you ready to test smarter and deploy faster?

                     Contact us at 1.866.836.1773 or visit
                      www.worksoft.com to learn more.
AutomAted
S    t
   oftwAre eSting
August 2009, Volume 1, Issue 2




Contents
Cover Story
Evolution of AutomAtEd SoftwArE tESting 14
Reviewing how test automation and its tools have evolved is useful only if it helps to understand where we go from here.
To do that, you must appreciate the technology and market forces that have shaped them and why, and what that portends
for the future. By Linda G. Hayes


FeatureS
EvolutionAry roAd 22
Software technology changes in existing applications often present roadblocks on the road to successful test automation.
Learn what steps will help you navigate your test automation over and around these barriers. By J.L. Perlin

5 StEpS to frAmEwork dEvElopmEnt 28
Developing an automated test framework can greatly increase the ROI of your test automation efforts, but it can also be a
daunting task. Read this article for a step-by-step framework development process. By Dion Johnson



ColumnS & DepartmentS
EditoriAl 4                                                    hot topicS in AutomAtion 38
evolve, adapt and Build                                        microblogging is hot!
Keeping automation on pace with three modes of change.         Test automation meets microblogging.

AuthorS And EvEntS 6                                           i ‘B’log to u 40
Learn about AST authors and upcoming events.                   Read featured blog posts from the web.


SwEAt thE tEchniquE 8                                          up-to-dAtE with Ati 42
                                                               Find out what’s happening on the Automated Testing
hand me the key...word
                                                               Institute’s Online Reference.
Learn how to build a keyword driver script.

                                                                  Automated Software Testing (AST) Magazine is an Automated
opEn SourcEry 12                                                  Testing Institute (ATI) publication that serves as a companion
Contribute to open Source projects in 4 Steps                                       to the ATI Online Reference.
Learn how to contribute to your favorite open source                   For more information regarding the magazine visit:
projects.                                                          http://www.astmagazine.automatedtestinginstitute.com

August 2009                            www.automatedtestinginstitute.com        Automated Software Testing Magazine                3
editorial

Evolve, Adapt and Build
by Dion Johnson


For anything to survive, it must be able
to change itself by three different ways:
evolving, adapting and building. This has
been evident in the natural world, and it also
holds true in the world of software. This
reality should be especially clear for software
testers and test automators, because the very
mission and reason for their existence is to
ensure systems are effectively evolving,
adapting and being built. This conceptual
trilogy is not confined, however, to only the
systems tested using automation, but also
applies to test automation itself.
     Evolution is defined as a gradual process     to aid in this understanding, this issue of the   frameworks. In addition, Hayes describes
in which something changes into a different        Automated Software Testing Magazine is            where the automation evolutionary approach
and usually more complex or better form.           dedicated to exploring each of these modes        has come up wanting, and proposes what the
This change is normally an innate response         of change.                                        next test automation evolutionary response
to some measured environmental stimulus                In the cover story, Linda Hayes delivers      should be.
that occurs over time. Adaptation is the act of    what is possibly one of the greatest articles          Next, J.L. Perlin provides a featured
adjusting oneself to                                                                                                            perspective on a
different conditions
or     a     shifting        ...possibly one of the greatest                                                                    modern day test
                                                                                                                                automation adaptation
environment. This
type of change is         articles ever on the topic of the test                                                                with a peek into a
                                                                                                                                project’s encounter

                                  automation evolution
typically marked                                                                                                                with contemporary
by a deliberate and                                                                                                             changes     to     the
direct      response
                                                                                                                                software environment.
to        immediate
modification in the environment. Building          ever on the topic of the test automation          In this article, an existing software application
is the act of increasing or strengthening, and     evolution. Titled “The Evolution of               with a substantial suite of automated tests is
may be a response to some outside stimulus,        Automated Software Testing”, this article         faced with changes to existing application
but it could also be based on a self-imposed       looks at software test automation from a          components. These changes threaten to render
decision to improve one’s situation.               macroscopic point of view and explores the        all existing automated tests useless as the
     The discipline of test automation has         gradual changes in software systems over          automated test tool contends with problems
experienced and been shaped by each of             the years. It takes us through a trip down        communicating with the new components.
these modes of change in the past, and             software computing memory lane, moving us         This is a problem that many automators have
continue to be defined by them even today.         from mainframe to Internet domination, and        come across, and if you haven’t, there is
Understanding these three modes of change          explaining how the test automation discipline     a good chance that you will. So J.L. Perlin
will help to better understand the current state   has responded to each evolutionary shift by       offers an approach for effectively navigating
of test automation, and will therefore help        moving from text capture/playback tools           your test automation through such modern
us all to chart a course forward. Therefore,       to the broad use of commercial automation         day technological changes.
                                                                                                                        (Continued on page 37)
4    Automated Software Testing Magazine                www.automatedtestinginstitute.com                                            August 2009
       How Do You Network?
       Whatever your choice, the Automated Testing Institute is there!



          YouT
                                                                          Twitter
                     Facebook                 Myspace




 Stay up-to-date with test automation by following the Automated Testing Institute on your so-
                                cial networking site of choice.




   Facebook        Myspace          Twitter           YouTube             LinkedIn           Blogger

       For more information, visit http://www.networking.automatedtestinginstitute.com
August 2009                   www.automatedtestinginstitute.com   Automated Software Testing Magazine   5
authors and events
Who’s In This Issue?
                           linda g. hayes, in the lead feature, offers an                     AutomAted
                           eyewitness account of test automation evolution. With over
                           20 years of experience in software development and testing,        SoftwAre teSting
                           Linda is the founder of three software companies including
                           AutoTester, the first PC-based test automation tool, and           Managing Editor
                           Worksoft, Inc., pioneer of next generation code-free test          Dion Johnson
                           solutions. Linda holds degrees in accounting, tax and law
                           and is a frequent industry speaker and award-winning author        Contributing Editor
on software quality. Named one of Fortune Magazine’s People to Watch and one of the           Edward Torrie
Top 40 Under 40 by Dallas Business Journal, she is a regular columnist and contributor
to numerous publications, including the Automated Software Testing Magazine,                  Director of Marketing and Events
StickyMinds.com and Better Software magazine. She is the author of the Automated              Christine Johnson
Testing Handbook, contributing author to Testing SAP R/3:A Manager’s Guide, and co-
editor of Dare to be Excellent with Alka Jarvis on best practices in the software industry.   A publICATIon of ThE AuTomATED
                                                                                                     TEsTIng InsTITuTE
Her article “Quality is Everyone’s Business” won a Most Significant Contribution
award from the Quality Assurance Institute and was published as part of the Auerbach
Systems Development Handbook. You can contact Linda at Lhayes@worksoft.com.


J.l. perlin offers a featured article that provides an
approach for handling modern day technology changes.
Perlin has over 20 years of manual & automated testing                                        ConTACT us
experience and has managed QA teams in both the                                               AST Magazine
public and private sector for organizations including                                         astmagazine@automatedtestinginstitute.com
Lockheed Martin, Social Security Administration
(SSA), General Electric, and the Centers for Medicare &                                       ATI Online Reference
                                                                                              contact@automatedtestinginstitute.com
Medicaid Services (CMS). Mr. Perlin is currently a lead
test automation engineer working on critical care systems for Philips Patient
Monitoring Informatics. You can email Mr. Perlin at jeffrey_perlin@yahoo.com.

                                                                                              ATI and Partner Events
                              dion Johnson provides step-by-step instructions
                               for building an automated test framework in which you          August 17
                               may develop and maintain your automated tests. Dion has        ATI honors finalists Announced
                               several years of experience in providing IT services to        http://www.atihonors.automatedtestinginsti-
                               both government and private industry in a manner that has      tute.com
                               demonstrated expertise in multiple areas of the software
                               development lifecycle. With a Bachelor of Science              August 17 - 31, 2009
                               degree in Electrical Engineering, he has spent much of         ATI honors Voting period
his professional career as a consultant in the areas of quality assurance (QA), quality       http://www.atihonors.automatedtestinginsti-
control (QC), software process improvements, requirements analysis and software test          tute.com
automation. As a regular conference speaker and presenter, Dion has delivered award
winning and highly acclaimed presentations at many of the most prestigious industry
conferences, including the StarEast and StarWest International Conference on Software
Testing, Analysis and Review, the Quality Assurance Institute Conference and Better
Software Conference. He also teaches college and business level classes relative to
testing and test automation, and has several published articles in various IT publications.
6    Automated Software Testing Magazine                 www.automatedtestinginstitute.com                                      August 2009
August 2009   www.automatedtestinginstitute.com   Automated Software Testing Magazine   7
Sweat the technique

hand me the key...word
Building A Keyword Driver Script In 4 Steps


Often called “Table Driven”, a Keyword                                                              advantage is that it is easier to read keyword
framework interprets automated tests that                                                           files than traditional scripts. Keyword files
are developed in a tabular format using                                                             typically mirror manual test procedures, and
a vocabulary of keywords. A keyword is                                                              the natural language of the verb-like phrases
typically a verb or verb-like phrase that is                                                        makes reading a keyword file similar to
associated with an external library function                                                        reading a collection of sentences. Finally,
responsible for executing the application                                                           while the Keyword framework itself may
commands necessary to satisfy the objective                                                         be more technical, the keyword files created
of the test.                                                                                        during everyday application automation are
     Therefore, in a keyword framework, a                                                           a lot less technical than files with statements
script that may normally appear as illustrated                                                      written in code. This allows less technical
in Figure 1 will instead appear as illustrated                                                      individuals to effectively contribute to
in Figure 2.                                                                                        automated test creation.
                                                                                                          These advantages don’t mean that we’ve
                                                                                                    escaped scripting altogether, however. We
                                                                                                    haven’t evaded scripting; we’ve merely
                                                                                                    changed the nature of the scripting that
                                                                                                    needs to be performed. There are at least
                                                                                                    two forms of scripting that remain: function
                                                  because many of the functions are created         development and driver script creation.
                                                  in such a way to not only be reusable for         If you purchase a commercial keyword
                                                  multiple tests within a single application, but   framework or obtain an open source keyword
                                                  also for tests across multiple applications.      framework, you may eliminate the need for
                                                  Redundancy may therefore be decreased             custom development of the driver script and
Figure 1: Code-based interface
                                                  across all applications that an organization      may also reduce the function development to
                                                  may be responsible for automating. Earlier        only those functions that are specific to your
     Figure 2 illustrates a tabular automated     script development also results from using        application and necessary for more efficient
test known as a keyword file, with the            this type of framework, due to an increased       automation. If you choose to create your own
keywords that dictate the actions to be           ability for automation to begin before the        framework however, the script development
performed shown in the Action column.             application is delivered. Using information       is solely up to you.
     There are several advantages to using        gathered from corresponding manual test                 This article will focus on the development
keyword files for automated test development      cases, requirements or other documentation,       of the driver script that interprets a keyword
including increased reusability. Reusability is   keyword files can be created well before the      file. Following are steps for successfully
increased with a Keyword Driven framework,        application is ready for automation. Another      creating and implementing a keyword driver




     Figure 2: keyword Driven interface

8    Automated Software Testing Magazine               www.automatedtestinginstitute.com                                            August 2009
script:                                             about the step to anyone reading the             application under test
 1. Create the keyword file structure               keyword file.
 2. Identify potential keywords                                                                  Step 3. IdentIfy Keyword utIlIty
 3. Identify keyword utility functions          Step 2. IdentIfy potentIal KeywordS              funCtIonS, argumentS and return ValueS
 4. Create driver script                        The keywords created will largely depend         Function definitions are a product of the
                                                on the functions deemed necessary for            framework and of the automator’s personal
Step 1. IdentIfy ColumnS In Keyword fIle        implementation of the associated framework.      preferences for how the function is to be
The keyword file columns will vary from         Not all of the keywords will be identified       implemented. One of the keys in developing
framework to framework. The only constant       during framework design and development,         an effective automation framework utility

            Keyword fIle




                                                                                    drIVer SCrIpt

                   The Driver ScripT
              callS funcTionS baSeD
             on The            KeyworD in The
                      acTion column

       Figure 3: Driver Script pseudocode and keyword File interaction

is that a column will almost certainly exist    but will rather continuously become apparent     function, however, is to make sure it is
for keywords to be identified. In addition,     as tests are automated. If developed properly,   independent, reusable, and able to be
the keyword file may have one or more of        the keyword framework will be seamlessly         implemented as intuitively as possible.
the columns illustrated in Figure 2. These      scalable with respect to the addition of new     Function arguments play a big role in
columns are defined as follows:                 keywords over time, but there are some           ensuring your functions met these criteria.
  • Action – Contains the step’s main           generic keywords that may be identified          For example, a function associated with the
     keyword, and identifies the action that    upfront, including:                              INPUT keyword might need the following
     will be executed                            • INPUT – Inputs a single data item             arguments:
  • Screen – Contains the name of the screen     • PRESS – Presses or clicks a button or          • Screen – This argument contains the
     on which the action is to be performed          link                                             screen on which the input action is to
  • Object – Contains the name of the object     • VERIFY_SCREEN – Verifies a screen                  occur
     on which the action is to be performed          has appeared                                 • Object – This argument contains the
  • Value – Contains values that may be          • VERIFY_FIELD_VALUE – Verifies                      object on which the input action is to
     entered into the application                    entered of selected field data               • Value – This argument contains the value
  • Recovery – Contains Exception                • VERIFY_TEXT – Verifies dynamic                     to be entered into an object
     Handling keywords.                              screen text                                 These arguments correspond to columns in
  • Comment – Contains helpful information       • INVOKE_APPLICATION – Opens the                the keyword file shown in Figure 2, therefore,

August 2009                                    www.automatedtestinginstitute.com           Automated Software Testing Magazine               9
                                                                                                         ati newsletter




                                                                                                     The ATI Newsletter keeps
                                                                                                     you abreast of the latest
                                                                                                     information relative to our
                                                                                                     Online Reference. Sent to
                                                                                                     registered members of the
                                                                                                     site on a bi-weekly basis,
                                                                                                     this newsletter keeps you
                                                                                                     informed on the follow:

                                                                                                     ATI’s featured Content
                                                                                                      • Featured Videos
                                                                                                      • Featured Tutorials
                                                                                                      • Featured Articles
                                                                                                      • Featured Tools
                                                                                                      • Featured Publication
                                                                                                      • Featured Poll
                                                                                                      • Featured Forum Post

                                                                                                     popular Content
                                                                                                      • Popular Tools
                                                                                                      • Popular Articles

                                                                                                     Automation omg!
                                                                                                      • Interesting News
                                                                                                      • Current Events
Figure 4: Sample Driver Script

the arguments will be fed values from these       If the driver script is developed using Ruby, it
three keyword file columns via a Driver           may appear as shown in Figure 4.                   Register on the ATI site today to
Script.                                                 Line 1 loads the UtilityFunctions.rb file.   receive this email newsletter. To
                                                  This is a file you develop that contains your      register visit:
Step 4. deSIgn and deVelop drIVer SCrIpt          user defined utility functions.
                                                                                                     www.registration.automatedtestinginstitute.com
As illustrated in Figure 3, the driver script           Lines 3 through 7 are responsible for
is responsible for calling the keyword file,      setting up and initializing the keyword file
reading all of the steps, and executing utility   which is developed in an Excel spreadsheet.
functions based on the keywords in each step            Line 9 begins the loop that reads all of
of the keyword file.                              the rows in the keyword file.
                                                                         (Continued on page 37)
10 Automated Software Testing Magazine                 www.automatedtestinginstitute.com                                            August 2009
Tired of searches that
return irrelevant information?




google                              - AutomAted teSting SeArch engine
The Automated Testing institute has partnered with google to create a software test
automation search engine. if you’re looking for software test automation information, you
will get the best results by using this search engine, because it only searches the sites
that matter most to automators. you can also restrict searches to blogs and forum sites.

                         www.googleautomation.com
August 2009                 www.automatedtestinginstitute.com   Automated Software Testing Magazine 11
open Sourcery

Ask not what your tool can
do for you
Learn to Contribute to Open Source Projects in 4 Steps


Ask not what your tool can do for you.                                                                     interested in automated software
Ask what you can do for your tool.                                                                         testing or interested in one of the
     My fellow automators, how often                                                                       authors writing about automated
have you pointed out the shortcomings                                                                      software testing. Either way, this is
of tools with which you have worked?                                                                       a good indicator that a project that
Now, how often have you sought to                                                                          maintains an automated test tool may
take an active role in fixing these                                                                        be a good fit for you.
shortcomings?                                                                                                Aside from simply picking a project
     I’m sure many of you are thinking,                                                                    because you like it, you may also
“Fixing tool shortcomings is not my                                                                        want to pick a project that is modest in
job! I use these tools to help me better                                                                   size; particularly if you are a beginner
perform my job. If I spend time fixing                                                                     to open source development. Smaller
tools, then I won’t have any time to do                                                                    projects typically have smaller code
my real job!”                                                                                              bases, provide easier access to other
     I hear all loud and clear, and you                                                                    developers, and have simpler bug
make good points, but while fixing               you on how this may be accomplished. In our               submission processes. Therefore,
tool shortcomings is not your responsibility,    research we came up with 4 steps.                smaller projects may make it easier for you to
it does have several benefits. Fixing tool                                                        quickly become productive.
shortcomings is normally not an option with      Step 1 – ChooSIng a projeCt                           Another option is to start your own open
commercial tools, but the beauty of open         There are numerous open source projects          source project. Maybe there is a browser
source tools, is that it is very much a viable                                                    extension or automated test tool plug-in that
                                                 ready and willing to receive contributions
option.                                                                                           you’d like to see created to support automation
                                                 from eager scripters, and the project list
     The May 2009 issue of the Automated                                                          of a particular technology; if so, you can lead
                                                 grows with each passing day. Following are
Software Testing Magazine contained an                                                            the effort in developing the desired extension
                                                 links that identify and provide access to some
editorial by Dion Johnson called, “Scripting                                                      or plug-in. Just search around first and make
                                                 of these projects:
Calisthenics”, and this article presented                                                         sure a project doesn’t already exist for what
various options for keeping automation                                                            you intend on doing. Not sure how to create
skills sharp during periods of inactivity.        • www.sourceforge.net                           a project? Some pretty good instructions are
The article was referenced in various             • www.opensourcetesting.org                     laid out at ehow.com (http://www.ehow.com/
automation Internet forums with a question        • www.code.google.com/soc/                      how_2091738_start-open-source-project-
posed to the forum’s community that                                                               sourceforge.html) for creating an open source
asked, “How do you keep your skills                   The challenge is now in determining         project with SourceForge.
sharp during periods of inactivity?” One         which project you should contribute to. The
of the community’s users suggested that          simplest way to pick an open source project      Step 2 – pICKIng your ContrIbutIon
contributing to open source projects is a good   is to look for one that maintains a tool that    Now that you’ve picked a project, the next
way to keep scripting skills sharp. We here      you use and/or are interested in. For example,   step is determining what you’re going to
at the Automated Testing Institute thought       given that you are reading the Automated         contribute to that project? You may begin by
this was an enterprising answer and decided      Software Testing magazine right now, it          asking yourself what shortcoming you’d like
to research and provide information for all of   may be safe to assume that you are either        to strengthen, and determining if it’s going

12 Automated Software Testing Magazine                www.automatedtestinginstitute.com                                           August 2009
to be a ‘fix’ or a ‘feature’? In other words,       important to do it with a productive tone.        or it may be as complicated as modifying
do you want to fix an existing bug, or do you       One way that you’ll be able to communicate        several code modules. However extensive
want to add a new feature? At this point in         with people on the project is by joining the      your scripting task, be sure to use the code
the process you may have already decided            project’s chat sessions and/or mailing list if    conventions defined by the project, including
what you want your contribution to be, and          they have one. Mailing lists are often used       conventions for indenting, variable naming,
that decision may have played a big part in         for discussing recent code changes or to          and commenting.
the project you chose to join. If not, however,     propose new code changes, so they offer a              Upon completion you need to post
there are some tips you                                                                                                         your changes to the
can use for picking a
contribution.               ...if key players on the project don’t                                                              project, a task that
                                                                                                                                is      accomplished
      The first and most
obvious thing to do is       like the ‘new guy’, there is a good                                                                differently
                                                                                                                                different projects.
                                                                                                                                                    for


                             chance that your proposals will get
to actually use open                                                                                                            Some          projects
source tools. There are                                                                                                         submit changes via
a bunch of them that                                                                                                            the mailing list,
perform a wide range                       rejected                                                                             and sometimes the
of functions. If you                                                                                                            project uses a version
work on a computer                                                                                                              control        system.
at all, you should have no problem finding at       great avenue for proposing your changes and       Whatever vehicle you use for sharing your
least one open source tool that can help you        soliciting feedback. Making your intentions       update, that update is normally presented in
with some of your tasks. Pick one, and then         known to the key group members early will         the form of a patch. A patch is a small piece
identify some of the features you’d like to see     improve the chances of your change being          of software that contains your fix or new
changed. What you come up with, could be            accepted, or it may allow you to receive          feature for a computer application. One way
your contribution to the project.                   feedback explaining why your proposed             to create a patch is to run what’s called a diff
      In addition, some projects maintain           change should be reconsidered.                    program. The diff program compares original
readily available “to-do” lists that identify            Using the bug system to study existing       files or directories against modified files
items that need to be fixed. Also, the project’s    bug reports is also a great way to learn the      or directories, and produces an output file
bug reporting system can be used for                culture of the project, because the reports       called a patch that contains the differences.
gathering ideas for what to fix.                    indicate who the key developers are and what      Upon submitting the patch, be prepared to
                                                    types of contributions normally get accepted.     make changes as suggested by the project
Step 3 – learn the projeCt Culture                  Aside from the bug system, you’ll want to         maintainers. For information on creating and
Much like countries and commercial                  appropriately utilize other project tools, such   submitting a patch, visit ehow.com (http://
businesses, each open source project has            as the version control system. Developers         www.ehow.com/how_2091741_make-patch-
its own unique culture, and your success in         on these projects often have little patience      open-source-project.html).
a project largely depends on your ability to        for individuals that can’t seem to follow              Before submitting your patch to the
learn and fit into that project’s culture. As       rules, standards and directions. To obtain        open source project, be sure you understand
much as you may like to believe success is          information on tools and conventions used         the open source licensing used by the project
only about technical abilities, it unfortunately    by the project, locate and read the developer     to ensure the patch is not rejected for legal
goes beyond that. No matter how good your           docs.                                             reasons, and to ensure any copyright you may
coding skills are, if key players on the project                                                      wish to impose on your developed code is
don’t like the ‘new guy’, there is a good           Step 4 – Code and poSt                            preserved.
chance that your proposals will get rejected.       At this point, it’s time to get down to the
     It is, therefore, important for you to be      business of developing your fix or feature,       ConCluSIon
aware of who the core maintainers are, who          and it’s imperative to begin by making sure       Therefore, my fellow automators, if you
the key contributors are, and who the more          you have the most up-to-date project code.        find yourself complaining about what your
vocal members of the project are. You’ll also       Next, search the code for the location in         test automation tool is not doing for you,
want to study how they communicate with             which the change is to be applied – a task        ask yourself, “What am I doing for my test
one another. This will provide insight into         that is often simplified by using an Integrated   automation tool?” If more people had this
who you may want to reach out to, and how           Development Environment (IDE). Now you            mentality, we’d all be a little more skilled,
best to reach out to them. It will be important     can begin scripting. Your scripting may be        and the discipline of test automation would
to make yourself known, but it’s equally            as simple as changing a line or function,         probably be a lot better off.

August 2009                                        www.automatedtestinginstitute.com            Automated Software Testing Magazine 13
     Evolution
of     AutomAtEd SoftwArE tESt
                                    An Eye Witness Account

                                                                      by linda Hayes
14 Automated Software Testing Magazine   www.automatedtestinginstitute.com       August 2009
                  Test automation has a history of failure. Despite the explosion of




n
                  software into every aspect of our lives, only a small percentage of tests
                  are actually automated today. Not that there aren’t pockets of success or
                  flashes of promise, but after more than a quarter century of investment by
                  vendors, customers and consultants alike, manual testing predominates.

                  So reviewing how test automation tools have evolved is useful only if it helps to
                  understand where we go from here. To do that, you need to appreciate the technology
ting              and market forces that have shaped them and why, and what that portends for the
                  future. For that reason, this article sets the context of each evolutionary period
                  within the state of software development and the applications market at the time.

                  Finally, test automation is a very broad topic, so when I say test automation I am
                  referring specifically to the automation of an otherwise manual test against a user
                  interface (UI). This does not mean that test tools cannot be used for non-UI testing,
                  or that there are not many other types of testing that are automated by a wide
                  array of approaches and tools, only that they are outside the scope of this article.

    August 2009                 www.automatedtestinginstitute.com   Automated Software Testing Magazine 15
Text Capture/Playback                                                       arose around dynamic content, like date and time stamps or frequently
                                                                            changing data fields. Timing synchronization was also problematic,
                                                                            especially as the user community expanded and performance
Mainframes dominated the early enterprise landscape. The mainframe
                                                                            fluctuated. Whenever the system changed or ran slower than the
was a monolithic environment, closely guarded behind glass walls
                                                                            original recording, context was lost and errors occurred.
and accessible only to a few. Computing resources were scarce
and expensive, so development processes were formal and highly
                                                                            But the killer issue turned out to be maintenance. If you recorded
structured, requiring flow charts and extensive documentation before
                                                                            100 transactions, there were 100 sets of the keystrokes and screens,
being committed to code. The majority of software was custom
                                                                            so any change had a wide ripple effect. It was usually easier to re-
developed for back office applications such as accounting and
                                                                            record the test than it was to fix it, which defeated the value premise
payroll. Annual release cycles were the norm, usually punctuated by
                                                                            of automation and encouraged reversion to manual testing.
emergency patches.
The first commercial test tools emerged as a result of the transition
                                                                            So early on, maintenance loomed as a primary obstacle to test
from batch to online applications, when multi-user terminal access
                                                                            automation success, with dynamic content and timing close behind.
first became available and manual testing became a necessity. These
tools recorded the keystroke inputs and character-based screen outputs
of a terminal session, then replayed the keystrokes and compared the
                                                                            Text Capture/Playback/Script
screens.
                                                                            Computing soon spread to mid-range systems, but the real
                                                                            breakthrough came with the announcement of the IBM PC. Personal
This approach was seductive. It sounded so simple: just do the test as
                                                                            computers had been around for a few years, but these 8-bit platforms
usual but record it into a script file, then play the test back as needed
                                                                            were generally perceived as truly “personal” computers used strictly
and compare the results. Unfortunately it was also deceptive. Issues
                                                                            by hobbyists and a few propeller heads (the early term for geeks). But
                                                                            with the imprimatur of Big Blue, PCs with 16-bit power entered the
                                                                            corporate computing world.

                                                                            Application development continued to expand on the larger platforms,
                                                                            but software for PCs was initially limited either to productivity tools
                                                                            such as spreadsheets or word processors, or to terminal emulators that
                                                                            turned them into smart terminals. The DOS operating system sporting
                                                                            a character-based interface, was single-threaded and severely limited
                                                                            on memory and storage; the first machines didn’t even offer hard
                                                                            drives, so functionality was constrained.

                                                                            The increasing popularity of spreadsheets and word processors
                                                                            propelled PCs into the corporate community, and the use of terminal
   The                                                                      emulators allowed them to replace “dumb” terminals. This drove
                                                                            manual testing from the mainframe to the PC. Soon automation tools
   Archaeological Dig
                                                              Mainframe domination                                 ibM PC Announced

         System Evolution



    Automation Evolutionary
          Response
                                                              Text Capture/Playback                                  emulator drivers


16 Automated Software Testing Magazine                  www.automatedtestinginstitute.com                                          August 2009
 appeared for the PC, capable of driving these emulators and other PC-       could use our tool because she was not a programmer.
 based applications, and these began to replace the mainframe-based
 capture/playback tools because they were cheaper and more powerful.         Until she uttered those words it had never occurred to me that all
                                                                             testers weren’t skilled at scripting. After all, Petroware had been a
 One of these tools was from my company, AutoTester. Our tool was            software company and we wrote code for a living. Although we sold
 originally developed by my first company, Petroware, when we ported         AutoTester as enhanced capture/playback/scripting, this was just a
 our oil and gas accounting software from an early IBM desktop to            euphemism for programming. And the stark fact of the matter is that
 the PC. Designed as a “session recorder” by my brother to automate          most manual testers are subject matter experts, not programmers.
 his programming tasks, we quickly realized we could use the tool
 for many purposes including demos, training, task automation and            So in the end we traded one set of problems for another: we addressed
 testing. But the more we used it the more we found we had to enhance        the maintenance, dynamic content and timing issues but introduced a
 it with additional commands to perform logic and branching as well          level of skill and complexity that excluded the majority of our target
 as to handle variable data.                                                 users.

 Before long, we had developed a proprietary language around the             I was devastated. Secretly convinced we had developed an essentially
 recording that enabled the scripts to handle timing and synchronization,    defective product that tormented our own customers, we started
 make decisions based on results, and externalize data into files. Now       a project to try to reduce or eliminate the programming aspect by
 a single script could loop through 100 records from a file and execute      generating scripts, and while we made progress it soon became
 reliably regardless of system speed or dynamic content. Because it          academic. Windows was born.
 also provided more power and control through its scripting language,
 increasing reliability while reducing maintenance, we consistently
 replaced older capture/playback tools and handily won competitions          GUI Capture/Playback/Script
 for new customers.
                                                                             The release of Windows brought even more opportunity but also more
 Yet in spite of our early success, nagging concerns persisted. Even         challenges. As a multi-threaded operating system it resolved many
 though we thought we had addressed the maintenance, dynamic                 low level issues; the maximum memory limit exploded and resources
 content and timing issues caused by capture/playback, we still              were managed across sessions. Windows also employed an API layer
 observed customers eventually reverting to manual testing. I was first      that converted mouse events into operating system messages that were
 puzzled then increasingly alarmed. What could be causing this?              intelligible and returned screen contents as a mix of objects that could
                                                                             be queried to return text and images. Recorded scripts became more
 I got my answer from a customer who sent their test manager to our          legible and reliable and the extra memory headroom enabled even
 offices for training. She was ideal for her job: she had over 20 years of   more advanced functionality.
 experience with the application that was being rewritten and knew the
 functionality inside and out. But after only a few hours of training, she   Another dynamic entered the mix as well. Personal computers were
 began to show signs of discomfort, and on the second day she openly         becoming not only ubiquitous but operating platforms in their own
 burst into tears. Horrified, I asked her what was wrong. She explained      right. Instead of just smart terminals for mainframes or personal
 that she was afraid she would lose her job since there was no way she       productivity tools, they offered powerful standalone applications.


                                               Client/Server becomes the                                    internet Takes Center
windows released
                                                          norm                                                      Stage




gui Capture/replay                                  Custom Frameworks                                    Commercial Frameworks


 August 2009                                     www.automatedtestinginstitute.com              Automated Software Testing Magazine 17
This transition from mainframes to personal computers caused a           slow lane as scripters took over automation.
corresponding shift from a formal and ponderous development process
to a more flexible, free-wheeling approach. Mainframes were so           Custom Frameworks
costly that few had access to them; personal computers were so cheap
and plentiful that the masses could afford them. This led to software    The PC continued to gain computing power, moving to a 32-bit
companies being born in garages by kids barely out of high school or     architecture, and Windows matured into a more stable and capable
                                                                         operating system that was a virtual monopoly on desktop machines.
                                                                         Client/server application development became the new norm,
  ...initial apparent success with re-                                   allowing PCs to offload tasks from backend mainframes and servers
                                                                         and provide users with ever-easier yet more powerful interfaces. The
   cording messages instead of key-                                      battle cry began to replace expensive mainframes with networks of
 strokes was soon replaced by disap-                                     PCs and servers.

         pointment and failure...                                        But the early days of client/server application development introduced
                                                                         even more issues for test automation. Commercial development
                                                                         toolkits became available, offering libraries of sexy new graphical
                                                                         controls that augmented the standard Win 32 object class library.
college with no patience or need for the constraints of formality.       Other component libraries also appeared offering everything from
                                                                         calculators to calendars to spell checkers, allowing programmers to
The type of applications changed as well. Whereas previously most        quickly assemble rich user interfaces with less and less effort.
applications performed back office tasks that were limited to a few
internal users, newer applications expanded their reach to front line    The problem was that many of these new controls were non-standard,
operations such as order entry and customer service. Many of the         so the standard Windows messaging layer was no longer applicable. A
original custom applications were replaced by commercial, off-the-       custom control did not necessarily implement the same messages that
shelf systems and new development moved into areas that interacted       the standard classes did. Testing tools that relied on the Windows API
more directly with core business processes.                              began to fail, diverting more and more automation time and effort to
                                                                         dealing with custom controls and other non-standard behaviors.
As a result, development cycles began to collapse. Market pressure
began to dictate functionality and annual delivery cycles gave way to    This new complexity raised the stakes. Now automation engineers
quarterly releases. More capability was needed faster and faster. This   were thrust into the development environment, forcing them to
added even more demand for test automation, because manual testing       learn and deal with the underlying implementation of the interface.
was less and less capable of keeping pace with changes and the risk      Whereas previously one had only to be comfortable with using logical
profile headed north.                                                    constructs and data file processing, now it was necessary to delve
                                                                         into object methods and properties and grapple with installing source
This new, more hospitable technical environment and market also          hooks or writing test APIs. This effort siphoned time and resources
brought new competitors. Frankly, I was hoping that the latest market    away from actually automating tests and into a struggle just to interact
entrants would know something we didn’t and show us how to assure        with the application’s user interface.
that our customers were successful over the long term. But the newest
tools presented the same problems. Scripting became even more            It could not have come at a worse time, as pressure on development
sophisticated to deal with ever more robust applications, and initial    continued to increase. The new client/server applications penetrated
apparent success with recording messages instead of keystrokes was       mainstream operations whose delivery schedules were driven by
soon replaced by disappointment and failure due to the skills and        user demand and stability became even more important. Greater and
effort required to write and maintain ever more complex script code.     greater functionality was needed and it had to be delivered faster and
                                                                         better. Quarterly cycles soon gave way to monthly.
Windows didn’t save test automation.
                                                                         More and more companies invested in automation to meet this demand
Presented with this reality, the most successful tool vendors            and automation engineers began looking for ways to reduce the time
aggressively enlisted partners and consultants who could offload the     and effort to code scripts. It became clear that test scripts were no
scripting effort from the manual testers. A new class of tester was      different than source code and adopting reusable components made
created: the automation engineer. Whereas previously manual testers      sense as a strategy to reduce development and maintenance. Thus
were courted with capture/playback, now they were relegated to the       frameworks were born.

18 Automated Software Testing Magazine                 www.automatedtestinginstitute.com                                         August 2009
                                                                                                                 Large companies soon found
   evolution in thought                                                                                          themselves with multiple
                                               Secretly                                                          tools and frameworks, some
                                            convinced we had                                                     of which had been rewritten
                                         developed an essentially                                                more than once.
                                      defective product that tormented                                           Complicating this was the fact
                                     our own customers, we started a                                             that shrinking development
                                       project to try to reduce or                                               schedules and expanding
                                                                                                                 functionality meant more and
                                              eliminate the                                                      more tests were needed. There
                                              programming                                                        was simply not enough time
                                                 aspect...                                                       and people available to write,
                                                                                                                 maintain and support one
                                                                                                                 or more frameworks while
                                                                                                                 also developing, executing
                                                                                                                 and maintaining the tests
                                                                                                                 themselves. Eventually the
                                                                                                                 overhead became so heavy
                                                                                                                 that companies – again –
Originally frameworks provided a basic infrastructure for test           reverted to manual testing just to keep up with delivery schedules.
execution, handling common tasks such as data file processing, error
and test logging and reporting. But they gradually became more           Commercial Frameworks
robust, providing simplified interfaces that employed spreadsheets
and flat files to allow non-technical testers to construct automated     By now the Web was taking center stage as a ubiquitous user interface.
test cases using pre-built script components. This introduced a role-    This was both good and bad. It was good because it provided a basic
based approach that allocated tasks based on skill sets and enabled      set of standards for access through browser APIs and HTML classes,
cooperation between automation engineers and manual testers.             but bad because the ability to instantly deploy changes meant delivery
                                                                         cycles could be daily. And, as static pages and controls gave way to
A number of framework types emerged, differentiated primarily by         dynamically scripted ones, object names deteriorated into gibberish
how the functionality was decomposed into components. The most           and behavior became unpredictable.
prevalent was referred to as “keyword” or “action word” based and
it entailed writing script code components around business tasks,        Pressure to deliver automation faster soared. Several enterprising
such as “Create Order” or “Invoice Customer”, then allowing users        automation engineers realized that they were constantly re-inventing
to invoke these from a spreadsheet and pass in data values. Others       the same framework over and over and decided to commercialize
organized their components around screens or windows, such as “Add       their solution and market it as a library of pre-built components. This
Customer Information” or “Delete Customer Information”.                  helped to accelerate the implementation curve and reduce the overall
                                                                         costs. But depending on the type of framework, there was still a fair
These frameworks succeeded in reducing overall development               amount of application-specific code required to be developed, usually
and maintenance while allowing non-coders to design and execute          complicated by the custom control or dynamic content challenge.
automated tests more easily, but still required custom code. Test
designers could write linear test cases but could not make decisions     One of these framework types – my favorite - was designed around
and control the flow of the test based on real-time execution results.   object classes. Commonly referred to as “table-driven” or “class-
That still required code.                                                action”, this approach sought to reduce or remove application-specific
                                                                         code by using components at the object class level: For example,
But frameworks fell victim to their own success. When the architect      “Press Button” or “Input Text”. A key advantage was that the same
and chief (often only) automation engineer left the company or moved     object classes and actions could be used across applications that used
to another area, his or her replacement almost inevitably believed       the same class library, allowing greater reusability with less code. It
it would be easier and better to rewrite the code left behind than it    was even possible to share libraries across companies.
would be to maintain it. With no de facto standards, almost every
automation engineer adopted their own pet approaches and designs.        As the amount of code that needed to be written continued to

August 2009                                   www.automatedtestinginstitute.com            Automated Software Testing Magazine 19
decline, the battlefield moved to the objects themselves. Virtually all     •    Test automation architectures will continue to shift from code
modern test tools rely on an inventory of the application’s testable             to data. The only way to keep up with the blistering pace of
objects. Variously referred to as an “application map”, “GUI map”                development will be to minimize or remove altogether the need
or “object repository”, this inventory contains a list of each object in         for custom code and accelerate or automate the acquisition of the
the application’s interface including its name, class and the attributes         application map.
used to identify it. This map enables an object to be added to a test,      •    Test automation penetration will continue to decline - unless
located and acted upon during execution, and provides a central point            something changes. The accepted rule of thumb says that 30-40%
of maintenance in the event of changes.                                          or even 50% of a project should be budgeted for testing, but that
                                                                                 assumed old-fashioned coding, not snap-together systems. There
But as previously noted, object inventories were becoming                        is no way that hand-crafted test scripts can keep up with high-
increasingly dynamic and unpredictable. Building the map was a slow              speed application production.


    Nonsense names, closed controls and unpredictable context will disap-
    pear when developers are both accountable and rewarded for success-
                            ful test automation.
and tedious process, often requiring the engineer to navigate to each       So what has to change? In my opinion, our only hope is for test
page and objects. But with dynamic pages creating scripted objects          automation to be integrated into the development process. No
on the fly, using names that weren’t just incomprehensible but also         application should be deemed complete without a test harness. Let’s
changing depending on user choices, session parameters or a host of         make developers responsible for making their software testable instead
other factors, this was often an impossible task. Without a predictable     of engaging in a constant struggle to compensate for their latest bad
and reusable object inventory, automated tests are unusable. Trying         behavior. Nonsense names, closed controls and unpredictable context
to maintain them with daily releases was crazy enough, but if the           will disappear when developers are both accountable and rewarded
attributes changed within a release without any predictability, it was      for successful test automation.
simply impossible.
                                                                            Such a harness would provide a test object inventory and interface
So history shows us that every time test tools and implementation           with defined methods and properties to be shared by developers and
approaches are enhanced to meet the latest challenge, new ones are          testers alike. It would be kept current with all software changes and be
introduced. Vendors and customers both find themselves in a perpetual       part of a holistic test process that includes automated unit, integration,
state of catch-up, seemingly doomed to repeat the past by continually       system and user acceptance.
reverting to manual testing.
                                                                            Test tool vendors needn’t become extinct; just shift their focus to
What’s Next                                                                 managing data instead of code. Automation engineers needn’t become
                                                                            an endangered species; just shift their focus to managing the object
                                                                            inventory and interface metadata. But don’t react just yet; history also
The only way to break this cycle seems to be to anticipate the future
                                                                            tells us that changes take decades. Even today there are companies
and be ready for it, instead of reacting to it. The harder question is to
                                                                            who believe capture/playback is a viable automation strategy, though
know what’s coming. Since I don’t own a crystal ball, all I can do is       we have known better for almost 30 years.
look at history and current trends to project what’s coming.
                                                                            But until management sees the light and directs development to play
History says that:                                                          their part, we all need to continue finding ways to reduce the amount
                                                                            of test code that must be written and maintained, work closely with
•   Development will continue to accelerate. Software will                  development to correct testability issues and manage the impact
    continue to insinuate itself into every aspect of our lives and         of changes, and educate management that the historical ratio of
    jobs, driving demand for more functionality faster. Companies           development to test effort is outdated but that the extra investment in
    will adopt packaged applications when possible. Developers              automation is worth it. And keep trying.
    will take advantage of increasing levels of abstraction to keep
                                                                                  The Automated Testing institute (ATi) conducted a
    up, using code libraries, code generators, web services and
    other middleware technologies to quickly assemble composite                   podcast interview with linda Hayes. To listen in, visit
    applications.                                                                         http://www.podcasts.automatedtestinginstitute.com.



20 Automated Software Testing Magazine                  www.automatedtestinginstitute.com                                                 August 2009
real adventures in automation

Submit your Stories!
Now’s Your Chance to be Heard




 Hav
              e so
                   me
                      thin
                           g to
                                say
                                    ?
                             o listen
              ATI i s here t
         Ad




    The Automated Testing institute believes that
      one of the best ways to improve upon test
   automation is to listen to the lessons learned by
   others. So, if you have an interesting story that
   you’d like to share with the world, contact us at
    stories@automatedtestinginstitute.com to find
                        out how.


August 2009             www.automatedtestinginstitute.com   Automated Software Testing Magazine 21
                         Navigating Existing
Evolutionary             Frameworks
                         through Modern
                         Day Technological
                         Evolutions




                                   by j.l. Perlin
Road




22 Automated Software Testing Magazine   www.automatedtestinginstitute.com   August 2009
   it is a natural occurrence for products to evolve over time. Advancements in technology and
  consumer demands spur radical changes in design that make products more efficient, easier to use,
  and less costly to produce. How can you prepare test automation to best handle these changes? it
  is becoming increasingly essential to plan for change when deciding upon an automation approach.
  in order to do this it helps to understand why these changes are occurring so we as test automation
  engineers can understand the catalysts behind these changes and be more prepared not only for
  these new technologies and their effects on automation, but also for the interactions we will have
  with the organizational stakeholders who have introduced these new improvements.



Evolution of a
Product
     So, why do products evolve?
let’s use the automobile as an
example. The first self-propelled
road vehicle was a steam-powered
military tractor invented by the
French engineer Cugnot. it was
a ground-breaking invention, but,
alas, due to low speeds (2.5 miles
per hour), operating dangers, and
the need to stop every ten to fifteen
minutes to build up steam, it was
not an improvement over the horse          speed of the vehicle was still slower    to the introduction of hybrid vehicles
and buggy.                                 than the horse and buggy, the            and all-electric cars, with steps
     The next attempt at making            range of the vehicle was very short      made to bring us closer to the day
a successful automobile was in             (about 15 miles before a recharge        when vehicles we drive every day
the early 1800’s when a dutch              was required), and the cost was          can run on hydrogen. but, why
engineer used an engine that ran on        exorbitant and out of reach for most     did the motor vehicle evolve in this
rechargeable batteries to design the       people of that era.                      manner? what was the catalyst for
first electric vehicle. The vehicle was         Fast forward to today, and          new improvements and changes
initially successful due to a smaller      we see that over the years cars          to the overall design? it is the
and lighter weight engine that             have become much more fuel               same underlying factor that causes
allowed the vehicle to travel faster       efficient, safer, and easier to drive.   other products to evolve, including
and further than its predecessor.          improvements to electric and             computer systems and software,
despite the improvements, the top-         combustion technologies have led         and that factor is market demand.


August 2009                               www.automatedtestinginstitute.com   Automated Software Testing Magazine 23
How technology Evolution                          efficiently improves upon the proverbial          forward and the number of products that have
Affects Test Automation                           horse and buggy. Changing major or even           been made available to development teams to
   Technologies of the Information                minor pieces of an application will most          help them create large, complex applications
Age are evolving at a rate of speed that          likely require changes to other components of     in less time. Automated test tools often have



             THE TRUE CHALLENGE FOR SOFTWARE TEST ENGINEERS IS
       DETERMINING WHETHER OR NOT THE CHANGING TECHNOLOGY
                 AFFECTS THE WAY IN WHICH THE PRODUCT IS TESTED.


increases exponentially with each passing         the system. And all of these new and updated      issues when interacting with newly
day. Applications are growing in size and         components need to be tested, along with          released third-party custom
complexity. One major software component          the legacy components that aren’t changing        controls and components,
that has undergone massive changes is the         to ensure that they still perform properly        and these have become
operating system. Much like the automobile        under the new design. The true challenge          very popular in recent
engine, operating systems have undergone          for software test engineers is determining        years. Although very useful
numerous changes since their inception in         whether or not the changing technology            to developers, they can be a major
order to meet consumer demands. These             affects the way in which the product is tested.   problem for the
demands have led to changes that                   In addition, we must determine if we can
have made operating systems                             use the same tools we used prior to the
increasingly efficient, faster,                             technology change to test these new
intuitive, scalable, and more                                components, or if we need to look at
secure while providing powerful                              new tools and/or a new approach.
features and useful functionality                        Just like the products themselves,
previously unavailable to users.                     these tools need to be more efficient,
     It seems pretty clear cut, no? Changes       easier to use, and affordable.
in market demands spur innovations in             But most
technology. What, however, does
this mean for software
t e s t                                                                                                                       automation
                                                                                                                engineers. There may also be
                                                                                                             issues with development technologies
                                                                                  of all the                like .NET, Windows Presentation
                                                                       automated tools need to      Foundation (WPF), ActiveX, etc., that are
                                                          be able to access the new components      being used by your developers to design the
                                                  of the application and have the ability to        bulk of your application. The majority of the
                                    engineers     perform all of the methods against the test       time we cannot find a tool that can address all
                             that          test   objects that our tests require to fulfill their   of our needs, but at times we do find one that
                         these        evolving    objectives. As technology moves forward, so       can handle the majority of the tasks at hand.
                       applications? It means     must our testing approaches, processes, and
                                                  tools.                                            A Modern-Day
                       that we have to lift the
                                                                                                    Evolutionary Scenario
                       hood,      kick      the        In today’s environment finding a tool
           tires        and take this new         that meets your project requirements can be           Imagine that you’re working at an
                       technology for a test      a difficult task, especially when you consider    organization and have been there for a couple
            drive to ensure it effectively and    how quickly technology has been moving            of years, and during this time you and your


24 Automated Software Testing Magazine                 www.automatedtestinginstitute.com                                           August 2009
team have been building and maintaining a             your concern was centered on testability. The     posed similar questions on how to automate
suite of automated tests to execute against the       first few controls presented by development       tests for these controls but, unfortunately
product that your company markets. When               were proposed for use in portions of the          lacked any useful responses. Some of these
automation was first introduced to the project        application that were minimally accessed by       threads were several years old, so it was
new test automation resources were hired              users, and that required only a small set of      becoming apparent that the tool vendor
(including you) and the first assignment was          tests for validation. Worst case scenario: If     probably had no plan to release any add-ins
to evaluate the AUT to identify the test tool         your automated tool couldn’t handle these         for this technology. You next contacted the
requirements. Most importantly you wanted             controls you may have only had a handful of       manufacturer of these 3rd-party controls
to find a tool that could interact with all of        additional tests to execute manually. Then        to see if it was possible to further explore
the different components of the application,          the other shoe dropped. A new grid control        automation possibilities with your tool. The
allowing you to automate more tests and               was presented that would be accessed by all       manufacturers’ response was that this was
exercise more of the AUT functionality. You           of the product’s users and a large portion        theoretically possible, but that the controls
looked at a number of different tools and             of your automated tests. In other words, if       do not officially support your automated
compared them using a set of pre-determined           you’re automated test tool couldn’t handle        testing tool. When you asked how to make
criteria/requirements.                                this control; the majority of your automated      the controls accessible by the automated tool,
     Upon completing the evaluation you               tests would become useless. You gently            you were only informed that a few of their
selected the test tool that would provide the         but firmly communicated your concerns             customers had attempted automating tests
most value to your project, identified an             regarding the affect that these new controls      against these controls but their results weren’t
automation approach that best suited your             would have on the existing automation             readily unavailable. Hmmm… someone was
project, and then began automating the smoke          suite so the developers agreed to send you        trying to sell something here.
and regression tests. Eventually your team            a sample grid allowing you to evaluate how             It was time to find other options for



                IT WoULD STILL BE UP To THE TEST TEAM, HoWEVER, To
        DETERMINE WHETHER OR NOT THESE NEW CONTROLS WERE AN
            IMPRoVEMENT oVER THE PRoVERBIAL ‘HoRSE AND BUggY’

built a large suite of tests that you designed        your automated test tool would react to the       making your tool adapt to the new technology.
with great deference given to maintainability,        newly proposed grid controls.
and you felt pretty comfortable knowing                                                                 Option 1: Extensibility – Some automated
                                                                 Researching Solutions                  testing tools offer extensibility, which means
that most updates and fixes made to the
application could be addressed easily using                                                             that you’re able to integrate the tool with the
                                                           After receiving the sample grids as
your well-planned framework.                                                                            new controls by executing small snippets of
                                                      promised you began trying to access the
                                                                                                        custom code via calls from the test script.
     After several new versions of your               controls using your existing automated
                                                                                                        This solution is often the last resort, because
application were released the development             testing tool, and that’s when you learned the
                                                                                                        it could require extensive setup, coding, and
team started looking at new 3rd-party                 bad news; the tool didn’t fully recognize the
                                                                                                        almost always requires assistance from the
application controls that could be used               grid components. It recognized that a new
                                                                                                        development team.
to improve upon the current design. In a              control object existed, but couldn’t access
meeting, the development team presented               any of the rows, columns, menus, or buttons       Option 2: Workaround – Another possible
the proposed controls, and these controls             contained within this new grid control. Was       solution might be to identify a workaround
appeared to provide required application              there a way to solve this issue? The first step   in the application. Was there another screen
functionality and also looked much nicer than         you took was to contact the automated test        in the application that you could be used to
the previous, less powerful controls that were        tool vendor to see if they supported these        accomplish the same goals? Was there a
presently in use. It would still be up to the test    controls. To this they responded “No, but         way to navigate around this control while
team, however, to determine whether or not            you can search our knowledge base to see          accomplishing the same test objectives? If
these new controls were an improvement over           if anyone has done this”. Searching their         so, this would allow the automated tests to
the proverbial ‘horse and buggy’, therefore           knowledge base turned up several threads that     execute, but you and the developers would
August 2009                                          www.automatedtestinginstitute.com            Automated Software Testing Magazine 25
                                                                                                    shoes. You’ve done your research and have
      which way to Maximum roi?                                                                     discovered this cool 3rd-party control that
                                                                                                    will fulfill the new and existing system
                                                                                                    requirements and/or is a major improvement
                                                                                                    over the previous control. You’ve performed
                                                                                                    your evaluation, probably of proof-of-
                                                                                                    concept, and have presented this idea to the
                                                                                                    rest of the development team. Now some
                                                                                                    automation engineer is walking in saying
                                                                                                    it shouldn’t be implemented because it will
                                                                                                    break the majority of the automated tests.
                                                                                                    That isn’t going to happen without a fight.
                                                                                                                  Offering Options
                                                                                                    Since it is good policy to avoid fighting in
                                                                                                    the workplace, the best thing to do would
                                                                                                    be to walk into the developer’s office not
                                                                                                    only with a list of constraints, but also a list
                                                                                                    of options. Keeping a positive demeanor
                                                                                                    is of utmost importance. If possible, the
                                                                                                    most effective way to communicate this
                                                                                                    information to development would be via
                                                                                                    the Quality Assurance (QA) Manager. Inter-
                                                                                                    team communication is one of the major
                                                                                                    roles of a manager, so as an automation
                                                                                                    engineer you might as well take advantage
have to remain aware of the fact that the        verify its functionality is worth the effort.      of it. Additionally, program management
bypassed controls would need to be tested        For example, if a new control is being added       may be involved and have input regarding
manually.                                        but it only affects a small number of tests, you   the final decision, so it is best for them to
Option 3: Programmatic Solution – Another        might want to see if the effort of automating      communicate with someone that they are
option would be to design a programmatic         the tests against the control is cost-effective.   familiar and comfortable with.
solution; a coded solution offered by            It may make more sense to allow the test to
                                                 remain manual. Conversely, if the new control      It is important to include caveats when
developers providing a custom way for                                                               communicating these options. For example,
your tests to bypass this screen while still     is going to be used frequently and is touched
                                                 by a large number of your automated scripts,       you might have a programmatic workaround
accomplishing the test objectives. For                                                              in mind, but this would require developer
example, if the new controls were being          it might be worth the effort. Ultimately, it
                                                 boils down to how easy or difficult it will be     involvement, add time to the project schedule,
implemented on an ‘Add User’ screen, was                                                            remove focus from other current automation
there a way that the developers could allow      for automation to adapt, and if the effort is
                                                 worth the result.                                  activities, and forego any testing of the new
you to bypass this screen using pre-built user                                                      control itself since it is being bypassed. You
data stored in the user information database             Presenting Automation’s Case               might have found an add-in that could resolve
that your test could access and use? This        So you’ve discovered that your automated           most of the issues you had documented
would again result in the tests for the new      testing tool has problems accessing and            during the evaluation, but the add-in was
controls being reserved for manual execution,    manipulating these new 3rd-party controls.         expensive, code needed to be re-written to
but it would also allow all of your tests that   Should you just run to the development team        replace the previous code used to manipulate
need to access information processed on this     and tell them that the new control can’t be        the previous controls, and the add-in still
screen to continue running properly.             automated? Before doing this you need to           wouldn’t provide all of the functionality
              Analyzing Impact                   keep several things in mind. The developer(s)      required to meet the test objectives for this
When identifying different options it also       who discovered these new controls and are          application component. Or you might explain
makes sense to look at the impact that           pushing for their implementation have not          that your automated test tool extensibility
the new control has on the application to        only a technical stake in this but an emotional    features would allow you to handle this new
determine if automating tests to exercise and    stake, as well. Put yourself in the developer’s    control via automation, but it would also

26 Automated Software Testing Magazine                www.automatedtestinginstitute.com                                             August 2009
require developer participation to help design      return-on-investment after considering all of     on test automation. This will in turn help
the code. Until this code is developed and          the activities that need to take place in order   your organization to release an improved,
deployed automation test coverage would be          for the effort to be successful.                  well-tested product to its customers while
drastically reduced.                                                                                  maintaining positive, synergistic inter-team
                                                    lessons learned
                                                                                                      relationships within your organization.
You or your manager might also want to point        After experiencing similar scenarios at
out how this new technology would affect            several different IT organizations I was able
                                                                                                      Final Thoughts
other teams in the organization. Business           to learn some valuable lessons that resulted      Products naturally evolve or they go extinct,
Analysts would have to rewrite, add, or             in a set of guidelines that I now follow          overtaken by new products that improve
revise requirements. Technical Writers              to make it through these situations while         upon the previous design. New technologies
would have to update User’s Guides and              experiencing as little pain as possible. The      are a part of this evolution, and automated
other documentation to reflect the changes to       most important guideline is to ensure that        testing tools and methodologies will continue
functionality. The Sales Team would need to         other teams in your organization have a good      to evolve to address changes in technology
get up-to-speed on these new controls so they       understanding of automated testing processes      that come down the pipeline in the future.
could market these improvements to existing         and procedures. This can be accomplished          So much like we’ve moved from steam to




                 ...ENSURE THAT OTHER TEAMS IN YOUR ORGANIzATION
               HAVE A GOOD UNDERSTANDING OF AUTOMATED TESTING
                                           PROCESSES AND PROCEDURES.

and potential customers. The Training Team          by offering to conduct formal presentations       gas to hybrid to electric in car engine design
would have to revise presentations, manuals,        or brown bag lunches, possibly resulting in       and application, the same has happened for
and other training materials.                       enhanced development processes that keep          IT systems and software development. It
The number one objective in making this a           automation in mind when developing new            is important to keep up-to-date with new
successful meeting is to avoid friction. Don’t      code. The developers may work harder to           technologies and their impact on automation,
make the developer view you as a roadblock          ensure that new code provides hooks for the       make your organization aware of automation
or the enemy. If you are excited about this         automation tool to grab on to. Another good       processes and the affect their decisions will
new control and agree that it will improve          guideline to follow is to always be aware         have on these processes, all while keeping an
your product, let them know! Tell them that         of what development is researching and            eye on their possible effect on your current
you will do everything within your power to         may be planning for future versions of your       automation model. This will help to make
make this work. Forming and maintaining a           product. The best way to do this is to ensure     the addition of new technologies to your
collaborative relationship with the developers      that you are invited to development meetings      product as painless an experience as possible
is of paramount importance.                         where these items may be discussed. This          while leveraging all of the advantages that
                                                    allows automation to have a voice in              automated testing offers to your organization.
often the development and QA teams can find
middle ground. The collective decision may          the decision making process throughout
                                                    the project lifecycle and also allows you
                                                                                                             the automated testing
be to delay the addition of these new controls                                                             institute (ati) conducted a
until you have fully prepared the automation        proactively conduct research prior to any
                                                    formal planning. Finally, be sure not only to
                                                                                                           podcast interview with J.l.
framework to handle them. All parties may                                                                perlin that offers a discussion
agree on QA halting all new automated test          research the technology being considered by
                                                                                                          of this article, as well as his
design while you concentrate your efforts on        your organization, but similar technologies
                                                                                                          test automation philosophy.
this compatibility assignment. Stakeholders         offered by other vendors, that may present
                                                                                                                   look for it at:
may decide to search for other controls that        more options.
may be more automation-friendly. Or the             Following these guidelines will increase             http://www.podcasts.automatedtestinginstitute.com
parties may decide that the impact of adding        the chance of effectively implementing new
these new controls does not result in a positive    technologies while reducing the impact
August 2009                                        www.automatedtestinginstitute.com            Automated Software Testing Magazine 27
  The creation of a

  well thought out                                It is not very difficult these days to sell
                                                  those who are involved in the Information
  framework that                                  Technology (IT) industry on the importance of
                                                  regression automation – at least the theoretical
                                                  importance. It seems as though most understand
  is customized to
                                                  that test automation saves time, is repeatable,
                                                  and if done correctly, can be very reliability.
  your particular
                                                  Putting this theory to action proves to be
                                                  problematic, however, due to the commonly
  environment can                                 held belief that automation should require
                                                  no additional thought other than that which
  be time consum-                                 requires one to execute existing manual
                                                  test procedures while being recorded by an
  ing, so this article                            automated test tool.
                                                  In order to facilitate the desire to crank out quick
  identifies 5 simple                             and thoughtless automated tests, the industry
                                                  is creating more and more “user friendly” GUI
  steps that will help                            automation testing tools, both commercial
                                                  and open source, and these tools offer record
  in speeding up the                              and playback features that generate code or
                                                  code-less visual scripts that may seemingly




                                              5
  process.
                                                                     StepS t
                                                                     Develo
28 Automated Software Testing Magazine   www.automatedtestinginstitute.com                 August 2009
 test an application with little       is often thought of as the collective
 effort. These applications, with      structures that compose a unit
 their increased capabilities, can     testing tool. In this article, however,
 be extremely helpful, but also add    frameworks will be discussed in a
 to the problem. One of the major      different context; one where it is
 selling points for some of these      defined as a set of abstract concepts,
 applications is that, “With our       processes,        procedures       and
 application, you save a lot of time   environment in which automated
 because you don’t have to spend       tests will be designed, created and
 time thinking about, and planning     implemented. In addition, this
 out how you want to perform your      framework definition includes
 tests. You just record your steps,    the physical structures used for
 and playback the resulting script.”   test creation and implementation,
 It is important to avoid falling      as well as the logical interactions
 into the trap of believing that       among those components.
 some magical tool is going to         The creation of a well thought out
 eliminate the need for thinking       framework, customized to your
 and planning, because automated       environment can be challenging,
 testing is most effective when        so this article identifies 5 simple
 planned and implemented within        steps that will help serve as your
 a framework. The term framework       blueprint for getting it done.


to Framework
opment
 August 2009                 www.automatedtestinginstitute.com   Automated Software Testing Magazine 29
1          Perform Initial                                                        to use or build upon in your automation efforts. There are four main
                                                                                  segments in this framework design:


           Framework Design                                                         •
                                                                                    •
                                                                                        Driver Segment
                                                                                        Test Segment
                                                                                    •   Function Segment
Designing an automated test framework involves first identifying                    •   Parameter Segment
the type of framework that will be developed. This typically
requires researching and understanding your software development                       The Driver Segment, represented by a script or tool, reads from
environment as well as the application(s) under test (AUTs), and                  the Execution Level File, which stores execution level information for
determining which framework will work best (see Table 1).                         each test script (test name, test priority, etc). Next, the Driver Segment




                            Figure 1: Framework Component interactions
     When in doubt about the framework that will work best in                     calls upon the Parameter Segment to initialize the environment, and
your environment, the safest choice is a functional decomposition                 set parameters necessary for the test run. After calling the Parameter
framework. Functional decomposition frameworks can be as simple                   Segment, the Driver Segment calls and executes tests based on the
or as complex as deemed necessary in order to handle the demands of
                                                                                  associated priority. Control is then yielded to the test scripts in the
the automation scope.
                                                                                  Test Segment for AUT verification. The Test Segment frequently
     Once a decision has been made about the framework type, it is
imperative to identify the basic components that will compose the                 calls functions from the Function Segment for performing succinct
framework, and have a high-level picture of how these components                  test case actions that collectively compose a test case. In addition, the
interact with one another.                                                        tests in the Test Segment may call upon Data Files to obtain the data
     Figure 1 presents a basic design example that you may decide                 necessary for test execution.

 Framework                                                                        Description
    Type
 Linear           All components that are executed by automated scripts in this type of framework largely exist within the body of that script. There is little-to-no
                  modularity or reusability in a Linear framework.
 Data-driven      Most of the components that are executed by a Data-driven script largely exist within the body of that script, with the exception of the data. The
                  data used in Data-driven scripts is typically stored in a file that is external to the script, which promotes script reusability
 Functional       This is a framework that involves the creation of modular functions for executing fundamental actions that occur during test execution.
 Decomposition    These actions may then be called upon and executed independent of one another, thus allowing them to be reused by multiple tests within the
                  automated test bed.
 Keyword          This type of framework is strongly dependent on modularity concepts embodied by functional decomposition. In comparison to the Functional
                  Decomposition framework, the keyword framework evaluates the functions at a much higher level of abstraction, so that tests are no longer
                  developed using code, but rather developed in a tabular format that is interpreted and executed by a driver script or utility.
 Model-based      Often called an “Intelligent framework”, Model-based frameworks go beyond creating automated tests that are executed by the tool. These
                  frameworks are typically “given” information about the application, in the form of state models, and the framework “creates” and executes tests
                  in a semi-intelligent, dynamic manner.

table 1: Framework types
30 Automated Software Testing Magazine                      www.automatedtestinginstitute.com                                                        August 2009
2          Create the                                                      automated tests execute on a different server, or run in a completely
                                                                           different environment. With such considerations made during the

           Directory Structure
                                                                           upfront creation of the physical structure, an ultimate increase of the
                                                                           framework’s portability will almost certainly be a positive by-product.
                                                                                The physical structure may be maintained by a test management
                                                                           tool, some sort of Integrated Development Environment (IDE),
Test automation is a small-scale form of software development that         or simply by a file system. Below is a pictorial representation of a
involves creating a software package. It is, therefore, important for      sample physical structure for an automated testing framework. All of
this package to have a preplanned physical structure. The physical         the components are housed under a single Root directory, which aids
structure is similar, but different from the component design identified   in portability. In this root directory are several sub folders, including:
in Figure 1. The component design is more of an abstract representation
of how the framework components work together, while the physical           •   Initialization Files
structure identifies how and where these components will be stored.         •   Configuration Files
     The upfront identification of the physical structure is important      •   Function Library
for several reasons. Obviously, it is important because you need            •   Driver Files
to know where things are going to be stored. It is also important           •   Test Scripts
because the framework may need to be collectively moved as a
single unit. Such a move may be prompted by the need to have the           These items will be discussed in subsequent steps.




                                                                    root
                                                                  Directory




                     init                  Config                Function                   Driver                   test
                    Files                   Files                 library                   Files                   Scripts

        Figure 2: Directory Structure

                                             Config               The Function                                    Test Scripts are
                       Init                                                              A Driver Script
                                        (configuration)          Library holds a                                  the components
                (initialization)                                                           coordinates
                                        Files establish          set of functions                                   that actually
                   Files bring                                                            test activities
                                           parameter                used by the                                        perform
                     the tests                                                           by calling and
                                        values that are          automated test                                      verification
                 environment                                                              executing the
                                         necessary for               suite. It is                                   activities on
                to a controlled                                                           test scripts. It
                                          the test run.              normally                                      the application
                 stable point.                                                             may run init
                                             Config                loaded at the                                      under test
                    This will                                                              files, set up
                                        parameters are             beginning of                                      (AUT). Test
                  increase the                                                                config
                                         similar to Init          a test run, and                                 scripts perform
                chances of the                                                             parameters,
                                        parameters but             its functions                                     actions, and
                   test scripts                                                            and run test
                                          are likely to              are called                                   generate results
                     running                                                                  scripts.
                                         change more             throughout the                                          files.
                    properly.
                                           frequently                 test run.

August 2009                                    www.automatedtestinginstitute.com              Automated Software Testing Magazine 31
The Automated Testing
Institute (ATI) Online
Reference is the web’s
foremost comprehen-
sive online resource for
obtaining, exchanging
and certifying industry
standard test automation
knowledge, information,
and techniques.


    Visit us today!
3          Develop Init and                                                      of the automated test suite. These settings are often specific to
                                                                                 application technology. For example, in web application testing


           Config Files
                                                                                 you might initialize the environment by doing the following:
                                                                                 »   Clearing the cache
                                                                                 »   Turning off auto-complete
                                                                                 »   Turning off browser animation effects
An Initialization File contains a set of parameters or code that brings          »   Setting cookies
the environment to a controlled stable point. If the environment                 »   Setting frame format
is initialized, there is a better chance of the test scripts running as
designed. A Configuration File contains a set of parameters needed               Configuration parameters are those parameters that may change
for the test run.                                                           more frequently, but upon changing, they will permeate through
      The main difference between initialization parameters and             multiple tests within that suite. Parameters that may be set in the
configuration parameters is that initialization parameters are not likely   configuration script include:
to change from test run to test run, or from machine to machine. Instead,
initialization parameters will remain fairly constant throughout the         •   User Ids and passwords,
project life cycle.                                                          •   Pointers to databases,
      Initialization parameters/actions that are addressed in                •   Public variables and constants, and
Initialization Files include those that are responsible for:                 •   Pointers to the application (domains, URLs, etc).

 • Setting Directory Paths – Directory paths help in finding key                 To help better understand configuration parameters, let’s examine
   automation components during runtime. At a minimum, all of               a scenario represented by the figure below. The automated tests in
   the main directories of the automated test framework should be           this environment may be executed by several different testers. Tester
   included in this.                                                        1 may log-in on one machine to run the tests, while Tester 2 logs into
 • Loading Library Files – Test automation library files contain            another. Each tester may have their own set of test data, and must also
   reusable functions that may be shared across multiple tests. These       choose, at run-time, what front-end server to connect to, then what
   libraries may be loaded during the automation initialization             back-end data server to connect to.
   process.                                                                      No one should have to make all of these determinations every
 • Modifying Application Settings – Initialization of application           time they sit down to run their automated test bed. Configuration
   specific settings is often important for the successful execution        parameters may be set up to maintain the desired configurations.



                                                      Test Data                  Application Layer                Database Layer




                      Tester 1




                      Tester 2




         Figure 3: Configurations

34 Automated Software Testing Magazine                  www.automatedtestinginstitute.com                                            August 2009
4          Develop the                                                      • Error handling functions
                                                                            • Loading functions


           Function Library
                                                                            • Miscellaneous verification functions

                                                                               Core Business function exercise basic application functionality.
                                                                         For example, a function that is responsible for logging into the
Often this step is placed after the creation of automated tests (Step 5) application would be considered a core business function. These
through the following process.                                           functions can be identified from the requirements or from the manual
                                                                         test cases, related to the groups of tests that are slated for automation.
    1. Generating multiple test scripts with identical code,                   Most applications have several main navigation paths that are
    2. Identifying the redundant code,                                   taken many times during testing. And if the application modifies these
    3. Copying the redundant code into a function within a function paths, all developed automated tests that follow these paths would be
       library, then                                                     broken. Therefore, developing navigation functions is imperative.
    4. Replacing the redundant                                                                                        Error Handling functions
       code in each of those                                                                                          are created to deal with
       scripts with a call to the
                                                                        FUNCTION LIBRARY unexpected events that may
       function.                                                                                                      occur during testing. Some
                                                                                Contains                              events that will definitely
     This        approach       is                                                                                    need error handling include:
extremely inefficient, and
                                                                                     building
produces unnecessary rework.                                                                                          •      The failure of an
The more effective approach
                                                                                           blocks                     expected window to appear.
is to develop functions before
developing tests via the
                                                                                           for                        •      The failure of an
                                                                                                                      expected object to appear.
following process.
                                                                                       tests                          •      The failure of an
                                                                                                                      expected value to appear.
    1. Identifying redundant                                                                                          •      The appearance of an
       actions,                                                                                                       unexpected popup window.
    2. Creating functions for these actions within a function library,
       then                                                                    Loading functions do exactly as the name implies – load things.
    3. Adding calls to the functions when creating the test scripts.
                                                                         These functions load files and compiled modules for use by the
                                                                           automation tool.
      Most of the functions that will be necessary for the efficient
implementation of an automated test bed can be identified before                Although many automated test tools provide built in verification
you create a single automated test script. This is possible because        checks, it is often necessary and more efficient to develop custom
Step 1 involved reviewing documentation related to the software            Verification Functions to verify certain application attributes. Some
development environment, test cases and the AUT; in addition, it           user-defined verification functions that you may want to consider
may involve reviewing the AUT itself, if it is available. This review      creating include the following:
time should provide some insight into some of the repetitive actions
that take place while exercising and testing the application, therefore,
                                                                            • Window check
there’s no need to wait until the test script creation begins before
                                                                            • Date/time check
functions are created.
                                                                            • Login ID check
      Various types of functions that may be necessary include:

                                                                               When creating functions, it is important to identify pertinent
 • Core Business (Utility) functions
 • Navigation functions                                                    function arguments and return values.

August 2009                                    www.automatedtestinginstitute.com              Automated Software Testing Magazine 35
5         Develop the Test                                                through 3 in Figure 4 have been parameterized and placed in a function
                                                                          called “Login”, while steps 4 through 8 have been parameterized and


          Scripts
                                                                         Figure 5: Functional Decomposition interface
The final step in the framework development process is the creation
of automated test scripts. The automated test interface is largely a      placed in a function called “Verify_Screen”. Note: depending on the
                                                                          configuration parameters that are created, the username (“John”)
                                                                          and password (“JPass”), may be stored in the configuration file and
                                                                          delivered to the script via configuration parameters.
                                                                                Keyword frameworks reduce the technical nature of automated
                                                                          test development even further by allowing them to be developed in a
                                                                          tabular format. For example, the Keyword equivalent of the statements
                                                                          illustrated in Figure 4 might appear as illustrated in Figure 6.
                                                                               The columns in this illustration represent different elements of
                                                                          the test that collectively perform the desired actions:

Figure 4: Code-based interface                                             • Screen – The screen on which the automated test step occurs
                                                                         • Object – The object on which the automated test step operates
function of the type of automation framework and automated tool that
is in use. For Linear and Data Driven frameworks, the tests are           • Action – The keyword that is tied to a reusable function and
normally code-based, as illustrated in Figure 4, and                               identifies what activity will take place on the test step
therefore fairly technical. These tests are therefore                                  • Value – The value that may be entered into the
composed of code that is very specific to the
                                                                                          application
test script in which it exists. Some tools offer
a tree structure and/or “keyword” interface                                  locks            •        Recovery – The error handling actions
                                                                       ing B
                                                                 Build                       for this step
that adds a more graphical and icon-based
view of code, but when these tools are used                                                 •          Comment – A note about the step’s main
in a Linear or Data Driven framework, the
                                                                                                   purpose, used to provide helpful information
tests still follow the basic precepts of a
                                                                                                       to anyone reading the keyword file
code-based test.
      Tests in Functional Decomposition                                                                  The keyword file is interpreted and
frameworks are still code-based, but as                                                                   executed using a driver script.
the framework becomes more defined, the                                                                          Following all five of the steps
tests become slightly less technical. This is                                                             presented in this article from start
due to the fact that the tests are largely created                                                     to finish will aid you in creating an
by treating reusable components as building blocks,                                                 automated test framework that will provide
and stacking them together.                                                                          an environment in which your automated
      Figure 5 reveals how the statements in Figure 4 might be                                      tests will thrive across multiple releases of
written in a Functional Decomposition framework. Statements 1                                      the AUT.




     Figure 6: keyword Driven interface

36 Automated Software Testing Magazine                 www.automatedtestinginstitute.com                                         August 2009
Evolve, Adapt and build                               hand me the Key...word (Continued from page 10)
(Continued from page 4)
                                                            Lines 10 through 22 use the DT_Get_           function is successful, the return value will
     Finally, yours truly chimes in on the third      Value function to get the values from each of       be 0 (zero). If the function fails, the return
mode of change, defined by building up your test      the columns of the keyword file (the ‘DT_’          value will be 1. In the event of a failure
automation framework. This change mode has            functions are not native Ruby functions, but        Lines 43 through 45 call an Exception
no particular catalyst, but is instead unprovoked,    are instead custom functions).                      Handling routine that you develop called
requiring implementation by a project looking               Line 24 through 41 compose a                  utilityRecover. The recovery action defined
to improve their test automation return-on-           branching construct that interprets the             in the Recovery column of the keyword
                                                      keyword in the Action column of the                 file will determine what action is taken on
investment. I provide a step-by-step approach for
                                                      keyword file (as illustrated in Figure 3). This     a failure. This is an effective and efficient
building an automated test framework that will
                                                      construct is the heart of the keyword file in       mechanism for implementing global
ultimately help to sustain multiple automated
                                                      that it ties each keyword to the execution of       exception handling on every line of an
tests over an extended period of time, therefore                                                          automated test.
                                                      a specific utility function. In addition, this
helping your automated tests proactively keep
                                                      construct determines which columns will                  Line 46 closes the loop that cycles
pace with application changes.
                                                      deliver arguments to the function. Line 27          through each line of the keyword file, while
     So, while I may be biased, I do believe          and 28 provide an example of this concept.          Line 49 closes the keyword Excel file.
you will thoroughly enjoy the three featured          If the current keyword file row being read
articles in this issue. These articles, along with    by the driver script uses the “INPUT”                   DT_OPEN, DT_GET_VALUE and DT_
the department articles - focused on keyword          keyword, Line 28 will call the utility Input          CLOSE are not native Ruby functions. This
driver script development, open source tool           function, and will deliver values from the             are custom functions for manipulating and
                                                      Screen, Object and Value column into the                Excel file. For guidance on how to create
contributions, and automation blogging - will
                                                      function’s arguments.                                these functions, use the Data-driven Scripting
help you understand present day test automation,
                                                            Each utility function should be                               With Excel entry at:
and help us all define the next phase in automation
evolutionary development.                             developed to return a value that the driver          http://www.techniques.automatedtestinginstitute.com

                                                      script assigns to a variable called ‘ret’. If the
August 2009                                     www.automatedtestinginstitute.com                Automated Software Testing Magazine 37
hot topics in automation


Automation micro
Microblogging is a form of communication based on            including Internet browsers, mobile phones, personal
the concept of blogging (also known as web logging),         digital assistants (PDAs), and desktop client software.
that allows subscribers of the microblogging service         Microblogging is a powerful tool for relaying an
to broadcast brief messages to other subscribers of the      assortment of information, a power that has definitely not
service. The main difference between microblogging           been lost on the test automation community. See some
and blogging is in the fact that microblog posts are         of the recent Twitter messages below for an example of
much shorter, with most services restricting messages        how test automators have used microblogging.
to about 140 to 200 characters.
Recently popularized by Twitter, there are numerous
other microblogging services, including Plurk, Jaiku,
Pownce and Tumblr, and the list goes on-and-on
because there is a tremendous appeal to microblogging.
The appeal is in the brevity of the message (important in
today’s fast paced society), the ability to stay connected
in real-time with one’s interests, and the portability of
the microblogging services. Many of these services
allow messages to be sent and received in a variety
ways, including instant messages, text messages,
email, video clips and audio sound bites. In addition,
message can be sent and received via a variety of tools,

                                                                         Twitter name: sietstweets
             Do you know about the AA-                               post Date/Time: 5:09 AM Jul 30th
             FTT, a community committed                               usage: Sharing Tool Frustrations
             to better functional testing
             tools for Agile teams? See
             http://is.gd/1HUvz
                                                                          Working with #qftest for
                                                                          automated gui testing... just
                                                                          found its not going to work for
                                                                          webtesting on MAC OS X...
                                                                          bastards!


             Twitter name: testobsessed
         post Date/Time: 11:49 AM Jul 22nd
           usage: Community Introduction


38 Automated Software Testing Magazine       www.automatedtestinginstitute.com                              August 2009
oblogging is hot!
                                           Microblogging is a powerful
                                         tool for relaying an assortment of
                                                     information

                                                             Twitter name: fredberinger
                                                          post Date/Time: 11:39 PM Jul 21st
                                                               usage: Sharing Articles



                                                              Automated testing of production
                                                              environment http://bit.ly/kvduU
                                                              Good article, so true !




                                                                          Twitter name: TestComplete
                  Software Application Test                             post Date/Time: 8:37 AM Jul 31st
                  Analyst    Paris,  France!!!:                           usage: Course Announcement
                  London-London, Job Purpose:
                  • Automate test plans.. http://
                  ub0.cc/3R/3F
                                                                             We’ve just released our new
                                                                             TestComplete course onto the
                                                                             public schedule. Check it out;
                                                                             http://tinyurl.com/nkofpa



                 Twitter name: Monsterjobsuk
                post Date/Time: 8:08 AM Jul 29th
                    usage: Job Announcement


  August 2009                            www.automatedtestinginstitute.com   Automated Software Testing Magazine 39
i ‘B’log to u


latest from the B
Automation blogs are one of the greatest sources
to-date test automation information, so the Autom
Testing Institute has decided to keep you up-to-da
some of the latest blog posts from around the web
interesting posts, and keep an eye out, because yo
your post will be spotlighted.
                blog name: google Testing Blog                                            blog name: I.M. Testy
                     post Date July 1, 2009                                              post Date August 1, 2009
                post Title: The Plague of Blindness                                post Title: UI Automation Out of Control
                   Author: James A. Whittaker                                               Author: Bj Rollison

        Imagine playing a video game blindfolded                                 I will say that in general I am not a big
        or even with the heads up display turned                                 fan of GUI automation for a litany of
        off. You cannot monitor your character’s                                 reasons, but mostly because it is misused,
        health, your targeting system is gone.                                   or applied to tests that are simply more
        There is no look ahead radar and no                                      effectively and efficiently performed by
        advance warning of any kind. In gaming,                                  a person. However, I also understand
        the inability to access information about                                the GUI automation does provide value
                                                                                 when used in the right context. I am not
        the campaign world is debilitating and
                                                                                 against GUI automation, but I certainly
        a good way to get your character killed.
                                                                                 don’t advocate automating a test just
        There are many aspects of testing                                        because we think we can, or because
        software that fall into this invisible                                   we have some bizarre idealistic belief
        spectrum. Software itself is invisible.                                  that everything should be automated.
Read more at:
                             Read More at:                                                            Read More at:
        http://googletesting.blogspot.com/2009_07_01_archive.html                 http://blogs.msdn.com/imtesty/archive/2009/08/01/auto-
                                                                                                       mation.aspx


40 Automated Software Testing Magazine                     www.automatedtestinginstitute.com                                       August 2009
Blogosphere
 of up-
mated
ate with
b. Read below for some
ou never know when

         blog name: Collaborative Software Testing                                   blog name: Creative Chaos
                 post Date July 29, 2009                                               post Date July 31, 2009
         post Title: Fuzzing Through The Side Door                                  post Title: The Meme’s the thing
                  Author: Jonathan Kohl                                                   Author: Matthew

       I’ve been looking into testing with                                  Test-Automation, for example, is
       fuzzers lately, and finally got the chance                           very appealing to developers, because
       to do this on a live project. While there                            automation is what they do. It’s no
                                                                            surprise, when devs look at testing as a
       are a good deal of black-box fuzzing
                                                                            computer science problem, Automation
       tools out there, if you want to go beyond                            is the first thing to come to mind. So
       that you are often on your own. At the                               we have generation after generation
       other end of the spectrum, MSDN has                                  talking about test automation as the
       a nice article on white box fuzzing.                                 be all and end all of the test process,
                                                                            without ever having actually studied
                                                                            the very human, cognitive and
                                                                            communication process of software
                                                                            testing, nor having done any research
                                                                            on failure modes of software defects.

                          Read More at:                                                        Read More at:
           http://www.kohl.ca/blog/archives/2009_07.html                                 http://xndev.blogspot.com/



   August 2009                                  www.automatedtestinginstitute.com    Automated Software Testing Magazine 41
up-to-Date with ati




   T         his Week In Automation Series
                  This Week In Automation is an ATi produced video series that summarizes current

   events relative to test automation. with an often tongue-in-cheek approach to discussing the latest

   news in tools, events (both small and large) and concepts, this series serves as the video companion

   to ATi’s newsletter, while keeping you abreast of what is going on in the world of software test automa-

   tion. See the latest edition of This Week In Automation at http://www.twia.automatedtestinginstitute.

   com.




   S        ee No evil, HeAr No evil
                  Have you seen the latest podcast produced by ATi? if not, navigate over to http://

   podcasts.automatedtestinginstitute.com and listen to the latest podcast featuring an interview with Ms.

   linda g Hayes. in this interview, she discusses her article entitled “evolution of Automated Software

   Testing: An eyewitness Account”. in addition, she discusses her thoughts on the present and future of

   test automation in general.




   I      t’s NeWS To Me
           over the past few weeks ATi has brought the user community some interesting and very popu-

   lar news stories related to test automation. These stories include:

    • Microsoft Powerpoint defect Posing Security risk

    • espionage and Software: u.S. Vulnerable to Cyberspies!?

    • Microsoft Powerpoint Security Vulnerability resolved

   For these and other stories visit http://www.news.automatedtestinginstitute.com.


42 Automated Software Testing Magazine    www.automatedtestinginstitute.com                          August 2009
   C          oming Highly rATed
                  wanna know which of the hundreds of indexed tool and article archives are the most

   popular? well, here goes. For the purposes of this section, popularity is based on the number of hits

   and the rating received. Two of the most popular indexed tools on the site are FunFX and jSystem.

   Two of the most popular indexed articles are “Test Automation roi” by dion johnson and “Automating

   Software Testing: A life-Cycle Methodology” by elfriede dustin. be sure to visit http://www.tools.

   automatedtestinginstitute.com and http://www.articles.automatedtestinginstitute.com for the latest

   tools and articles, and be sure to rate them after you have read them!




   A           utomation Honors
                  Test automation is an integral part of ensuring the production of quality systems in a

   world where software development is becoming increasingly fast-paced. And one way to help elevate

   test automation so that it can be more effective is to celebrate it and its practitioners. Thus, the ATi

   Automation Honors have been created to celebrate excellence in the discipline of test automation from

   the corporate level down to the practitioner level. This celebration specifically pays tribute to:

    • Those that have displayed leadership in moving and keeping test automation in its proper place

      as a distinct iT discipline,

    • Those that drive innovation within the field, and

    • Those that display excellence in automation implementation, thus playing a big role in the delivery

      of a high quality product to customers and/or production.

    • Tools, individuals, publications, websites and more.

   The nomination period has reached a conclusion, and finalists will be identified by august 17, at

   which time voting will begin and commence through october 31, 2009. For more information on the

   Automation Honors, the nominating process or the honoree categories, visit http://www.atihonors.

   automatedtestinginstitute.com.


August 2009                          www.automatedtestinginstitute.com    Automated Software Testing Magazine 43
        How Do You Network?
        Whatever your choice, the Automated Testing Institute is there!



          YouT
                                                                             Twitter
                        Facebook                   Myspace




 Stay up-to-date with test automation by following the Automated Testing Institute on your so-
                                cial networking site of choice.




   Facebook           Myspace            Twitter           YouTube           LinkedIn   Blogger

        For more information, visit http://www.networking.automatedtestinginstitute.com
44 Automated Software Testing Magazine   www.automatedtestinginstitute.com              August 2009
Still Testing SAP Manually?




Worksoft Certify®   Get ahead of the curve and maximize the value of
                    your SAP investments with Worksoft Certify®, the
for SAP®            only code-free approach to end-to-end,
                    cross-platform business process validation and
Automate            automated testing.

Accelerate          Worksoft Certify® for SAP® is a specialized
                    lifecycle management solution for SAP that
Optimize            automates and optimizes SAP lifecycle tasks from
                    change impact analysis through functional testing,
                    performance testing and audit documentation.
                    Certify for SAP is unmatched in the industry for
                    accelerating time to value and reducing overall
                    project costs by up 40-80%.

                    Are you ready to test smarter and deploy faster?

                     Contact us at 1.866.836.1773 or visit
                      www.worksoft.com to learn more.

				
DOCUMENT INFO