VIEWS: 61 PAGES: 45 POSTED ON: 6/24/2011
An AutomAted testing institute Publication - www.automatedtestinginstitute.com AutomAted ....... SoftwAre teSting . AuguST 2009 $8.95 MAGAZINE Evolution of AutomAtEd SoftwArE tESting evolutionary 5 StepS to Framework roaD Development inTroduCing your ProjeCT How To CreATe A FrAMework To new TeCHnology For MAinTAinAbiliTy August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 1 Give me the key...worD: Building A Keyword driver Script in 4 StepS Still Testing SAP Manually? Worksoft Certify® Get ahead of the curve and maximize the value of your SAP investments with Worksoft Certify®, the for SAP® only code-free approach to end-to-end, cross-platform business process validation and Automate automated testing. Accelerate Worksoft Certify® for SAP® is a specialized lifecycle management solution for SAP that Optimize automates and optimizes SAP lifecycle tasks from change impact analysis through functional testing, performance testing and audit documentation. Certify for SAP is unmatched in the industry for accelerating time to value and reducing overall project costs by up 40-80%. Are you ready to test smarter and deploy faster? Contact us at 1.866.836.1773 or visit www.worksoft.com to learn more. AutomAted S t oftwAre eSting August 2009, Volume 1, Issue 2 Contents Cover Story Evolution of AutomAtEd SoftwArE tESting 14 Reviewing how test automation and its tools have evolved is useful only if it helps to understand where we go from here. To do that, you must appreciate the technology and market forces that have shaped them and why, and what that portends for the future. By Linda G. Hayes FeatureS EvolutionAry roAd 22 Software technology changes in existing applications often present roadblocks on the road to successful test automation. Learn what steps will help you navigate your test automation over and around these barriers. By J.L. Perlin 5 StEpS to frAmEwork dEvElopmEnt 28 Developing an automated test framework can greatly increase the ROI of your test automation efforts, but it can also be a daunting task. Read this article for a step-by-step framework development process. By Dion Johnson ColumnS & DepartmentS EditoriAl 4 hot topicS in AutomAtion 38 evolve, adapt and Build microblogging is hot! Keeping automation on pace with three modes of change. Test automation meets microblogging. AuthorS And EvEntS 6 i ‘B’log to u 40 Learn about AST authors and upcoming events. Read featured blog posts from the web. SwEAt thE tEchniquE 8 up-to-dAtE with Ati 42 Find out what’s happening on the Automated Testing hand me the key...word Institute’s Online Reference. Learn how to build a keyword driver script. Automated Software Testing (AST) Magazine is an Automated opEn SourcEry 12 Testing Institute (ATI) publication that serves as a companion Contribute to open Source projects in 4 Steps to the ATI Online Reference. Learn how to contribute to your favorite open source For more information regarding the magazine visit: projects. http://www.astmagazine.automatedtestinginstitute.com August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 3 editorial Evolve, Adapt and Build by Dion Johnson For anything to survive, it must be able to change itself by three different ways: evolving, adapting and building. This has been evident in the natural world, and it also holds true in the world of software. This reality should be especially clear for software testers and test automators, because the very mission and reason for their existence is to ensure systems are effectively evolving, adapting and being built. This conceptual trilogy is not confined, however, to only the systems tested using automation, but also applies to test automation itself. Evolution is defined as a gradual process to aid in this understanding, this issue of the frameworks. In addition, Hayes describes in which something changes into a different Automated Software Testing Magazine is where the automation evolutionary approach and usually more complex or better form. dedicated to exploring each of these modes has come up wanting, and proposes what the This change is normally an innate response of change. next test automation evolutionary response to some measured environmental stimulus In the cover story, Linda Hayes delivers should be. that occurs over time. Adaptation is the act of what is possibly one of the greatest articles Next, J.L. Perlin provides a featured adjusting oneself to perspective on a different conditions or a shifting ...possibly one of the greatest modern day test automation adaptation environment. This type of change is articles ever on the topic of the test with a peek into a project’s encounter automation evolution typically marked with contemporary by a deliberate and changes to the direct response software environment. to immediate modification in the environment. Building ever on the topic of the test automation In this article, an existing software application is the act of increasing or strengthening, and evolution. Titled “The Evolution of with a substantial suite of automated tests is may be a response to some outside stimulus, Automated Software Testing”, this article faced with changes to existing application but it could also be based on a self-imposed looks at software test automation from a components. These changes threaten to render decision to improve one’s situation. macroscopic point of view and explores the all existing automated tests useless as the The discipline of test automation has gradual changes in software systems over automated test tool contends with problems experienced and been shaped by each of the years. It takes us through a trip down communicating with the new components. these modes of change in the past, and software computing memory lane, moving us This is a problem that many automators have continue to be defined by them even today. from mainframe to Internet domination, and come across, and if you haven’t, there is Understanding these three modes of change explaining how the test automation discipline a good chance that you will. So J.L. Perlin will help to better understand the current state has responded to each evolutionary shift by offers an approach for effectively navigating of test automation, and will therefore help moving from text capture/playback tools your test automation through such modern us all to chart a course forward. Therefore, to the broad use of commercial automation day technological changes. (Continued on page 37) 4 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 How Do You Network? Whatever your choice, the Automated Testing Institute is there! YouT Twitter Facebook Myspace Stay up-to-date with test automation by following the Automated Testing Institute on your so- cial networking site of choice. Facebook Myspace Twitter YouTube LinkedIn Blogger For more information, visit http://www.networking.automatedtestinginstitute.com August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 5 authors and events Who’s In This Issue? linda g. hayes, in the lead feature, offers an AutomAted eyewitness account of test automation evolution. With over 20 years of experience in software development and testing, SoftwAre teSting Linda is the founder of three software companies including AutoTester, the first PC-based test automation tool, and Managing Editor Worksoft, Inc., pioneer of next generation code-free test Dion Johnson solutions. Linda holds degrees in accounting, tax and law and is a frequent industry speaker and award-winning author Contributing Editor on software quality. Named one of Fortune Magazine’s People to Watch and one of the Edward Torrie Top 40 Under 40 by Dallas Business Journal, she is a regular columnist and contributor to numerous publications, including the Automated Software Testing Magazine, Director of Marketing and Events StickyMinds.com and Better Software magazine. She is the author of the Automated Christine Johnson Testing Handbook, contributing author to Testing SAP R/3:A Manager’s Guide, and co- editor of Dare to be Excellent with Alka Jarvis on best practices in the software industry. A publICATIon of ThE AuTomATED TEsTIng InsTITuTE Her article “Quality is Everyone’s Business” won a Most Significant Contribution award from the Quality Assurance Institute and was published as part of the Auerbach Systems Development Handbook. You can contact Linda at Lhayes@worksoft.com. J.l. perlin offers a featured article that provides an approach for handling modern day technology changes. Perlin has over 20 years of manual & automated testing ConTACT us experience and has managed QA teams in both the AST Magazine public and private sector for organizations including email@example.com Lockheed Martin, Social Security Administration (SSA), General Electric, and the Centers for Medicare & ATI Online Reference firstname.lastname@example.org Medicaid Services (CMS). Mr. Perlin is currently a lead test automation engineer working on critical care systems for Philips Patient Monitoring Informatics. You can email Mr. Perlin at email@example.com. ATI and Partner Events dion Johnson provides step-by-step instructions for building an automated test framework in which you August 17 may develop and maintain your automated tests. Dion has ATI honors finalists Announced several years of experience in providing IT services to http://www.atihonors.automatedtestinginsti- both government and private industry in a manner that has tute.com demonstrated expertise in multiple areas of the software development lifecycle. With a Bachelor of Science August 17 - 31, 2009 degree in Electrical Engineering, he has spent much of ATI honors Voting period his professional career as a consultant in the areas of quality assurance (QA), quality http://www.atihonors.automatedtestinginsti- control (QC), software process improvements, requirements analysis and software test tute.com automation. As a regular conference speaker and presenter, Dion has delivered award winning and highly acclaimed presentations at many of the most prestigious industry conferences, including the StarEast and StarWest International Conference on Software Testing, Analysis and Review, the Quality Assurance Institute Conference and Better Software Conference. He also teaches college and business level classes relative to testing and test automation, and has several published articles in various IT publications. 6 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 7 Sweat the technique hand me the key...word Building A Keyword Driver Script In 4 Steps Often called “Table Driven”, a Keyword advantage is that it is easier to read keyword framework interprets automated tests that files than traditional scripts. Keyword files are developed in a tabular format using typically mirror manual test procedures, and a vocabulary of keywords. A keyword is the natural language of the verb-like phrases typically a verb or verb-like phrase that is makes reading a keyword file similar to associated with an external library function reading a collection of sentences. Finally, responsible for executing the application while the Keyword framework itself may commands necessary to satisfy the objective be more technical, the keyword files created of the test. during everyday application automation are Therefore, in a keyword framework, a a lot less technical than files with statements script that may normally appear as illustrated written in code. This allows less technical in Figure 1 will instead appear as illustrated individuals to effectively contribute to in Figure 2. automated test creation. These advantages don’t mean that we’ve escaped scripting altogether, however. We haven’t evaded scripting; we’ve merely changed the nature of the scripting that needs to be performed. There are at least two forms of scripting that remain: function because many of the functions are created development and driver script creation. in such a way to not only be reusable for If you purchase a commercial keyword multiple tests within a single application, but framework or obtain an open source keyword also for tests across multiple applications. framework, you may eliminate the need for Redundancy may therefore be decreased custom development of the driver script and Figure 1: Code-based interface across all applications that an organization may also reduce the function development to may be responsible for automating. Earlier only those functions that are specific to your Figure 2 illustrates a tabular automated script development also results from using application and necessary for more efficient test known as a keyword file, with the this type of framework, due to an increased automation. If you choose to create your own keywords that dictate the actions to be ability for automation to begin before the framework however, the script development performed shown in the Action column. application is delivered. Using information is solely up to you. There are several advantages to using gathered from corresponding manual test This article will focus on the development keyword files for automated test development cases, requirements or other documentation, of the driver script that interprets a keyword including increased reusability. Reusability is keyword files can be created well before the file. Following are steps for successfully increased with a Keyword Driven framework, application is ready for automation. Another creating and implementing a keyword driver Figure 2: keyword Driven interface 8 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 script: about the step to anyone reading the application under test 1. Create the keyword file structure keyword file. 2. Identify potential keywords Step 3. IdentIfy Keyword utIlIty 3. Identify keyword utility functions Step 2. IdentIfy potentIal KeywordS funCtIonS, argumentS and return ValueS 4. Create driver script The keywords created will largely depend Function definitions are a product of the on the functions deemed necessary for framework and of the automator’s personal Step 1. IdentIfy ColumnS In Keyword fIle implementation of the associated framework. preferences for how the function is to be The keyword file columns will vary from Not all of the keywords will be identified implemented. One of the keys in developing framework to framework. The only constant during framework design and development, an effective automation framework utility Keyword fIle drIVer SCrIpt The Driver ScripT callS funcTionS baSeD on The KeyworD in The acTion column Figure 3: Driver Script pseudocode and keyword File interaction is that a column will almost certainly exist but will rather continuously become apparent function, however, is to make sure it is for keywords to be identified. In addition, as tests are automated. If developed properly, independent, reusable, and able to be the keyword file may have one or more of the keyword framework will be seamlessly implemented as intuitively as possible. the columns illustrated in Figure 2. These scalable with respect to the addition of new Function arguments play a big role in columns are defined as follows: keywords over time, but there are some ensuring your functions met these criteria. • Action – Contains the step’s main generic keywords that may be identified For example, a function associated with the keyword, and identifies the action that upfront, including: INPUT keyword might need the following will be executed • INPUT – Inputs a single data item arguments: • Screen – Contains the name of the screen • PRESS – Presses or clicks a button or • Screen – This argument contains the on which the action is to be performed link screen on which the input action is to • Object – Contains the name of the object • VERIFY_SCREEN – Verifies a screen occur on which the action is to be performed has appeared • Object – This argument contains the • Value – Contains values that may be • VERIFY_FIELD_VALUE – Verifies object on which the input action is to entered into the application entered of selected field data • Value – This argument contains the value • Recovery – Contains Exception • VERIFY_TEXT – Verifies dynamic to be entered into an object Handling keywords. screen text These arguments correspond to columns in • Comment – Contains helpful information • INVOKE_APPLICATION – Opens the the keyword file shown in Figure 2, therefore, August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 9 ati newsletter The ATI Newsletter keeps you abreast of the latest information relative to our Online Reference. Sent to registered members of the site on a bi-weekly basis, this newsletter keeps you informed on the follow: ATI’s featured Content • Featured Videos • Featured Tutorials • Featured Articles • Featured Tools • Featured Publication • Featured Poll • Featured Forum Post popular Content • Popular Tools • Popular Articles Automation omg! • Interesting News • Current Events Figure 4: Sample Driver Script the arguments will be fed values from these If the driver script is developed using Ruby, it three keyword file columns via a Driver may appear as shown in Figure 4. Register on the ATI site today to Script. Line 1 loads the UtilityFunctions.rb file. receive this email newsletter. To This is a file you develop that contains your register visit: Step 4. deSIgn and deVelop drIVer SCrIpt user defined utility functions. www.registration.automatedtestinginstitute.com As illustrated in Figure 3, the driver script Lines 3 through 7 are responsible for is responsible for calling the keyword file, setting up and initializing the keyword file reading all of the steps, and executing utility which is developed in an Excel spreadsheet. functions based on the keywords in each step Line 9 begins the loop that reads all of of the keyword file. the rows in the keyword file. (Continued on page 37) 10 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 Tired of searches that return irrelevant information? google - AutomAted teSting SeArch engine The Automated Testing institute has partnered with google to create a software test automation search engine. if you’re looking for software test automation information, you will get the best results by using this search engine, because it only searches the sites that matter most to automators. you can also restrict searches to blogs and forum sites. www.googleautomation.com August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 11 open Sourcery Ask not what your tool can do for you Learn to Contribute to Open Source Projects in 4 Steps Ask not what your tool can do for you. interested in automated software Ask what you can do for your tool. testing or interested in one of the My fellow automators, how often authors writing about automated have you pointed out the shortcomings software testing. Either way, this is of tools with which you have worked? a good indicator that a project that Now, how often have you sought to maintains an automated test tool may take an active role in fixing these be a good fit for you. shortcomings? Aside from simply picking a project I’m sure many of you are thinking, because you like it, you may also “Fixing tool shortcomings is not my want to pick a project that is modest in job! I use these tools to help me better size; particularly if you are a beginner perform my job. If I spend time fixing to open source development. Smaller tools, then I won’t have any time to do projects typically have smaller code my real job!” bases, provide easier access to other I hear all loud and clear, and you developers, and have simpler bug make good points, but while fixing you on how this may be accomplished. In our submission processes. Therefore, tool shortcomings is not your responsibility, research we came up with 4 steps. smaller projects may make it easier for you to it does have several benefits. Fixing tool quickly become productive. shortcomings is normally not an option with Step 1 – ChooSIng a projeCt Another option is to start your own open commercial tools, but the beauty of open There are numerous open source projects source project. Maybe there is a browser source tools, is that it is very much a viable extension or automated test tool plug-in that ready and willing to receive contributions option. you’d like to see created to support automation from eager scripters, and the project list The May 2009 issue of the Automated of a particular technology; if so, you can lead grows with each passing day. Following are Software Testing Magazine contained an the effort in developing the desired extension links that identify and provide access to some editorial by Dion Johnson called, “Scripting or plug-in. Just search around first and make of these projects: Calisthenics”, and this article presented sure a project doesn’t already exist for what various options for keeping automation you intend on doing. Not sure how to create skills sharp during periods of inactivity. • www.sourceforge.net a project? Some pretty good instructions are The article was referenced in various • www.opensourcetesting.org laid out at ehow.com (http://www.ehow.com/ automation Internet forums with a question • www.code.google.com/soc/ how_2091738_start-open-source-project- posed to the forum’s community that sourceforge.html) for creating an open source asked, “How do you keep your skills The challenge is now in determining project with SourceForge. sharp during periods of inactivity?” One which project you should contribute to. The of the community’s users suggested that simplest way to pick an open source project Step 2 – pICKIng your ContrIbutIon contributing to open source projects is a good is to look for one that maintains a tool that Now that you’ve picked a project, the next way to keep scripting skills sharp. We here you use and/or are interested in. For example, step is determining what you’re going to at the Automated Testing Institute thought given that you are reading the Automated contribute to that project? You may begin by this was an enterprising answer and decided Software Testing magazine right now, it asking yourself what shortcoming you’d like to research and provide information for all of may be safe to assume that you are either to strengthen, and determining if it’s going 12 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 to be a ‘fix’ or a ‘feature’? In other words, important to do it with a productive tone. or it may be as complicated as modifying do you want to fix an existing bug, or do you One way that you’ll be able to communicate several code modules. However extensive want to add a new feature? At this point in with people on the project is by joining the your scripting task, be sure to use the code the process you may have already decided project’s chat sessions and/or mailing list if conventions defined by the project, including what you want your contribution to be, and they have one. Mailing lists are often used conventions for indenting, variable naming, that decision may have played a big part in for discussing recent code changes or to and commenting. the project you chose to join. If not, however, propose new code changes, so they offer a Upon completion you need to post there are some tips you your changes to the can use for picking a contribution. ...if key players on the project don’t project, a task that is accomplished The first and most obvious thing to do is like the ‘new guy’, there is a good differently different projects. for chance that your proposals will get to actually use open Some projects source tools. There are submit changes via a bunch of them that the mailing list, perform a wide range rejected and sometimes the of functions. If you project uses a version work on a computer control system. at all, you should have no problem finding at great avenue for proposing your changes and Whatever vehicle you use for sharing your least one open source tool that can help you soliciting feedback. Making your intentions update, that update is normally presented in with some of your tasks. Pick one, and then known to the key group members early will the form of a patch. A patch is a small piece identify some of the features you’d like to see improve the chances of your change being of software that contains your fix or new changed. What you come up with, could be accepted, or it may allow you to receive feature for a computer application. One way your contribution to the project. feedback explaining why your proposed to create a patch is to run what’s called a diff In addition, some projects maintain change should be reconsidered. program. The diff program compares original readily available “to-do” lists that identify Using the bug system to study existing files or directories against modified files items that need to be fixed. Also, the project’s bug reports is also a great way to learn the or directories, and produces an output file bug reporting system can be used for culture of the project, because the reports called a patch that contains the differences. gathering ideas for what to fix. indicate who the key developers are and what Upon submitting the patch, be prepared to types of contributions normally get accepted. make changes as suggested by the project Step 3 – learn the projeCt Culture Aside from the bug system, you’ll want to maintainers. For information on creating and Much like countries and commercial appropriately utilize other project tools, such submitting a patch, visit ehow.com (http:// businesses, each open source project has as the version control system. Developers www.ehow.com/how_2091741_make-patch- its own unique culture, and your success in on these projects often have little patience open-source-project.html). a project largely depends on your ability to for individuals that can’t seem to follow Before submitting your patch to the learn and fit into that project’s culture. As rules, standards and directions. To obtain open source project, be sure you understand much as you may like to believe success is information on tools and conventions used the open source licensing used by the project only about technical abilities, it unfortunately by the project, locate and read the developer to ensure the patch is not rejected for legal goes beyond that. No matter how good your docs. reasons, and to ensure any copyright you may coding skills are, if key players on the project wish to impose on your developed code is don’t like the ‘new guy’, there is a good Step 4 – Code and poSt preserved. chance that your proposals will get rejected. At this point, it’s time to get down to the It is, therefore, important for you to be business of developing your fix or feature, ConCluSIon aware of who the core maintainers are, who and it’s imperative to begin by making sure Therefore, my fellow automators, if you the key contributors are, and who the more you have the most up-to-date project code. find yourself complaining about what your vocal members of the project are. You’ll also Next, search the code for the location in test automation tool is not doing for you, want to study how they communicate with which the change is to be applied – a task ask yourself, “What am I doing for my test one another. This will provide insight into that is often simplified by using an Integrated automation tool?” If more people had this who you may want to reach out to, and how Development Environment (IDE). Now you mentality, we’d all be a little more skilled, best to reach out to them. It will be important can begin scripting. Your scripting may be and the discipline of test automation would to make yourself known, but it’s equally as simple as changing a line or function, probably be a lot better off. August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 13 Evolution of AutomAtEd SoftwArE tESt An Eye Witness Account by linda Hayes 14 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 Test automation has a history of failure. Despite the explosion of n software into every aspect of our lives, only a small percentage of tests are actually automated today. Not that there aren’t pockets of success or flashes of promise, but after more than a quarter century of investment by vendors, customers and consultants alike, manual testing predominates. So reviewing how test automation tools have evolved is useful only if it helps to understand where we go from here. To do that, you need to appreciate the technology ting and market forces that have shaped them and why, and what that portends for the future. For that reason, this article sets the context of each evolutionary period within the state of software development and the applications market at the time. Finally, test automation is a very broad topic, so when I say test automation I am referring specifically to the automation of an otherwise manual test against a user interface (UI). This does not mean that test tools cannot be used for non-UI testing, or that there are not many other types of testing that are automated by a wide array of approaches and tools, only that they are outside the scope of this article. August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 15 Text Capture/Playback arose around dynamic content, like date and time stamps or frequently changing data fields. Timing synchronization was also problematic, especially as the user community expanded and performance Mainframes dominated the early enterprise landscape. The mainframe fluctuated. Whenever the system changed or ran slower than the was a monolithic environment, closely guarded behind glass walls original recording, context was lost and errors occurred. and accessible only to a few. Computing resources were scarce and expensive, so development processes were formal and highly But the killer issue turned out to be maintenance. If you recorded structured, requiring flow charts and extensive documentation before 100 transactions, there were 100 sets of the keystrokes and screens, being committed to code. The majority of software was custom so any change had a wide ripple effect. It was usually easier to re- developed for back office applications such as accounting and record the test than it was to fix it, which defeated the value premise payroll. Annual release cycles were the norm, usually punctuated by of automation and encouraged reversion to manual testing. emergency patches. The first commercial test tools emerged as a result of the transition So early on, maintenance loomed as a primary obstacle to test from batch to online applications, when multi-user terminal access automation success, with dynamic content and timing close behind. first became available and manual testing became a necessity. These tools recorded the keystroke inputs and character-based screen outputs of a terminal session, then replayed the keystrokes and compared the Text Capture/Playback/Script screens. Computing soon spread to mid-range systems, but the real breakthrough came with the announcement of the IBM PC. Personal This approach was seductive. It sounded so simple: just do the test as computers had been around for a few years, but these 8-bit platforms usual but record it into a script file, then play the test back as needed were generally perceived as truly “personal” computers used strictly and compare the results. Unfortunately it was also deceptive. Issues by hobbyists and a few propeller heads (the early term for geeks). But with the imprimatur of Big Blue, PCs with 16-bit power entered the corporate computing world. Application development continued to expand on the larger platforms, but software for PCs was initially limited either to productivity tools such as spreadsheets or word processors, or to terminal emulators that turned them into smart terminals. The DOS operating system sporting a character-based interface, was single-threaded and severely limited on memory and storage; the first machines didn’t even offer hard drives, so functionality was constrained. The increasing popularity of spreadsheets and word processors propelled PCs into the corporate community, and the use of terminal The emulators allowed them to replace “dumb” terminals. This drove manual testing from the mainframe to the PC. Soon automation tools Archaeological Dig Mainframe domination ibM PC Announced System Evolution Automation Evolutionary Response Text Capture/Playback emulator drivers 16 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 appeared for the PC, capable of driving these emulators and other PC- could use our tool because she was not a programmer. based applications, and these began to replace the mainframe-based capture/playback tools because they were cheaper and more powerful. Until she uttered those words it had never occurred to me that all testers weren’t skilled at scripting. After all, Petroware had been a One of these tools was from my company, AutoTester. Our tool was software company and we wrote code for a living. Although we sold originally developed by my first company, Petroware, when we ported AutoTester as enhanced capture/playback/scripting, this was just a our oil and gas accounting software from an early IBM desktop to euphemism for programming. And the stark fact of the matter is that the PC. Designed as a “session recorder” by my brother to automate most manual testers are subject matter experts, not programmers. his programming tasks, we quickly realized we could use the tool for many purposes including demos, training, task automation and So in the end we traded one set of problems for another: we addressed testing. But the more we used it the more we found we had to enhance the maintenance, dynamic content and timing issues but introduced a it with additional commands to perform logic and branching as well level of skill and complexity that excluded the majority of our target as to handle variable data. users. Before long, we had developed a proprietary language around the I was devastated. Secretly convinced we had developed an essentially recording that enabled the scripts to handle timing and synchronization, defective product that tormented our own customers, we started make decisions based on results, and externalize data into files. Now a project to try to reduce or eliminate the programming aspect by a single script could loop through 100 records from a file and execute generating scripts, and while we made progress it soon became reliably regardless of system speed or dynamic content. Because it academic. Windows was born. also provided more power and control through its scripting language, increasing reliability while reducing maintenance, we consistently replaced older capture/playback tools and handily won competitions GUI Capture/Playback/Script for new customers. The release of Windows brought even more opportunity but also more Yet in spite of our early success, nagging concerns persisted. Even challenges. As a multi-threaded operating system it resolved many though we thought we had addressed the maintenance, dynamic low level issues; the maximum memory limit exploded and resources content and timing issues caused by capture/playback, we still were managed across sessions. Windows also employed an API layer observed customers eventually reverting to manual testing. I was first that converted mouse events into operating system messages that were puzzled then increasingly alarmed. What could be causing this? intelligible and returned screen contents as a mix of objects that could be queried to return text and images. Recorded scripts became more I got my answer from a customer who sent their test manager to our legible and reliable and the extra memory headroom enabled even offices for training. She was ideal for her job: she had over 20 years of more advanced functionality. experience with the application that was being rewritten and knew the functionality inside and out. But after only a few hours of training, she Another dynamic entered the mix as well. Personal computers were began to show signs of discomfort, and on the second day she openly becoming not only ubiquitous but operating platforms in their own burst into tears. Horrified, I asked her what was wrong. She explained right. Instead of just smart terminals for mainframes or personal that she was afraid she would lose her job since there was no way she productivity tools, they offered powerful standalone applications. Client/Server becomes the internet Takes Center windows released norm Stage gui Capture/replay Custom Frameworks Commercial Frameworks August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 17 This transition from mainframes to personal computers caused a slow lane as scripters took over automation. corresponding shift from a formal and ponderous development process to a more flexible, free-wheeling approach. Mainframes were so Custom Frameworks costly that few had access to them; personal computers were so cheap and plentiful that the masses could afford them. This led to software The PC continued to gain computing power, moving to a 32-bit companies being born in garages by kids barely out of high school or architecture, and Windows matured into a more stable and capable operating system that was a virtual monopoly on desktop machines. Client/server application development became the new norm, ...initial apparent success with re- allowing PCs to offload tasks from backend mainframes and servers and provide users with ever-easier yet more powerful interfaces. The cording messages instead of key- battle cry began to replace expensive mainframes with networks of strokes was soon replaced by disap- PCs and servers. pointment and failure... But the early days of client/server application development introduced even more issues for test automation. Commercial development toolkits became available, offering libraries of sexy new graphical controls that augmented the standard Win 32 object class library. college with no patience or need for the constraints of formality. Other component libraries also appeared offering everything from calculators to calendars to spell checkers, allowing programmers to The type of applications changed as well. Whereas previously most quickly assemble rich user interfaces with less and less effort. applications performed back office tasks that were limited to a few internal users, newer applications expanded their reach to front line The problem was that many of these new controls were non-standard, operations such as order entry and customer service. Many of the so the standard Windows messaging layer was no longer applicable. A original custom applications were replaced by commercial, off-the- custom control did not necessarily implement the same messages that shelf systems and new development moved into areas that interacted the standard classes did. Testing tools that relied on the Windows API more directly with core business processes. began to fail, diverting more and more automation time and effort to dealing with custom controls and other non-standard behaviors. As a result, development cycles began to collapse. Market pressure began to dictate functionality and annual delivery cycles gave way to This new complexity raised the stakes. Now automation engineers quarterly releases. More capability was needed faster and faster. This were thrust into the development environment, forcing them to added even more demand for test automation, because manual testing learn and deal with the underlying implementation of the interface. was less and less capable of keeping pace with changes and the risk Whereas previously one had only to be comfortable with using logical profile headed north. constructs and data file processing, now it was necessary to delve into object methods and properties and grapple with installing source This new, more hospitable technical environment and market also hooks or writing test APIs. This effort siphoned time and resources brought new competitors. Frankly, I was hoping that the latest market away from actually automating tests and into a struggle just to interact entrants would know something we didn’t and show us how to assure with the application’s user interface. that our customers were successful over the long term. But the newest tools presented the same problems. Scripting became even more It could not have come at a worse time, as pressure on development sophisticated to deal with ever more robust applications, and initial continued to increase. The new client/server applications penetrated apparent success with recording messages instead of keystrokes was mainstream operations whose delivery schedules were driven by soon replaced by disappointment and failure due to the skills and user demand and stability became even more important. Greater and effort required to write and maintain ever more complex script code. greater functionality was needed and it had to be delivered faster and better. Quarterly cycles soon gave way to monthly. Windows didn’t save test automation. More and more companies invested in automation to meet this demand Presented with this reality, the most successful tool vendors and automation engineers began looking for ways to reduce the time aggressively enlisted partners and consultants who could offload the and effort to code scripts. It became clear that test scripts were no scripting effort from the manual testers. A new class of tester was different than source code and adopting reusable components made created: the automation engineer. Whereas previously manual testers sense as a strategy to reduce development and maintenance. Thus were courted with capture/playback, now they were relegated to the frameworks were born. 18 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 Large companies soon found evolution in thought themselves with multiple Secretly tools and frameworks, some convinced we had of which had been rewritten developed an essentially more than once. defective product that tormented Complicating this was the fact our own customers, we started a that shrinking development project to try to reduce or schedules and expanding functionality meant more and eliminate the more tests were needed. There programming was simply not enough time aspect... and people available to write, maintain and support one or more frameworks while also developing, executing and maintaining the tests themselves. Eventually the overhead became so heavy that companies – again – Originally frameworks provided a basic infrastructure for test reverted to manual testing just to keep up with delivery schedules. execution, handling common tasks such as data file processing, error and test logging and reporting. But they gradually became more Commercial Frameworks robust, providing simplified interfaces that employed spreadsheets and flat files to allow non-technical testers to construct automated By now the Web was taking center stage as a ubiquitous user interface. test cases using pre-built script components. This introduced a role- This was both good and bad. It was good because it provided a basic based approach that allocated tasks based on skill sets and enabled set of standards for access through browser APIs and HTML classes, cooperation between automation engineers and manual testers. but bad because the ability to instantly deploy changes meant delivery cycles could be daily. And, as static pages and controls gave way to A number of framework types emerged, differentiated primarily by dynamically scripted ones, object names deteriorated into gibberish how the functionality was decomposed into components. The most and behavior became unpredictable. prevalent was referred to as “keyword” or “action word” based and it entailed writing script code components around business tasks, Pressure to deliver automation faster soared. Several enterprising such as “Create Order” or “Invoice Customer”, then allowing users automation engineers realized that they were constantly re-inventing to invoke these from a spreadsheet and pass in data values. Others the same framework over and over and decided to commercialize organized their components around screens or windows, such as “Add their solution and market it as a library of pre-built components. This Customer Information” or “Delete Customer Information”. helped to accelerate the implementation curve and reduce the overall costs. But depending on the type of framework, there was still a fair These frameworks succeeded in reducing overall development amount of application-specific code required to be developed, usually and maintenance while allowing non-coders to design and execute complicated by the custom control or dynamic content challenge. automated tests more easily, but still required custom code. Test designers could write linear test cases but could not make decisions One of these framework types – my favorite - was designed around and control the flow of the test based on real-time execution results. object classes. Commonly referred to as “table-driven” or “class- That still required code. action”, this approach sought to reduce or remove application-specific code by using components at the object class level: For example, But frameworks fell victim to their own success. When the architect “Press Button” or “Input Text”. A key advantage was that the same and chief (often only) automation engineer left the company or moved object classes and actions could be used across applications that used to another area, his or her replacement almost inevitably believed the same class library, allowing greater reusability with less code. It it would be easier and better to rewrite the code left behind than it was even possible to share libraries across companies. would be to maintain it. With no de facto standards, almost every automation engineer adopted their own pet approaches and designs. As the amount of code that needed to be written continued to August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 19 decline, the battlefield moved to the objects themselves. Virtually all • Test automation architectures will continue to shift from code modern test tools rely on an inventory of the application’s testable to data. The only way to keep up with the blistering pace of objects. Variously referred to as an “application map”, “GUI map” development will be to minimize or remove altogether the need or “object repository”, this inventory contains a list of each object in for custom code and accelerate or automate the acquisition of the the application’s interface including its name, class and the attributes application map. used to identify it. This map enables an object to be added to a test, • Test automation penetration will continue to decline - unless located and acted upon during execution, and provides a central point something changes. The accepted rule of thumb says that 30-40% of maintenance in the event of changes. or even 50% of a project should be budgeted for testing, but that assumed old-fashioned coding, not snap-together systems. There But as previously noted, object inventories were becoming is no way that hand-crafted test scripts can keep up with high- increasingly dynamic and unpredictable. Building the map was a slow speed application production. Nonsense names, closed controls and unpredictable context will disap- pear when developers are both accountable and rewarded for success- ful test automation. and tedious process, often requiring the engineer to navigate to each So what has to change? In my opinion, our only hope is for test page and objects. But with dynamic pages creating scripted objects automation to be integrated into the development process. No on the fly, using names that weren’t just incomprehensible but also application should be deemed complete without a test harness. Let’s changing depending on user choices, session parameters or a host of make developers responsible for making their software testable instead other factors, this was often an impossible task. Without a predictable of engaging in a constant struggle to compensate for their latest bad and reusable object inventory, automated tests are unusable. Trying behavior. Nonsense names, closed controls and unpredictable context to maintain them with daily releases was crazy enough, but if the will disappear when developers are both accountable and rewarded attributes changed within a release without any predictability, it was for successful test automation. simply impossible. Such a harness would provide a test object inventory and interface So history shows us that every time test tools and implementation with defined methods and properties to be shared by developers and approaches are enhanced to meet the latest challenge, new ones are testers alike. It would be kept current with all software changes and be introduced. Vendors and customers both find themselves in a perpetual part of a holistic test process that includes automated unit, integration, state of catch-up, seemingly doomed to repeat the past by continually system and user acceptance. reverting to manual testing. Test tool vendors needn’t become extinct; just shift their focus to What’s Next managing data instead of code. Automation engineers needn’t become an endangered species; just shift their focus to managing the object inventory and interface metadata. But don’t react just yet; history also The only way to break this cycle seems to be to anticipate the future tells us that changes take decades. Even today there are companies and be ready for it, instead of reacting to it. The harder question is to who believe capture/playback is a viable automation strategy, though know what’s coming. Since I don’t own a crystal ball, all I can do is we have known better for almost 30 years. look at history and current trends to project what’s coming. But until management sees the light and directs development to play History says that: their part, we all need to continue finding ways to reduce the amount of test code that must be written and maintained, work closely with • Development will continue to accelerate. Software will development to correct testability issues and manage the impact continue to insinuate itself into every aspect of our lives and of changes, and educate management that the historical ratio of jobs, driving demand for more functionality faster. Companies development to test effort is outdated but that the extra investment in will adopt packaged applications when possible. Developers automation is worth it. And keep trying. will take advantage of increasing levels of abstraction to keep The Automated Testing institute (ATi) conducted a up, using code libraries, code generators, web services and other middleware technologies to quickly assemble composite podcast interview with linda Hayes. To listen in, visit applications. http://www.podcasts.automatedtestinginstitute.com. 20 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 real adventures in automation Submit your Stories! Now’s Your Chance to be Heard Hav e so me thin g to say ? o listen ATI i s here t Ad The Automated Testing institute believes that one of the best ways to improve upon test automation is to listen to the lessons learned by others. So, if you have an interesting story that you’d like to share with the world, contact us at firstname.lastname@example.org to find out how. August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 21 Navigating Existing Evolutionary Frameworks through Modern Day Technological Evolutions by j.l. Perlin Road 22 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 it is a natural occurrence for products to evolve over time. Advancements in technology and consumer demands spur radical changes in design that make products more efficient, easier to use, and less costly to produce. How can you prepare test automation to best handle these changes? it is becoming increasingly essential to plan for change when deciding upon an automation approach. in order to do this it helps to understand why these changes are occurring so we as test automation engineers can understand the catalysts behind these changes and be more prepared not only for these new technologies and their effects on automation, but also for the interactions we will have with the organizational stakeholders who have introduced these new improvements. Evolution of a Product So, why do products evolve? let’s use the automobile as an example. The first self-propelled road vehicle was a steam-powered military tractor invented by the French engineer Cugnot. it was a ground-breaking invention, but, alas, due to low speeds (2.5 miles per hour), operating dangers, and the need to stop every ten to fifteen minutes to build up steam, it was not an improvement over the horse speed of the vehicle was still slower to the introduction of hybrid vehicles and buggy. than the horse and buggy, the and all-electric cars, with steps The next attempt at making range of the vehicle was very short made to bring us closer to the day a successful automobile was in (about 15 miles before a recharge when vehicles we drive every day the early 1800’s when a dutch was required), and the cost was can run on hydrogen. but, why engineer used an engine that ran on exorbitant and out of reach for most did the motor vehicle evolve in this rechargeable batteries to design the people of that era. manner? what was the catalyst for first electric vehicle. The vehicle was Fast forward to today, and new improvements and changes initially successful due to a smaller we see that over the years cars to the overall design? it is the and lighter weight engine that have become much more fuel same underlying factor that causes allowed the vehicle to travel faster efficient, safer, and easier to drive. other products to evolve, including and further than its predecessor. improvements to electric and computer systems and software, despite the improvements, the top- combustion technologies have led and that factor is market demand. August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 23 How technology Evolution efficiently improves upon the proverbial forward and the number of products that have Affects Test Automation horse and buggy. Changing major or even been made available to development teams to Technologies of the Information minor pieces of an application will most help them create large, complex applications Age are evolving at a rate of speed that likely require changes to other components of in less time. Automated test tools often have THE TRUE CHALLENGE FOR SOFTWARE TEST ENGINEERS IS DETERMINING WHETHER OR NOT THE CHANGING TECHNOLOGY AFFECTS THE WAY IN WHICH THE PRODUCT IS TESTED. increases exponentially with each passing the system. And all of these new and updated issues when interacting with newly day. Applications are growing in size and components need to be tested, along with released third-party custom complexity. One major software component the legacy components that aren’t changing controls and components, that has undergone massive changes is the to ensure that they still perform properly and these have become operating system. Much like the automobile under the new design. The true challenge very popular in recent engine, operating systems have undergone for software test engineers is determining years. Although very useful numerous changes since their inception in whether or not the changing technology to developers, they can be a major order to meet consumer demands. These affects the way in which the product is tested. problem for the demands have led to changes that In addition, we must determine if we can have made operating systems use the same tools we used prior to the increasingly efficient, faster, technology change to test these new intuitive, scalable, and more components, or if we need to look at secure while providing powerful new tools and/or a new approach. features and useful functionality Just like the products themselves, previously unavailable to users. these tools need to be more efficient, It seems pretty clear cut, no? Changes easier to use, and affordable. in market demands spur innovations in But most technology. What, however, does this mean for software t e s t automation engineers. There may also be issues with development technologies of all the like .NET, Windows Presentation automated tools need to Foundation (WPF), ActiveX, etc., that are be able to access the new components being used by your developers to design the of the application and have the ability to bulk of your application. The majority of the engineers perform all of the methods against the test time we cannot find a tool that can address all that test objects that our tests require to fulfill their of our needs, but at times we do find one that these evolving objectives. As technology moves forward, so can handle the majority of the tasks at hand. applications? It means must our testing approaches, processes, and tools. A Modern-Day that we have to lift the Evolutionary Scenario hood, kick the In today’s environment finding a tool tires and take this new that meets your project requirements can be Imagine that you’re working at an technology for a test a difficult task, especially when you consider organization and have been there for a couple drive to ensure it effectively and how quickly technology has been moving of years, and during this time you and your 24 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 team have been building and maintaining a your concern was centered on testability. The posed similar questions on how to automate suite of automated tests to execute against the first few controls presented by development tests for these controls but, unfortunately product that your company markets. When were proposed for use in portions of the lacked any useful responses. Some of these automation was first introduced to the project application that were minimally accessed by threads were several years old, so it was new test automation resources were hired users, and that required only a small set of becoming apparent that the tool vendor (including you) and the first assignment was tests for validation. Worst case scenario: If probably had no plan to release any add-ins to evaluate the AUT to identify the test tool your automated tool couldn’t handle these for this technology. You next contacted the requirements. Most importantly you wanted controls you may have only had a handful of manufacturer of these 3rd-party controls to find a tool that could interact with all of additional tests to execute manually. Then to see if it was possible to further explore the different components of the application, the other shoe dropped. A new grid control automation possibilities with your tool. The allowing you to automate more tests and was presented that would be accessed by all manufacturers’ response was that this was exercise more of the AUT functionality. You of the product’s users and a large portion theoretically possible, but that the controls looked at a number of different tools and of your automated tests. In other words, if do not officially support your automated compared them using a set of pre-determined you’re automated test tool couldn’t handle testing tool. When you asked how to make criteria/requirements. this control; the majority of your automated the controls accessible by the automated tool, Upon completing the evaluation you tests would become useless. You gently you were only informed that a few of their selected the test tool that would provide the but firmly communicated your concerns customers had attempted automating tests most value to your project, identified an regarding the affect that these new controls against these controls but their results weren’t automation approach that best suited your would have on the existing automation readily unavailable. Hmmm… someone was project, and then began automating the smoke suite so the developers agreed to send you trying to sell something here. and regression tests. Eventually your team a sample grid allowing you to evaluate how It was time to find other options for IT WoULD STILL BE UP To THE TEST TEAM, HoWEVER, To DETERMINE WHETHER OR NOT THESE NEW CONTROLS WERE AN IMPRoVEMENT oVER THE PRoVERBIAL ‘HoRSE AND BUggY’ built a large suite of tests that you designed your automated test tool would react to the making your tool adapt to the new technology. with great deference given to maintainability, newly proposed grid controls. and you felt pretty comfortable knowing Option 1: Extensibility – Some automated Researching Solutions testing tools offer extensibility, which means that most updates and fixes made to the application could be addressed easily using that you’re able to integrate the tool with the After receiving the sample grids as your well-planned framework. new controls by executing small snippets of promised you began trying to access the custom code via calls from the test script. After several new versions of your controls using your existing automated This solution is often the last resort, because application were released the development testing tool, and that’s when you learned the it could require extensive setup, coding, and team started looking at new 3rd-party bad news; the tool didn’t fully recognize the almost always requires assistance from the application controls that could be used grid components. It recognized that a new development team. to improve upon the current design. In a control object existed, but couldn’t access meeting, the development team presented any of the rows, columns, menus, or buttons Option 2: Workaround – Another possible the proposed controls, and these controls contained within this new grid control. Was solution might be to identify a workaround appeared to provide required application there a way to solve this issue? The first step in the application. Was there another screen functionality and also looked much nicer than you took was to contact the automated test in the application that you could be used to the previous, less powerful controls that were tool vendor to see if they supported these accomplish the same goals? Was there a presently in use. It would still be up to the test controls. To this they responded “No, but way to navigate around this control while team, however, to determine whether or not you can search our knowledge base to see accomplishing the same test objectives? If these new controls were an improvement over if anyone has done this”. Searching their so, this would allow the automated tests to the proverbial ‘horse and buggy’, therefore knowledge base turned up several threads that execute, but you and the developers would August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 25 shoes. You’ve done your research and have which way to Maximum roi? discovered this cool 3rd-party control that will fulfill the new and existing system requirements and/or is a major improvement over the previous control. You’ve performed your evaluation, probably of proof-of- concept, and have presented this idea to the rest of the development team. Now some automation engineer is walking in saying it shouldn’t be implemented because it will break the majority of the automated tests. That isn’t going to happen without a fight. Offering Options Since it is good policy to avoid fighting in the workplace, the best thing to do would be to walk into the developer’s office not only with a list of constraints, but also a list of options. Keeping a positive demeanor is of utmost importance. If possible, the most effective way to communicate this information to development would be via the Quality Assurance (QA) Manager. Inter- team communication is one of the major roles of a manager, so as an automation engineer you might as well take advantage have to remain aware of the fact that the verify its functionality is worth the effort. of it. Additionally, program management bypassed controls would need to be tested For example, if a new control is being added may be involved and have input regarding manually. but it only affects a small number of tests, you the final decision, so it is best for them to Option 3: Programmatic Solution – Another might want to see if the effort of automating communicate with someone that they are option would be to design a programmatic the tests against the control is cost-effective. familiar and comfortable with. solution; a coded solution offered by It may make more sense to allow the test to remain manual. Conversely, if the new control It is important to include caveats when developers providing a custom way for communicating these options. For example, your tests to bypass this screen while still is going to be used frequently and is touched by a large number of your automated scripts, you might have a programmatic workaround accomplishing the test objectives. For in mind, but this would require developer example, if the new controls were being it might be worth the effort. Ultimately, it boils down to how easy or difficult it will be involvement, add time to the project schedule, implemented on an ‘Add User’ screen, was remove focus from other current automation there a way that the developers could allow for automation to adapt, and if the effort is worth the result. activities, and forego any testing of the new you to bypass this screen using pre-built user control itself since it is being bypassed. You data stored in the user information database Presenting Automation’s Case might have found an add-in that could resolve that your test could access and use? This So you’ve discovered that your automated most of the issues you had documented would again result in the tests for the new testing tool has problems accessing and during the evaluation, but the add-in was controls being reserved for manual execution, manipulating these new 3rd-party controls. expensive, code needed to be re-written to but it would also allow all of your tests that Should you just run to the development team replace the previous code used to manipulate need to access information processed on this and tell them that the new control can’t be the previous controls, and the add-in still screen to continue running properly. automated? Before doing this you need to wouldn’t provide all of the functionality Analyzing Impact keep several things in mind. The developer(s) required to meet the test objectives for this When identifying different options it also who discovered these new controls and are application component. Or you might explain makes sense to look at the impact that pushing for their implementation have not that your automated test tool extensibility the new control has on the application to only a technical stake in this but an emotional features would allow you to handle this new determine if automating tests to exercise and stake, as well. Put yourself in the developer’s control via automation, but it would also 26 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 require developer participation to help design return-on-investment after considering all of on test automation. This will in turn help the code. Until this code is developed and the activities that need to take place in order your organization to release an improved, deployed automation test coverage would be for the effort to be successful. well-tested product to its customers while drastically reduced. maintaining positive, synergistic inter-team lessons learned relationships within your organization. You or your manager might also want to point After experiencing similar scenarios at out how this new technology would affect several different IT organizations I was able Final Thoughts other teams in the organization. Business to learn some valuable lessons that resulted Products naturally evolve or they go extinct, Analysts would have to rewrite, add, or in a set of guidelines that I now follow overtaken by new products that improve revise requirements. Technical Writers to make it through these situations while upon the previous design. New technologies would have to update User’s Guides and experiencing as little pain as possible. The are a part of this evolution, and automated other documentation to reflect the changes to most important guideline is to ensure that testing tools and methodologies will continue functionality. The Sales Team would need to other teams in your organization have a good to evolve to address changes in technology get up-to-speed on these new controls so they understanding of automated testing processes that come down the pipeline in the future. could market these improvements to existing and procedures. This can be accomplished So much like we’ve moved from steam to ...ENSURE THAT OTHER TEAMS IN YOUR ORGANIzATION HAVE A GOOD UNDERSTANDING OF AUTOMATED TESTING PROCESSES AND PROCEDURES. and potential customers. The Training Team by offering to conduct formal presentations gas to hybrid to electric in car engine design would have to revise presentations, manuals, or brown bag lunches, possibly resulting in and application, the same has happened for and other training materials. enhanced development processes that keep IT systems and software development. It The number one objective in making this a automation in mind when developing new is important to keep up-to-date with new successful meeting is to avoid friction. Don’t code. The developers may work harder to technologies and their impact on automation, make the developer view you as a roadblock ensure that new code provides hooks for the make your organization aware of automation or the enemy. If you are excited about this automation tool to grab on to. Another good processes and the affect their decisions will new control and agree that it will improve guideline to follow is to always be aware have on these processes, all while keeping an your product, let them know! Tell them that of what development is researching and eye on their possible effect on your current you will do everything within your power to may be planning for future versions of your automation model. This will help to make make this work. Forming and maintaining a product. The best way to do this is to ensure the addition of new technologies to your collaborative relationship with the developers that you are invited to development meetings product as painless an experience as possible is of paramount importance. where these items may be discussed. This while leveraging all of the advantages that allows automation to have a voice in automated testing offers to your organization. often the development and QA teams can find middle ground. The collective decision may the decision making process throughout the project lifecycle and also allows you the automated testing be to delay the addition of these new controls institute (ati) conducted a until you have fully prepared the automation proactively conduct research prior to any formal planning. Finally, be sure not only to podcast interview with J.l. framework to handle them. All parties may perlin that offers a discussion agree on QA halting all new automated test research the technology being considered by of this article, as well as his design while you concentrate your efforts on your organization, but similar technologies test automation philosophy. this compatibility assignment. Stakeholders offered by other vendors, that may present look for it at: may decide to search for other controls that more options. may be more automation-friendly. Or the Following these guidelines will increase http://www.podcasts.automatedtestinginstitute.com parties may decide that the impact of adding the chance of effectively implementing new these new controls does not result in a positive technologies while reducing the impact August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 27 The creation of a well thought out It is not very difficult these days to sell those who are involved in the Information framework that Technology (IT) industry on the importance of regression automation – at least the theoretical importance. It seems as though most understand is customized to that test automation saves time, is repeatable, and if done correctly, can be very reliability. your particular Putting this theory to action proves to be problematic, however, due to the commonly environment can held belief that automation should require no additional thought other than that which be time consum- requires one to execute existing manual test procedures while being recorded by an ing, so this article automated test tool. In order to facilitate the desire to crank out quick identifies 5 simple and thoughtless automated tests, the industry is creating more and more “user friendly” GUI steps that will help automation testing tools, both commercial and open source, and these tools offer record in speeding up the and playback features that generate code or code-less visual scripts that may seemingly 5 process. StepS t Develo 28 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 test an application with little is often thought of as the collective effort. These applications, with structures that compose a unit their increased capabilities, can testing tool. In this article, however, be extremely helpful, but also add frameworks will be discussed in a to the problem. One of the major different context; one where it is selling points for some of these defined as a set of abstract concepts, applications is that, “With our processes, procedures and application, you save a lot of time environment in which automated because you don’t have to spend tests will be designed, created and time thinking about, and planning implemented. In addition, this out how you want to perform your framework definition includes tests. You just record your steps, the physical structures used for and playback the resulting script.” test creation and implementation, It is important to avoid falling as well as the logical interactions into the trap of believing that among those components. some magical tool is going to The creation of a well thought out eliminate the need for thinking framework, customized to your and planning, because automated environment can be challenging, testing is most effective when so this article identifies 5 simple planned and implemented within steps that will help serve as your a framework. The term framework blueprint for getting it done. to Framework opment August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 29 1 Perform Initial to use or build upon in your automation efforts. There are four main segments in this framework design: Framework Design • • Driver Segment Test Segment • Function Segment Designing an automated test framework involves first identifying • Parameter Segment the type of framework that will be developed. This typically requires researching and understanding your software development The Driver Segment, represented by a script or tool, reads from environment as well as the application(s) under test (AUTs), and the Execution Level File, which stores execution level information for determining which framework will work best (see Table 1). each test script (test name, test priority, etc). Next, the Driver Segment Figure 1: Framework Component interactions When in doubt about the framework that will work best in calls upon the Parameter Segment to initialize the environment, and your environment, the safest choice is a functional decomposition set parameters necessary for the test run. After calling the Parameter framework. Functional decomposition frameworks can be as simple Segment, the Driver Segment calls and executes tests based on the or as complex as deemed necessary in order to handle the demands of associated priority. Control is then yielded to the test scripts in the the automation scope. Test Segment for AUT verification. The Test Segment frequently Once a decision has been made about the framework type, it is imperative to identify the basic components that will compose the calls functions from the Function Segment for performing succinct framework, and have a high-level picture of how these components test case actions that collectively compose a test case. In addition, the interact with one another. tests in the Test Segment may call upon Data Files to obtain the data Figure 1 presents a basic design example that you may decide necessary for test execution. Framework Description Type Linear All components that are executed by automated scripts in this type of framework largely exist within the body of that script. There is little-to-no modularity or reusability in a Linear framework. Data-driven Most of the components that are executed by a Data-driven script largely exist within the body of that script, with the exception of the data. The data used in Data-driven scripts is typically stored in a file that is external to the script, which promotes script reusability Functional This is a framework that involves the creation of modular functions for executing fundamental actions that occur during test execution. Decomposition These actions may then be called upon and executed independent of one another, thus allowing them to be reused by multiple tests within the automated test bed. Keyword This type of framework is strongly dependent on modularity concepts embodied by functional decomposition. In comparison to the Functional Decomposition framework, the keyword framework evaluates the functions at a much higher level of abstraction, so that tests are no longer developed using code, but rather developed in a tabular format that is interpreted and executed by a driver script or utility. Model-based Often called an “Intelligent framework”, Model-based frameworks go beyond creating automated tests that are executed by the tool. These frameworks are typically “given” information about the application, in the form of state models, and the framework “creates” and executes tests in a semi-intelligent, dynamic manner. table 1: Framework types 30 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 2 Create the automated tests execute on a different server, or run in a completely different environment. With such considerations made during the Directory Structure upfront creation of the physical structure, an ultimate increase of the framework’s portability will almost certainly be a positive by-product. The physical structure may be maintained by a test management tool, some sort of Integrated Development Environment (IDE), Test automation is a small-scale form of software development that or simply by a file system. Below is a pictorial representation of a involves creating a software package. It is, therefore, important for sample physical structure for an automated testing framework. All of this package to have a preplanned physical structure. The physical the components are housed under a single Root directory, which aids structure is similar, but different from the component design identified in portability. In this root directory are several sub folders, including: in Figure 1. The component design is more of an abstract representation of how the framework components work together, while the physical • Initialization Files structure identifies how and where these components will be stored. • Configuration Files The upfront identification of the physical structure is important • Function Library for several reasons. Obviously, it is important because you need • Driver Files to know where things are going to be stored. It is also important • Test Scripts because the framework may need to be collectively moved as a single unit. Such a move may be prompted by the need to have the These items will be discussed in subsequent steps. root Directory init Config Function Driver test Files Files library Files Scripts Figure 2: Directory Structure Config The Function Test Scripts are Init A Driver Script (configuration) Library holds a the components (initialization) coordinates Files establish set of functions that actually Files bring test activities parameter used by the perform the tests by calling and values that are automated test verification environment executing the necessary for suite. It is activities on to a controlled test scripts. It the test run. normally the application stable point. may run init Config loaded at the under test This will files, set up parameters are beginning of (AUT). Test increase the config similar to Init a test run, and scripts perform chances of the parameters, parameters but its functions actions, and test scripts and run test are likely to are called generate results running scripts. change more throughout the files. properly. frequently test run. August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 31 The Automated Testing Institute (ATI) Online Reference is the web’s foremost comprehen- sive online resource for obtaining, exchanging and certifying industry standard test automation knowledge, information, and techniques. Visit us today! 3 Develop Init and of the automated test suite. These settings are often specific to application technology. For example, in web application testing Config Files you might initialize the environment by doing the following: » Clearing the cache » Turning off auto-complete » Turning off browser animation effects An Initialization File contains a set of parameters or code that brings » Setting cookies the environment to a controlled stable point. If the environment » Setting frame format is initialized, there is a better chance of the test scripts running as designed. A Configuration File contains a set of parameters needed Configuration parameters are those parameters that may change for the test run. more frequently, but upon changing, they will permeate through The main difference between initialization parameters and multiple tests within that suite. Parameters that may be set in the configuration parameters is that initialization parameters are not likely configuration script include: to change from test run to test run, or from machine to machine. Instead, initialization parameters will remain fairly constant throughout the • User Ids and passwords, project life cycle. • Pointers to databases, Initialization parameters/actions that are addressed in • Public variables and constants, and Initialization Files include those that are responsible for: • Pointers to the application (domains, URLs, etc). • Setting Directory Paths – Directory paths help in finding key To help better understand configuration parameters, let’s examine automation components during runtime. At a minimum, all of a scenario represented by the figure below. The automated tests in the main directories of the automated test framework should be this environment may be executed by several different testers. Tester included in this. 1 may log-in on one machine to run the tests, while Tester 2 logs into • Loading Library Files – Test automation library files contain another. Each tester may have their own set of test data, and must also reusable functions that may be shared across multiple tests. These choose, at run-time, what front-end server to connect to, then what libraries may be loaded during the automation initialization back-end data server to connect to. process. No one should have to make all of these determinations every • Modifying Application Settings – Initialization of application time they sit down to run their automated test bed. Configuration specific settings is often important for the successful execution parameters may be set up to maintain the desired configurations. Test Data Application Layer Database Layer Tester 1 Tester 2 Figure 3: Configurations 34 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 4 Develop the • Error handling functions • Loading functions Function Library • Miscellaneous verification functions Core Business function exercise basic application functionality. For example, a function that is responsible for logging into the Often this step is placed after the creation of automated tests (Step 5) application would be considered a core business function. These through the following process. functions can be identified from the requirements or from the manual test cases, related to the groups of tests that are slated for automation. 1. Generating multiple test scripts with identical code, Most applications have several main navigation paths that are 2. Identifying the redundant code, taken many times during testing. And if the application modifies these 3. Copying the redundant code into a function within a function paths, all developed automated tests that follow these paths would be library, then broken. Therefore, developing navigation functions is imperative. 4. Replacing the redundant Error Handling functions code in each of those are created to deal with scripts with a call to the FUNCTION LIBRARY unexpected events that may function. occur during testing. Some Contains events that will definitely This approach is need error handling include: extremely inefficient, and building produces unnecessary rework. • The failure of an The more effective approach blocks expected window to appear. is to develop functions before developing tests via the for • The failure of an expected object to appear. following process. tests • The failure of an expected value to appear. 1. Identifying redundant • The appearance of an actions, unexpected popup window. 2. Creating functions for these actions within a function library, then Loading functions do exactly as the name implies – load things. 3. Adding calls to the functions when creating the test scripts. These functions load files and compiled modules for use by the automation tool. Most of the functions that will be necessary for the efficient implementation of an automated test bed can be identified before Although many automated test tools provide built in verification you create a single automated test script. This is possible because checks, it is often necessary and more efficient to develop custom Step 1 involved reviewing documentation related to the software Verification Functions to verify certain application attributes. Some development environment, test cases and the AUT; in addition, it user-defined verification functions that you may want to consider may involve reviewing the AUT itself, if it is available. This review creating include the following: time should provide some insight into some of the repetitive actions that take place while exercising and testing the application, therefore, • Window check there’s no need to wait until the test script creation begins before • Date/time check functions are created. • Login ID check Various types of functions that may be necessary include: When creating functions, it is important to identify pertinent • Core Business (Utility) functions • Navigation functions function arguments and return values. August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 35 5 Develop the Test through 3 in Figure 4 have been parameterized and placed in a function called “Login”, while steps 4 through 8 have been parameterized and Scripts Figure 5: Functional Decomposition interface The final step in the framework development process is the creation of automated test scripts. The automated test interface is largely a placed in a function called “Verify_Screen”. Note: depending on the configuration parameters that are created, the username (“John”) and password (“JPass”), may be stored in the configuration file and delivered to the script via configuration parameters. Keyword frameworks reduce the technical nature of automated test development even further by allowing them to be developed in a tabular format. For example, the Keyword equivalent of the statements illustrated in Figure 4 might appear as illustrated in Figure 6. The columns in this illustration represent different elements of the test that collectively perform the desired actions: Figure 4: Code-based interface • Screen – The screen on which the automated test step occurs • Object – The object on which the automated test step operates function of the type of automation framework and automated tool that is in use. For Linear and Data Driven frameworks, the tests are • Action – The keyword that is tied to a reusable function and normally code-based, as illustrated in Figure 4, and identifies what activity will take place on the test step therefore fairly technical. These tests are therefore • Value – The value that may be entered into the composed of code that is very specific to the application test script in which it exists. Some tools offer a tree structure and/or “keyword” interface locks • Recovery – The error handling actions ing B Build for this step that adds a more graphical and icon-based view of code, but when these tools are used • Comment – A note about the step’s main in a Linear or Data Driven framework, the purpose, used to provide helpful information tests still follow the basic precepts of a to anyone reading the keyword file code-based test. Tests in Functional Decomposition The keyword file is interpreted and frameworks are still code-based, but as executed using a driver script. the framework becomes more defined, the Following all five of the steps tests become slightly less technical. This is presented in this article from start due to the fact that the tests are largely created to finish will aid you in creating an by treating reusable components as building blocks, automated test framework that will provide and stacking them together. an environment in which your automated Figure 5 reveals how the statements in Figure 4 might be tests will thrive across multiple releases of written in a Functional Decomposition framework. Statements 1 the AUT. Figure 6: keyword Driven interface 36 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 Evolve, Adapt and build hand me the Key...word (Continued from page 10) (Continued from page 4) Lines 10 through 22 use the DT_Get_ function is successful, the return value will Finally, yours truly chimes in on the third Value function to get the values from each of be 0 (zero). If the function fails, the return mode of change, defined by building up your test the columns of the keyword file (the ‘DT_’ value will be 1. In the event of a failure automation framework. This change mode has functions are not native Ruby functions, but Lines 43 through 45 call an Exception no particular catalyst, but is instead unprovoked, are instead custom functions). Handling routine that you develop called requiring implementation by a project looking Line 24 through 41 compose a utilityRecover. The recovery action defined to improve their test automation return-on- branching construct that interprets the in the Recovery column of the keyword keyword in the Action column of the file will determine what action is taken on investment. I provide a step-by-step approach for keyword file (as illustrated in Figure 3). This a failure. This is an effective and efficient building an automated test framework that will construct is the heart of the keyword file in mechanism for implementing global ultimately help to sustain multiple automated that it ties each keyword to the execution of exception handling on every line of an tests over an extended period of time, therefore automated test. a specific utility function. In addition, this helping your automated tests proactively keep construct determines which columns will Line 46 closes the loop that cycles pace with application changes. deliver arguments to the function. Line 27 through each line of the keyword file, while So, while I may be biased, I do believe and 28 provide an example of this concept. Line 49 closes the keyword Excel file. you will thoroughly enjoy the three featured If the current keyword file row being read articles in this issue. These articles, along with by the driver script uses the “INPUT” DT_OPEN, DT_GET_VALUE and DT_ the department articles - focused on keyword keyword, Line 28 will call the utility Input CLOSE are not native Ruby functions. This driver script development, open source tool function, and will deliver values from the are custom functions for manipulating and Screen, Object and Value column into the Excel file. For guidance on how to create contributions, and automation blogging - will function’s arguments. these functions, use the Data-driven Scripting help you understand present day test automation, Each utility function should be With Excel entry at: and help us all define the next phase in automation evolutionary development. developed to return a value that the driver http://www.techniques.automatedtestinginstitute.com script assigns to a variable called ‘ret’. If the August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 37 hot topics in automation Automation micro Microblogging is a form of communication based on including Internet browsers, mobile phones, personal the concept of blogging (also known as web logging), digital assistants (PDAs), and desktop client software. that allows subscribers of the microblogging service Microblogging is a powerful tool for relaying an to broadcast brief messages to other subscribers of the assortment of information, a power that has definitely not service. The main difference between microblogging been lost on the test automation community. See some and blogging is in the fact that microblog posts are of the recent Twitter messages below for an example of much shorter, with most services restricting messages how test automators have used microblogging. to about 140 to 200 characters. Recently popularized by Twitter, there are numerous other microblogging services, including Plurk, Jaiku, Pownce and Tumblr, and the list goes on-and-on because there is a tremendous appeal to microblogging. The appeal is in the brevity of the message (important in today’s fast paced society), the ability to stay connected in real-time with one’s interests, and the portability of the microblogging services. Many of these services allow messages to be sent and received in a variety ways, including instant messages, text messages, email, video clips and audio sound bites. In addition, message can be sent and received via a variety of tools, Twitter name: sietstweets Do you know about the AA- post Date/Time: 5:09 AM Jul 30th FTT, a community committed usage: Sharing Tool Frustrations to better functional testing tools for Agile teams? See http://is.gd/1HUvz Working with #qftest for automated gui testing... just found its not going to work for webtesting on MAC OS X... bastards! Twitter name: testobsessed post Date/Time: 11:49 AM Jul 22nd usage: Community Introduction 38 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 oblogging is hot! Microblogging is a powerful tool for relaying an assortment of information Twitter name: fredberinger post Date/Time: 11:39 PM Jul 21st usage: Sharing Articles Automated testing of production environment http://bit.ly/kvduU Good article, so true ! Twitter name: TestComplete Software Application Test post Date/Time: 8:37 AM Jul 31st Analyst Paris, France!!!: usage: Course Announcement London-London, Job Purpose: • Automate test plans.. http:// ub0.cc/3R/3F We’ve just released our new TestComplete course onto the public schedule. Check it out; http://tinyurl.com/nkofpa Twitter name: Monsterjobsuk post Date/Time: 8:08 AM Jul 29th usage: Job Announcement August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 39 i ‘B’log to u latest from the B Automation blogs are one of the greatest sources to-date test automation information, so the Autom Testing Institute has decided to keep you up-to-da some of the latest blog posts from around the web interesting posts, and keep an eye out, because yo your post will be spotlighted. blog name: google Testing Blog blog name: I.M. Testy post Date July 1, 2009 post Date August 1, 2009 post Title: The Plague of Blindness post Title: UI Automation Out of Control Author: James A. Whittaker Author: Bj Rollison Imagine playing a video game blindfolded I will say that in general I am not a big or even with the heads up display turned fan of GUI automation for a litany of off. You cannot monitor your character’s reasons, but mostly because it is misused, health, your targeting system is gone. or applied to tests that are simply more There is no look ahead radar and no effectively and efficiently performed by advance warning of any kind. In gaming, a person. However, I also understand the inability to access information about the GUI automation does provide value when used in the right context. I am not the campaign world is debilitating and against GUI automation, but I certainly a good way to get your character killed. don’t advocate automating a test just There are many aspects of testing because we think we can, or because software that fall into this invisible we have some bizarre idealistic belief spectrum. Software itself is invisible. that everything should be automated. Read more at: Read More at: Read More at: http://googletesting.blogspot.com/2009_07_01_archive.html http://blogs.msdn.com/imtesty/archive/2009/08/01/auto- mation.aspx 40 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 Blogosphere of up- mated ate with b. Read below for some ou never know when blog name: Collaborative Software Testing blog name: Creative Chaos post Date July 29, 2009 post Date July 31, 2009 post Title: Fuzzing Through The Side Door post Title: The Meme’s the thing Author: Jonathan Kohl Author: Matthew I’ve been looking into testing with Test-Automation, for example, is fuzzers lately, and finally got the chance very appealing to developers, because to do this on a live project. While there automation is what they do. It’s no surprise, when devs look at testing as a are a good deal of black-box fuzzing computer science problem, Automation tools out there, if you want to go beyond is the first thing to come to mind. So that you are often on your own. At the we have generation after generation other end of the spectrum, MSDN has talking about test automation as the a nice article on white box fuzzing. be all and end all of the test process, without ever having actually studied the very human, cognitive and communication process of software testing, nor having done any research on failure modes of software defects. Read More at: Read More at: http://www.kohl.ca/blog/archives/2009_07.html http://xndev.blogspot.com/ August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 41 up-to-Date with ati T his Week In Automation Series This Week In Automation is an ATi produced video series that summarizes current events relative to test automation. with an often tongue-in-cheek approach to discussing the latest news in tools, events (both small and large) and concepts, this series serves as the video companion to ATi’s newsletter, while keeping you abreast of what is going on in the world of software test automa- tion. See the latest edition of This Week In Automation at http://www.twia.automatedtestinginstitute. com. S ee No evil, HeAr No evil Have you seen the latest podcast produced by ATi? if not, navigate over to http:// podcasts.automatedtestinginstitute.com and listen to the latest podcast featuring an interview with Ms. linda g Hayes. in this interview, she discusses her article entitled “evolution of Automated Software Testing: An eyewitness Account”. in addition, she discusses her thoughts on the present and future of test automation in general. I t’s NeWS To Me over the past few weeks ATi has brought the user community some interesting and very popu- lar news stories related to test automation. These stories include: • Microsoft Powerpoint defect Posing Security risk • espionage and Software: u.S. Vulnerable to Cyberspies!? • Microsoft Powerpoint Security Vulnerability resolved For these and other stories visit http://www.news.automatedtestinginstitute.com. 42 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 C oming Highly rATed wanna know which of the hundreds of indexed tool and article archives are the most popular? well, here goes. For the purposes of this section, popularity is based on the number of hits and the rating received. Two of the most popular indexed tools on the site are FunFX and jSystem. Two of the most popular indexed articles are “Test Automation roi” by dion johnson and “Automating Software Testing: A life-Cycle Methodology” by elfriede dustin. be sure to visit http://www.tools. automatedtestinginstitute.com and http://www.articles.automatedtestinginstitute.com for the latest tools and articles, and be sure to rate them after you have read them! A utomation Honors Test automation is an integral part of ensuring the production of quality systems in a world where software development is becoming increasingly fast-paced. And one way to help elevate test automation so that it can be more effective is to celebrate it and its practitioners. Thus, the ATi Automation Honors have been created to celebrate excellence in the discipline of test automation from the corporate level down to the practitioner level. This celebration specifically pays tribute to: • Those that have displayed leadership in moving and keeping test automation in its proper place as a distinct iT discipline, • Those that drive innovation within the field, and • Those that display excellence in automation implementation, thus playing a big role in the delivery of a high quality product to customers and/or production. • Tools, individuals, publications, websites and more. The nomination period has reached a conclusion, and finalists will be identified by august 17, at which time voting will begin and commence through october 31, 2009. For more information on the Automation Honors, the nominating process or the honoree categories, visit http://www.atihonors. automatedtestinginstitute.com. August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 43 How Do You Network? Whatever your choice, the Automated Testing Institute is there! YouT Twitter Facebook Myspace Stay up-to-date with test automation by following the Automated Testing Institute on your so- cial networking site of choice. Facebook Myspace Twitter YouTube LinkedIn Blogger For more information, visit http://www.networking.automatedtestinginstitute.com 44 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009 Still Testing SAP Manually? Worksoft Certify® Get ahead of the curve and maximize the value of your SAP investments with Worksoft Certify®, the for SAP® only code-free approach to end-to-end, cross-platform business process validation and Automate automated testing. Accelerate Worksoft Certify® for SAP® is a specialized lifecycle management solution for SAP that Optimize automates and optimizes SAP lifecycle tasks from change impact analysis through functional testing, performance testing and audit documentation. Certify for SAP is unmatched in the industry for accelerating time to value and reducing overall project costs by up 40-80%. Are you ready to test smarter and deploy faster? Contact us at 1.866.836.1773 or visit www.worksoft.com to learn more.
Pages to are hidden for
"Evolution Evolution"Please download to view full document