CheckList To Test Web sites

Document Sample
CheckList To Test Web sites Powered By Docstoc
					1.0 Website Testing
1.1 Introduction
The nearly instant worldwide audience makes a Website‟s quality and reliability crucial to its success. The nature of the WWW and of Website software poses unique software testing challenges. Webmasters, WWW applications developers, and Website quality assurance managers need tools and methods that meet very specific needs. Our technical approach, based on extending existing WWW browsers, offers many attractive benefits in meeting these needs.

1.2 Background
Within minutes of going live, a WWW application can have many thousands more users than a conventional, non-WWW application. The immediacy of a Website creates immediate expectations of quality, but the technical complexities of a Website and variances in the available browsers make testing and quality control that much more difficult, than "conventional" client/server or application testing. Automated testing of Websites thus is both an opportunity and a significant challenge.

1.3 Defining Website Quality and Reliability
Like any complex piece of software there is no single quality measure to fully characterize a Website. There are many dimensions of quality, and each measure will pertain to a particular Website in varying degrees. Here are some of them:       Timeliness: How much has a Website changed since the last upgrade? Structural Quality: Are all links inside and outside the Website working? Do all of the images work? Content: Does the content of critical pages match what is expected? Accuracy and Consistency: Are today's copies of the pages downloaded the same as yesterday's? Response Time and Latency: Does the Website server respond to a browser request within certain parameters? In an E-commerce context, what is the end to end response time after a SUBMIT? Performance: Is the Browser -> Web -> Website -> Web -> Browser connection quick enough? How does the performance vary by time of day, by load and usage?

Clearly, “Quality” is in the mind of the Website user. A poor-quality Website, one with many broken pages and faulty images, with Cgi-Bin error messages, etc. may cost in poor customer relations, lost corporate image, and even in lost sales revenue. Very complex Websites can sometimes overload the user.

1.4 Website Architectural Factors
A Website can be quite complex, and that complexity -- which is what provides the power, of course -- can be a real impediment in assuring Website Quality. Add in the possibilities of multiple Website page authors, very-rapid updates and changes, and the problem compounds. Here are the major pieces of Websites as seen from a Quality perspective. Browser: The browser is the viewer of a Website and there are so many different browsers and browser options that a well-done Website is probably designed to look good on as many browsers as possible. This imposes a kind of de facto standard: the Website must use only those constructs that work with the majority of browsers. But this still leaves room for a lot of creativity, and a range of technical difficulties. And, multiple browsers' renderings and responses to a Website have to be checked. Display Technologies: What you see in your browser is actually composed from many sources:    HTML. There are various versions of HTML supported, and the Website ought to be built in a version of HTML that is compatible. This should be checkable. Java, JavaScript, ActiveX. Obviously JavaScript and Java applets will be part of any serious Website, so the quality process must be able to support these. On the Windows side, ActiveX controls have to be handled well. Cgi-Bin Scripts. This is link from a user action of some kind (typically, from a FORM passage or otherwise directly from the HTML, and possibly also from within a Java applet). All of the different types of Cgi-Bin Scripts (perl, awk, shell-scripts, etc.) need to be handled, and tests need to check "end to end" operation. This kind of a "loop" check is crucial for E-commerce situations. Database Access. In E-commerce applications you are either building data up or retrieving data from a database. How does that interaction perform in real world use? If you give in "correct" or "specified" input does the result produce what you expect? Some access to information from the database may be appropriate, depending on the application, but this is typically found by other means. Navigation: Users move to and from pages, click on links, click on images (thumbnails), etc. Navigation in a Website is often complex and has to be quick and error free. Object Mode: The display you see changes dynamically; the only constants are the "objects" that make up the display. These aren't real objects in the OO sense; but they have to be treated that way. So, the quality test tools have to be able to handle URL links, forms, tables, anchors, buttons of all types in an "object like" manner so that validations are independent of representation. Server Response: How fast the Website host responds influences whether a user (i.e. someone on the browser) moves on or gives up. Obviously, InterNet loading affects this too, but this factor is often outside the Webmaster's



control at least in terms of how the Website is written. Instead, it seems to be more an issue of server hardware capacity and throughput. Yet, if a Website becomes very popular -- this can happen overnight! -- Loading and tuning are real issues that often are imposed -- perhaps not fairly -- on the WebMaster. Interaction & Feedback: For passive, content-only sites the only real quality issue is availability. For a Website that interacts with the user, the big factor is how fast and how reliable that interaction is. Concurrent Users: Do multiple users interact on a Website? Can they get in each others' way? While Websites often resemble client/server structures, with multiple users at multiple locations a Website can be much different, and much more complex, than complex applications.

2.0 Website Checklist
Here is a checklist of things to run though before going public with your web-site. Most of them are things which seem to have been overlooked by large numbers of web developers. This checklist is oriented towards matters which can be – more or less – objectively determined. It does not attempt to cover matters which are very subjective, such as clarity of layout or text, though these are of course just as important. It covers only matters which affect the readability of your site – not those which affect how easy it will be for you to maintain it. A useful resource to help with further checking of your site, including the more subjective matters, is the alt.html.critique Usenet newsgroup. The links in the checklist lead to a discussion of each issue.

2.1 Validation
2.1.1 Validate the HTML

The first stage of checking a web-site is to make sure that you have valid HTML (or XHTML). Your own browser may ignore certain errors, but there is a significant risk that markup errors will result in display problems in some browser or other. There are sometimes reasons for including invalid HTML. For example some pages on my own site use the non-standard NOINDEX element, for the benefit of my site search engine. But you should only tolerate HTML validation errors if there is a clear need for the non-standard markup, and if you have tested the result in a variety of browsers. If you are using transitional HTML to support old browsers such as Netscape 4, it may be worth doing a quick scan with the DOCTYPE override set to Strict, and removing any transitional markup that isn‟t actually necessary.

2.1.2

Validate the CSS

CSS can be validated with for example the W3C CSS validator. The considerations here are much the same as HTML validation, except that there shouldn‟t be any need for CSS syntax errors. It may sometimes be necessary to use something nonstandard to get Internet Explorer to work, but such rules can be placed in a separate CSS file and hidden in an Internet Explorer conditional comment, where they won‟t bother other browsers or a validator.

2.1.3

Check for broken links

Obviously you don‟t want broken links on your site. There are various tools available to help find these, such as the Link Valet (which is convenient for checking a few pages) or Xenulink (convenient for checking a whole site).

2.2 Flexibility
2.2.1 Try varying window-sizes

A very important aspect of web design is coping with different window sizes. Window widths may vary from about 200 pixels on a web-enabled telephone to 2000+ pixels on a technical workstation in full-screen mode. While providing a readable site to the very smallest screens is something of a challenge, your site should still be readable on a wide variety of sizes. As of mid 2003, about half of all readers are still using screens of 800x600 pixels or smaller, and if the reader wishes to compare the contents of your site with another document, it is entirely possible that he/she may want to use a window-width of around 400 pixels. Fortunately, at least as far as the text goes, this is not very difficult – just refrain from specifying sizes in pixels or points and you are most of the way. It is obviously easy to test window-sizes smaller than your own screen-size. Testing larger window-sizes might seem impossible, but you can do a rough simulation using the zoom facility of the Opera browser – zoom down to 50% and you get a screen twice the size. It may not be very readable, but any major layout errors should be obvious. Incidentally don‟t worry too much about the very long lines of text that appear at large screen sizes when small fonts are used. If the reader doesn‟t use a large font, he can always reduce the window size to something comfortable – that is, after all, half the point of having windowing user interfaces in the first place. But if you wish to, you can also use the CSS2 „max-width‟ property to limit column width, just as this page does (though only discerning readers are currently able to benefit from this, as IE does not support it).

2.2.2

Try varying font sizes

Some people use large screen fonts because they have a large screen with a very fine resolution; other people have to use large screen fonts because of declining eyesight. On the other hand, some people like to use microscopic fonts with their nose pressed against the screen (well, it‟s a free country). So while doing the above activity, adjust the default text-size in your browser, and check that all the text scales appropriately. One other aspect to consider is that users may impose a minimum font size to make sure that all text is readable for them. This means that font sizes show a smaller range than you had in mind. If you have a complex page with a wide range of font sizes, it would be worth imposing a minimum size larger than your smallest font (this

can be done in e.g. Opera or Mozilla) and checking that this does not make parts unreadable.

2.3 Speed
2.3.1 Access the site via a modem

Put your site on the server, then dial in via a modem (a real modem – not an ADSL gateway, which is sometimes erroneously referred to as a modem). Does it look so good half a minute later? Or are you still staring at a pretty-much blank screen? If the opening page of your site takes more than half a minute to appear, then you are losing visitors fast. Don‟t overload it. If you have to include large objects on your site – perhaps it revolves around high-resolution reproductions of fine art – put them on later pages and tell (or warn?) your users what is coming. By the way, don‟t make the mistake of assuming that you don‟t have to worry about bandwidth issues if your site is mainly aimed at companies. I have more than once worked at companies which did have a broadband connection, but had it shared between so many users and applications that the net result was no faster than a 28K modem.

2.3.2

Check image size specifications

While doing the test above, check that at least the text of the page appears quickly. If it doesn‟t (or if it dances all over the place while the page is loading) it is probably because you have forgotten to specify HEIGHT and WIDTH attributes on some of your images. Specifying these enables the browser to work out where the text needs to go, and to display it before the images have finished downloading.

2.4 Accessibility
2.4.1 Test accessibility

This is mainly important for handicapped users, but also relevant for e.g. people who use a text-only browser, or disable images, because of a slow connection. Automated accessibility checking does need to be taken with a pinch of salt. Many aspects of the guidelines require human judgement to say whether a page is accessible or not – for example whether HTML Heading tags are used correctly. And even when the guidelines are unambiguous, you don‟t need to follow them slavishly. For example the absence of a caption on a table is unimportant if the previous paragraph explained what the table is about. Nonetheless it is well worth running a few pages through a checker such as Bobby or Accessibility Valet in order to familiarise yourself with the issues involved. You can then make the necessary improvements.

2.4.2

View in text browser

It is also worth running pages through a text-only browser, or text-browser emulator to see what e.g. a blind person using a text-to-speech converter will encounter. It will help you pick up on badly-chosen or missing ALT texts for example. It also shows you the site pretty much as a search engine will see it. Incidentally the Opera 7 browser has a built-in text-browser emulator.

2.4.3

Browser independence

Your site may be viewed in a large variety of situations: different browsers, different operating systems, different features enabled or disabled. It is important that your site stands up well in these different situations. The first point of attention here is validation – described separately above. Then there are the following points.

2.4.3.1 Try different browsers
Almost all web developers are aware of the need to check how their site looks in a variety of browsers. How far you go obviously depends on the resources available – not everyone is in a position to check Windows, Mac, Unix and Linux platforms. The minimum test would probably be: 1. A Gecko-based browser – Netscape 7, Mozilla or Firefox – as these seem to have the best standards compliance; 2. Internet Explorer for Windows – currently the most widely used browser; 3. Opera – growing in popularity due to its speed and pretty good standards compliance.

2.4.3.2 Check printed pages
Print some of the pages on a normal printer (i.e. with a paper size of A4, quarto or letter) and check that they appear sensibly. Due to the somewhat limited formatting options available for printing, you probably can‟t achieve an appearance comparable

to a document produced by a word-processor, but you should at least be able to read the text easily, and not have lines running off the right-hand side of the page. You should also consider using CSS to adjust the appearance of the page when printed. For example you could suppress the printing of information which is not relevant to the printed page, such as navigation bars. This can be done using the “@media print” or “@import print” CSS features. Some sites provide separate “printer friendly” versions of their pages, which the user can select and print. While this may occasionally be necessary as a last resort, it significantly increases the amount of work needed to maintain the site, is inconvenient for the reader and shouldn‟t usually be needed.

2.4.3.3 Switch Javascript off
There are unfortunately quite a number of Internet sites which abuse Javascript by, for example, generating unwanted pop-ups and irritating animations. There are also a number of Javascript-related security holes in browsers, especially Internet Explorer. As a result a lot of readers (a figure of 20% is often quoted) switch Javascript off – indeed I often do myself. Some organisations even block the usage of Javascript completely. Furthermore few, if any, search engines support Javascript. It is therefore important to check that your site still functions with Javascript disabled. A lot of sites rely – quite unnecessarily – on Javascript for navigation, with the result that the lack of Javascript renders the site unusable. Clearly if you use it to do something important (perhaps an animation of an educational nature) that functionality will be lost. But there is no reason why the basic text of the site should be unavailable. Avoid nearly-meaningless messages like “Javascript needed to view this site”. If you have something worth showing, tell the user what it is, e.g. “enable Javascript to see animation of solar system”.

2.4.3.4

Switch plug-ins off

The considerations for plug-ins (such as Flash or Java) are very similar to those for Javascript above. Check the site with any plug-ins disabled. The basic text and navigation should still work. Interest the reader sufficiently, and he might just go to the trouble of down-loading the plug-in. Greet him with a blank screen or a “You need Flash to read this site” message and he will probably go away, never to return

2.4.3.5 Switch images off
If scanning a number of sites quickly for information, many readers (including myself) switch images off, for quick loading. Other people cannot view images. So switch images off and check that the site is readable and navigable. This means, in particular, checking that sensible ALT texts have been provided for images. (This check is similar to using a text browser, but not quite the same).

2.5 Other checks
2.5.1 Check non-reliance on mailto

In order to give readers a chance to contact them, web authors often include a link of the form “mailto:abc@xyz.org”. However this unfortunately does not work for anything like all browser/e-mail client combinations. And people in e.g. an Internet cafe cannot use this type of link. Indeed the figure is sometimes circulated that as many as 50% of readers cannot use mailto links. While I doubt whether the true figure is quite this high, it is a substantial number. Many readers prefer a form anyway: the majority of the responses I get to this site come via the contact form rather than the mailto link. Therefore the best thing is to provide a contact page which has both a mailto link and a contact form; the user can then choose which to use.

2.5.2

Check for orphan pages

An orphan page is one that contains no links back up to its ancestors – i.e. to pages higher in the hierarchy of the site. Once one arrives at an orphan page, the only way to get to the rest of the site is via the „back‟ button. Which is fine until people arrive at the page via a search engine, or via a link that someone else gave to them. They cannot then visit the rest of the site. (True, they may be able to get up the hierarchy by lopping off the end of the URL, but this depends on how the site is built, and is in any case not very reader-friendly.) So ensure all pages include a link back up the hierarchy. Orphan pages are particularly easy to overlook in sites with frames. Remember that when one addresses the page directly the other frames are absent.

2.5.3

Check sensible page titles

Check that the page titles (i.e. the contents of the <TITLE> elements) are sensible. Page titles are important, as for example they show up prominently in search-engine results, in bookmarks and also on the tabs of multi-tab browsers such as Opera. Generally speaking, each page of a site should have a unique title. They seem to be somewhat prone to the dreaded cut & paste disease. The “grep” tool is convenient for quickly checking the titles in all your page source files. A quick web search reveals over a million pages with the illuminating title “New Page 1”, while “Untitled” pages run into several millions.

3.0 Functionality
3.1 Links
Objective is to check for all the links in the website.      All Internal Links All External Links All mail to links Check for orphan Pages Check for Broken Links

3.2 Forms
Test for the integrity of submission of all forms.       All Field Level Checks All Field Level Validations. Functionality of Create, Modify, Delete & View. Handling of Wrong inputs (Both client & Server) Default Values if any Optional versus Mandatory fields.

3.3 Cookies
Check for the cookies that has to be enabled and how it has to be expired.

3.4 Web Indexing
Depending on how the site is designed using Meta tags, frames, HTML syntax, dynamically created pages, passwords or different languages, our site will be searchable in different ways.    Meta Tags Frames HTML syntax.

3.5 Database
3.5.1 Data Integrity
Missing or wrong data in table.

3.5.2

Output Error

Errors in writing, editing or reading operations in the tables. The issue is to test the functionality of the database, not the content and the focus here is therefore on output errors. Verify that queries, writing, retrieving or editing in the database is performed in a correct way.

4.0 Usability
4.1 Navigation
Navigation describes the way users navigate within a page, between different user interface controls (buttons, boxes, lists, windows etc.), or between pages via e.g. links.     Application navigation is proper through tab Navigation through Mouse Main features accessible from the main/home page. Any hot keys, control keys to access menus.

4.2 Content
Correctness is whether the information is truthful or contains misinformation. The accuracy of the information is whether it is without grammatical or spelling errors. Remove irrelevant information from your site. This may otherwise cause misunderstandings or confusion.   Spellings and Grammars Updated information

4.3 General Appearance
    Page appearance Color, font and size Frames Consistent design

5.0 Server Side Interfaces
5.1 Server Interface
   Verify that communication is done correctly, Web server-application server, application server-database server and vice versa. Compatibility of server software, hardware, network connections. Database compatibility (SQL, Oracle etc.)

5.2 External Interface (if any)

6.0 Client Side Compatibility
6.1 Platform
Check for the compatibility of      Windows (95, 98, 2000, NT, XP) Unix (different sets) Macintosh (If applicable) Linux Solaris (If applicable)

6.2 Browsers
Check for the various combinations: Internet Explorer (3.X 4.X, 5.X) Netscape Navigator (3.X, 4.X, 6.X) AOL Opera iTune Chrome Browser settings (security settings, graphics, Java etc.) Frames and Cascade Style sheets Applets, ActiveX controls, DHTML, client side scripting HTML specifications. Graphics: Loading of images, graphics, etc.

6.3 Printing
Despite the paperless society the web was to introduce, printing is done more than ever. Verify that pages are printable with considerations on:     Text and image alignment Colours of text, foreground and background Scalability to fit paper size Tables and borders

7.0 Performance:
7.1 Connection Speed
  Try with Connection speed: 14.4, 28.8, 33.6, 56.6, ISDN, cable, DSL, T1, T3 Time-out

7.2 Load
Check/Measure the following: a. What is the estimated number of users per time period and how will it be divided over the period? b. Will there be peak loads and how will the system react? c. Can your site handle a large amount of users requesting a certain page? d. Large amount of data from users.

7.3 Stress
Stress testing is done in order to actually break a site or a certain feature to determine how the system reacts. Stress tests are designed to push and test system limitations and determine whether the system recovers gracefully from crashes. Hackers often stress systems by providing loads of wrong in-data until it crash and then gain access to it during start-up. a. Typical areas to test are forms, logins or other information transaction components. b. Performance of memory, CPU, file handling etc. c. Error in software, hardware, memory errors (leakage, overwrite or pointers)

7.4 Continuous Use
a. Is the application or certain features going to be used only during certain periods of time or will it be used continuously 24 hours a day 7 days a week? b. Will downtime be allowed or is that out of the question? c. Verify that the application is able to meet the requirements and does not run out of memory or disk space.

8.0 Security:
      Valid and Invalid Login Limit defined for the number of tries. Can it be bypassed by typing URL to a page inside directly in the browser? Verify Log files are maintained to store the information for traceability Verify encryption is done correctly if SSL is used (If applicable) No access to edit scripts on the server without authorization.

Checklist before hosting a website: ** Enter “Not Applicable” whichever test not carried out. Test Carried out 1. Functionality 1.1 Links Internal Links External Links Mail to links Orphan pages Broken links Should be present Should be present Should open mailbox Should not be present Should not be present Expected Actual Remarks Ticket Reference

1.2 Forms Field Level checks Checked for length, special characters, numerical characters etc., Checked Unique records, Date validation

Field Level Validation

Functional checks

Create, modify, view and delete are working. Appropriate error messages to be displayed. Mandatory field should not be left blank. Optional should allow the user to skip the field.

Error Handling for wrong inputs or actions. Optional and mandatory fields.

1.3 Cookies Check whether cookies are enabled. **Depends on project

1.4 Web Indexing Meta tags Html Syntax Frames Should be present. Should be valid To be found ok

1.5 Database Data Integrity Should not be any missing or wrong data in the database Errors in writing, reading or editing operations should not be present

Output Errors

2. Usability 2.1 Navigation

Navigation through Mouse Navigation through Tab Main features access

Should be proper Should be proper Should be accessed from home/Main page Should be present

Hot Keys, Control Keys for menu or action

2.2 Content Spelling and Grammar Updated information. To be proper Past events/ information to be removed.

2.3 General Appearance Page Appearance Should not be any overlapping, missing etc., Should be as per standard All frames to be appeared Everywhere in the website consistent layout and design should be carried out.

Colour, font and size Frames Consistent Design

3. Server Side Interface 3.1 Server Interface Communication should be correct with respect to Web server, App server and DB

server Compatibility with server hardware, Sw, network connections Database compatibility Should be proper.

Should be easily portable to other database.

4. Client side Compatibility 4.1 Platform Windows2000, NT Unix Linux Solaris Macintosh Should be working Should be working Should be working Should be working Should be working

4.2 Browsers I.E Should work in all versions above 4.x Should work in all versions above 3.x

Netscape Navigator AOL Any ActiveX controls

Graphics

Load of images, graphics should be proper

4.3 Printing

Text and alignment Colours of text, foreground and background Scalability to fit paper size. Tables and borders.

Should be proper Should be readable. Should print in A4, Letter size. Should be proper.

5. Performance 5.1 Connection speed Should be measured in 14.4, 28.8, 33.6, 56.6, ISDN, Cable and DSL Timeout Should give appropriate time to search. Incorrect message, data loss should not be present.

5.2 Load Estimated users. Peak load Large amount of data from users Per requirements Should withstand Should accept

5.3 Stress System Crash Error in Sw, HW, Memory Should not be present Leakage, overwrite should not

happen.

5.4 Continuous use Estimate whether available for 24 Hrs, 7 days a week Downtime Memory or disk space Try with various timings.

Measure the downtime Should not run out of memory or disk space.

6. Security 6.1 Valid and Invalid Number of tries Should not enter with Invalid login Should not be more than 3 times for invalid try. Should not display information. Should be maintained Authenticated.

Enter url directly without logging in. Logfiles Access to server scripts

9.0 Website Test Automation Requirements:
Assuring Website quality automatically requires conducting sets of tests, automatically and repeatably, that demonstrate required properties and behaviors. Here are some required elements of tools that aim to do this.      Browser Independent Tests should be realistic, but not be dependent on a particular browser. No Buffering, Caching. Local caching and buffering should be disabled so that timed experiments are true measures of performance. Object Mode. Object mode operation is essential to protect an investment in test suites and to assure that test suites continue operating when Website pages experience change. Tables and Forms. Even when the layout of a table or form varies in the browser's view, tests of it should continue independent of these factors. Frames. Windows with multiple frames ought to be processed simply, i.e. as if they were multiple single-page frames

Test Context: Tests need to operate from the browser level for two reasons: (1) this is where users see a Website, so tests based in browser operation are the most realistic; and (2) tests based in browsers can be run locally or across the Web equally well. Local execution is fine for quality control, but not for performance measurement work, where response time including Web-variable delays reflective of real-world usage is essential.

10.0

Website Dynamic Validation:

Confirming validity of what is tested is the key to assuring Website quality the most difficult challenge of all. Here are four key areas where test automation will have a significant impact.

10.1 Operational Testing Individual test steps may involve a variety of
checks on individual pages in the Website: o o Page Consistency: Is the entire page identical to a prior version? Are key parts of the text the same or different? Table, Form Consistency: Are all of the parts of a table or form present? Correctly laid out? Can you confirm that selected texts are in the "right place" Page Relationships: Are all of the links on a page the same as they were before? Are there new or missing links? Are there any broken links? Performance Consistency, Response Times: Is the response time for a user action the same as it was (within a range)?

o

o

10.2 Test Suites
Typically you may have dozens or hundreds (or thousands?) of tests, and you may wish to run tests in a variety of modes: unattended Testing. Individual and/or groups of tests should be executable singly or in parallel from one or many workstations.  Background Testing: Tests should be executable from multiple browsers running "in the background" on an appropriately equipped workstation. Distributed Testing: Independent parts of a test suite should be executable from separate workstations without conflict. Performance Testing: Timing in performance tests should be resolved to the millisecond; this gives a strong basis for averaging data. Random Testing: There should be a capability for randomizing certain parts of tests. Error Recovery: While browser failure due to user inputs is rare, test suites should have the capability of resynchronizing after an error.









10.3 Content Validation
Apart from how a Website responds dynamically, the content should be checkable either exactly or approximately. Here are some ways that content validation could be accomplished: o Structural: All of the links and anchors should match with prior baseline data.

o o o

Checkpoints, Exact Reproduction: One or more text elements in a page should be markable as "required to match". Gross Statistics. Page statistics (e.g. line, word, byte-count, checksum, etc.). Selected Images/Fragments: The tester should be able to rubber-band sections of an image and require that the selection image match later during a subsequent rendition of it.

10.4 Load Simulation
Load analysis needs to proceed by having a special purpose browser act like a human user. This assures that the performance checking experiment indicates true performance -- not performance on simulated but unrealistic conditions. There are many "http torture machines" that generate large numbers of http requests, but that is not necessarily the way real-world users generate requests. Sessions should be recorded live or edited from live recordings to assure faithful timing. There should be adjustable speed up and slow down ratios and intervals.  o o Load generation should proceed from: Single Browser Sessions. One session played on a browser with one or multiple responses. Timing data should be put in a file for separate analysis. Multiple Independent Browser Sessions. Multiple sessions played on multiple browsers with one or multiple responses. Timing data should be put in a file for separate analysis. Multivariate statistical methods may be needed for a complex but general performance model.

11.0

Website Loading and Capacity Analysis:

11.1 Abstract
Estimating the actual capacity of a Website server requires a variety of different types of analyses. There are many deep, highly technical methods for calculating the capacity of a server to meet imposed load. However, a more practical way is to simply measure the perceived response time accurately, and scale up server capacity to make sure that maximum real-user experiences don't exceed unacceptable delays.

11.2 Why is Website Capacity an Issue?
We all have had the exasperating experience of waiting too long for a page to arrive at our Web browser. Ultimately, if the response time is too long, we "click away" and do something else. Even when the Web is heavily saturated with requests, if you are patient enough every page you request will -- ultimately -- be delivered to your browser. But that's not good enough. Too slow response times turn users away, or, worse yet, because the user has moved on to another page or context, important session data could be lost.

11.3 How Much Capacity Is Enough?
The Website sever is configured properly and effectively when it has "enough capacity to meet the customer demands."  How Do You Know If Your Server Capacity Is Large Enough? Good engineering practice suggests that Web server machines have a Safety Factor of 2 or more. This means, that, when serving the design load, the Web servers ought to be running at about 50% of maximum capacity. There is very good software available that will indicate how many URLs are being requested per second, and how many bytes are downloaded per second. If you know the server machine capability you could average these figures and estimate accurately if the average capacity to deliver pages exceeds the average demand. But this leaves unanswered the key question: What will the peak load be?  How Do You Know If Your Server Capacity Is Not Large Enough? Another way to look at this is to look for the reverse information: when is the server delivering a request too slowly? While you may never really know what the peak deliverable capacity of the server is, you might be able to tell fairly accurately when the imposed server load (i.e. the queue of incomings requests) is not being served within a specified time limit, say 10 seconds. Then you surely will know when there is NOT enough capacity -- when this limit is exceeded.

But if the Website involves a two-tier or three-tier structure -- which is increasingly common for e-commerce sites -- then measuring one machine may lead to false conclusions. For example, all three machines in a three-tier structure could be achieving their performance goals -- that is, not setting off alarms -- but the overall cumulative application could still be "too slow" to satisfy users.  Can You Measure Response Time From a Typical Users' Perspective? It may be simpler to measure response time at the client side of the picture, i.e. from the browser. If you can pre-establish a multi-case scenario that is typical of what users do, then it is possible to engineer series of experiments that measure Website performance against that specific scenario. Even though it is relatively easy to measure if such tests run slower than a threshold -- for example, the 10 second overall response time limit -- this approach has a fundamental limit: it is only as good as the set of scenarios that you develop to emulate actual Website use.

11.4 What Affects Website Performance?
Many factors affect how fast a Website appears to a user. There are many, many stages between a RETURN typed on a browser and a completed [or complete enough] page being rendered on the client.  Client Machine Speed and LAN Factors. Before the request is delivered to the Web by the client machine it may have to work its way through the local LAN and enter the Web, where the connect speed and local saturation affects performance. The Outbound Request to the Server. Before the request gets to the server the Web routing and other technical overhead produces delays. The Server Response Time. After the request gets to the server the response speed is affected by the current request backlog, where the request is in a multi-layer queue, and how long-running the last-tier request (typically some kind of database access) takes. The Inbound Delivery to the Client. After the response is delivered by the server the reverse of the above sequence takes over: the delivered pages have to wend their way through the Web back to the client. The Client Response Time. After the data arrives at the client access point client machine speed and local LAN factors come into play again. The Client Rendering. The browser now has all the parts it needs; it is a matter of how long it takes the browser to render the page/image -- or at least a minimal part of it -- so that the user can take action.

 

  

Note: An apology is due for the simplifications here. In fact, the sequence is rather more complex because all of the machines along the way between a request by a browser and a response seen in the browser involve, typically, multi-programming, multi-threaded executions that dynamically adapt to changing Web conditions that are at least in part adjusting to the actual requests that are being discussed. Left out are such technologies as threading requests via different routs, packet re-transmits and asynchronous arrivals, and much LAN protocol complexity.

Still, the bottom line is clear: elapsed time perceived by the user actually occurs, as they say, in true real time.

11.5 How Do You Impose Loads on a Website?
The main goal of creating an artificial load to impose on a Website is to permit the load to emulate one or a dozen or a hundred or thousands of users actually using the Website. There are two main ways to do this. Caveats: On the Web, this may be quite difficult. On the LAN this is easier, but may require a 100Mbps LAN to saturate the servers. Hope: At some point the capacity will scale linearly: 2X machines means 2X capacity.  HTTP Protocol Based. No matter what a user does on a Website the HTTP protocol prevails, so it is natural enough to start thinking about imposing load by finding a way to simulate the HTTP protocol. You can relatively easily create or record HTTP requests by a browser to the candidate Website. There are many tools that can do the get URL page [a simple command/action] -and do this under user control. A particularly simple one is called TorturePL and runs in PERL on UNIX machines. While relatively easy to use to generate basic retrievals of pages, this approach suffers from the fact that all of the URLs associate with a page have to be included if the simulation is to be a realistic one. The disadvantage is that you could easily create a test scenario that fails to include important load factors such as download times for images and other slow-to-respond page components.  Browser Based Playback Simulations. In this approach, the test scenarios are scripts that control a test browser and, working through the browser, emulate the typical users actions, responses and timings. This method has the advantage of reality, but may involve more work in deciding on a scenario and making sure that the scenario plays back realistically.

11.6 Capacity Testing Exercise We can now outline a basic experiment format that provides a high level of realistic loading of a candidate Website server using eValid's unique client-side browser based testing technology. The goal of each experiment is to determine the ability of the subject Website to sustain load from a varying number of realistic simulated users. Generally this goal is obtained if we can develop a set of response-time curves that illustrate typical average response times from the server via the Web, as a function of Website load that has been imposed.
 Phase 1: Create Typical Scripts. Using the eValid toolset, we record a collection of scripts that represent typical usage of the candidate Website. The recordings will be made with the Real Time Recording switched ON so that all

wait times are kept in the script. A few tests will generally allow gaining enough basic experience with the candidate Website so that the wait times are typical of a normal user. Experts in use of the Website may also contribute eValid scripts. To make the overall test scenario as realistic as possible these recorded tests should include: o o o Some small tests, involving only some simple navigation around the site. Some medium tests, involving basic operations on the site. Some long tests that involve deep operation of critical features of the Website.

After recording we make sure each test script is self-restoring, that is, we make sure that running the test script multiple times will not perturb the Website being measured in a way that will affect the results. This is a relatively common attribute of a regression test and may involve the use of test accounts or special logins that avoid irreversible second-tier or third-tier database moves. At this point we can calculate, for each test script, the load each script represents in terms of the total number of URLs involved and the size [or average size] of the URLs retrieved. From this we can make basic estimates of the URLs/sec and Kbytes/sec that might represent a serious load on the candidate Website.  Phase 2: Identify The Test Bed. Normally all tests are run without any data in the browsers' caches -- doing this assures that the tests do in fact really download all of the relevant URLs. [Otherwise one might make measurement runs of the client machine performance rather than of the performance of the servers for the candidate Website.] The bandwidth from the machine or machines running the scenario affects the meaningfulness of the tests. The bandwidth available from the test machines to the Website being tested should be large enough that the duty cycle does not exceed 50%. PC's with large enough memory and high-enough speed access, e.g. T1 or T3, may have to be identified for large experiments.  Phase 3: Establish Scenarios of Tests. The collection of tests recorded in Phase 1 need to be organized into groups, with specified delay multipliers and playback repetition counts of each test taken into consideration. Each test can be given individual local test names to make the reporting easier. Guidelines for scenario design include: o Total running time. The total number of tests and the repetition counts should be enough so that a "steady state" can be reached and

o o

sustained for a long-enough period of time to have confidence that the Website server performance at that level is adequate. Scale up. The tests should begin with a smaller scenario and then scale up on the 1/2/5/10 ratio. Data Rate. The total data rate for all of the activity generated while the scenario runs should be less than the available bandwidth from the client to the server. Otherwise the test scenario will tend to first saturate the channel rather than the server.

This planning should be done in advance insofar as possible because the higher loadings will likely involve use of substantial resources. For example if we are testing a high-capacity site, capable of 1000's of simultaneous users, then the tests will have to consume the corresponding amount of bandwidth.  Phase 4: Establish Load Target. Tests you run on a Website using eValid technology will, unless interrupted, run to completion. The same is true for your own work on a Website. Unless you "click away" even the slowest Website will eventually finish downloading. This fact is important because if you load up a site with 1000's of users using eValid playbacks ultimately, depending on the server capacity and the bandwidth of the pipeline to the client, all of the requested pages will, ultimately, be downloaded successfully. Hence, the only really effective measure of the capacity of a Web server is how fast it succeeds in delivering pages to your client, relative to how fast it could do that if the server and the intervening WWW infrastructure were all infinitely fast. This time is called the unloaded performance time or the base performance time. A very simple way to establish measurable capacity criteria for your Web server is to require that the slowest overall test time -- i.e. the overall test time for a script in the heaviest load that you expect the server to have -- is always below some multiplier over the base performance time. A good, practical measure you can set in advance for this is 2X or 3X or 4X or even 5X. This figure is called the server slowdown value. It is a fixed value that your server must never exceed for a specified server capacity. If you choose, for example, a 2X factor, that means your server will be judged to be at 2X server capacity when the average download time of a particular scenario is no more than 2 times as long as the base performance time.  Phase 5: Run Tests. If the planning described has been well done running the tests should be straightforward. The basic load experiment structure to be imposed by each scenario is known and the sequence in which to run them is explicit in the scenario definition file. The first part is to establish the base performance time for a very lightly loaded server. After this is established, the tests are run up in stages to the target maximum load. The average response time is measured in each successive stage.



Phase 6: Estimate Capacity. As you run increasingly complex scenarios, you build up a simple average time chart like the following. Note that the base performance time is at the left of the chart.

For example, if you have chosen a 2X-3X server capacity target, then this chart suggests that your Website server, as measured by the tests you have chosen, will saturate after the Nmax = 800 value shown. So, you have a server with a capacity of 800 simultaneous users.

11.7 Summary
While the issues of measuring Website server capacity are many and complex, what becomes clear is that users' perception of effective, useful response time is the dominant factor in assuring adequate capacity. This can be accomplished by realistic browser-based experiments that measure aggregate response time as a function of gradually increased parallelism. Estimating the actual capacity of a Website server requires a variety of different types of analyses. There are many deep, highly technical methods for calculating the capacity of a server to meet imposed load. However, a more practical way is to simply measure the perceived response time accurately, and scale up server capacity to make sure that maximum real-user experiences don't exceed unacceptable delays.

12.0
12.1 Abstract

The Website Quality Challenge:

Because of its possible instant worldwide audience a Website's quality and reliability are crucial. The very special nature of the WWW and Websites pose unique software testing challenges. Webmasters, WWW applications developers, and Website quality assurance managers need tools and methods that can match up to the new needs. Mechanized testing via special purpose WWW testing software offers the potential to meet these challenges.

12.2 Introduction
Websites are something entirely new in the world of software quality! Within minutes of going live, a WWW application can have many thousands more users than a conventional, non-WWW application. The immediacy of the WWW creates an immediate expectation of quality and rapid application delivery, but the technical complexities of a Website and variances in the browser make testing and quality control more difficult, and in some ways, more subtle. Automated testing of Websites is both an opportunity and a challenge.

12.3 Defining Website Quality and Reliability
A Website is like any piece of software: no single, all-inclusive quality measure applies, and even multiple quality metrics may not apply. Yet, verifying user-critical impressions of "quality" and "reliability" take on new importance.

12.3.1 Dimensions of Quality
There are many dimensions of quality, and each measure will pertain to a particular Website in varying degrees. Here are some of them:    Time: Websites change often and rapidly? How much has a Website changed since the last upgrade? How do you highlight the parts that have changed? Structural: How well do all of the parts of the Website hold together. Are all links inside and outside the Website working? Do all of the images work? Are there parts of the Website that are not connected? Content: Does the content of critical pages match what is supposed to be there? Do key phrases exist continually in highly-changeable pages? Do critical pages maintain quality content from version to version? What about dynamically generated HTML pages? Accuracy and Consistency: Are today's copies of the pages downloaded the same as yesterday's? Close enough? Is the data presented accurate enough? How do you know? Response Time and Latency: Does the Website server respond to a browser request within certain parameters? In an E-commerce context, how is the end to end response time after a SUBMIT? Are there parts of a site that are so slow the user declines to continue working on it? Performance: Is the Browser-Web-Website-Web-Browser connection quick enough? How does the performance vary by time of day, by load and usage?

 



Is performance adequate for E-commerce applications? Taking 10 minutes to respond to an E-commerce purchase is clearly not acceptable!

12.3.2 Impact of Quality
Quality is in the mind of the user. A poor-quality Website, one with many broken pages and faulty images, with Cgi-Bin error messages, etc. may cost in poor customer relations, lost corporate image, and even in lost revenue. Very complex Websites can sometimes overload the user. The combination of Website complexity and low quality is potentially lethal to an Ecommerce operation. Unhappy users will quickly depart for a different site! And they won't leave with any good impressions.

12.4 Website Architecture
A Website can be complex, and that complexity -- which is what, provides the power, of course -- can be an impediment in assuring Website Quality. Add in the possibilities of multiple authors, very-rapid updates and changes, and the problem compounds. Here are the major parts of Websites as seen from a Quality perspective.

12.4.1 Browser
The browser is the viewer of a Website and there are so many different browsers and browser options that a well-done Website is probably designed to look good on as many browsers as possible. This imposes a kind of de facto standard: the Website must use only those constructs that work with the majority of browsers. But this still leaves room for a lot of creativity, and a range of technical difficulties.

12.4.2 Display Technologies
What you see in your browser is actually composed from many sources:    HTML. There are various versions of HTML supported, and the Website ought to be built in a version of HTML that is compatible. And this should be checkable. Java, JavaScript, ActiveX. Obviously JavaScript and Java applets will be part of any serious Website, so the quality process must be able to support these. On the Windows side, ActiveX controls have to be handled as well. Cgi-Bin Scripts. This is link from a user action of some kind (typically, from a FORM passage or otherwise directly from the HTML, and possibly also from within a Java applet). All of the different types of Cgi-Bin Scripts (perl, awk, shell-scripts, etc.) need to be handled, and tests need to check "end to end" operation. This kind of a "loop" check is crucial for E-commerce situations. Database Access. In E-commerce applications either you are building data up or retrieving data from a database. How does that interaction perform in



real world use? If you give in "correct" or "specified" input does the result produce what you expect? Some access to information from the database may be appropriate, depending on the application, but this is typically found by other means.

12.4.3 Navigation
Users move to and from pages, click on links, click on images (thumbnails), etc. Navigation in a Website often is complex and has to be quick and error free.

12.4.4 Object Mode
The display you see changes dynamically; the only constants are the "objects" that make up the display. These aren't real objects in the OO sense; but they have to be treated that way. So, the quality test tools have to be able to handle URL links, forms, tables, anchors, buttons of all types in an "object like" manner so that validations are independent of representation.

12.4.5 Server Response
How fast the Website host responds influences whether a user (i.e. someone on the browser) moves on or continues. Obviously, Internet loading affects this too, but this factor is often outside the Webmaster's control at least in terms of how the Website is written. Instead, it seems to be more an issue of server hardware capacity and throughput. Yet, if a Website becomes very popular -- this can happen overnight! -loading and tuning are real issues that often are imposed -- perhaps not fairly -- on the WebMaster.

12.4.6 Interaction & Feedback
For passive, content-only sites the only issue is availability, but for a Website that interacts with the user, how fast and how reliable that interaction is can be a big factor.

12.4.7 Concurrent Users
Do multiple users interact on a Website? Can they get in each others' way? While Websites often resemble conventional client/server software structures, with multiple users at multiple locations a Website can be much different, and much more complex, than complex applications.

12.5 Assuring Website Quality Automatically
12.5.1 Test Sessions
Typical elements of tests involve these characteristics:   Browser Independent: Tests should be realistic, but not be dependent on a particular browser, whose biases and characteristics might mask a Website's problems. No Buffering, Caching: Local caching and buffering -- often a way to improve apparent performance -- should be disabled so that timed experiments are a true measure of the Browser-Web-Website-Web-Browser response time.

 

Fonts and Preferences: Most browsers support a wide range of fonts and presentation preferences, and these should not affect how quality on a Website is assessed or assured. Object Mode: Edit fields, push buttons, radio buttons, check boxes, etc. All should be treatable in object mode, i.e. independent of the fonts and preferences. Object mode operation is essential to protect an investment in tests and to assure tests' continued operation when Website pages change. When buttons and form entries change location -- as they often do -- the tests should still work. When a button or other object is deleted, that error should be sensed! Adding objects to a page clearly implies re-making the test.

 

Tables and Forms: Even when the layout of a table or form varies in the browser's view, tests of it should continue independent of these factors. Frames: Windows with multiple frames ought to be processed simply, i.e. as if they were multiple single-page frames.

12.5.2 Test Context
Tests need to operate from the browser level for two reasons: (1) this is where users see a Website, so tests based in browser operation are the most realistic; and (2) tests based in browsers can be run locally or across the Web equally well. Local execution is fine for quality control, but not for performance measurement work, where response time including Web-variable delays reflective of real-world usage is essential.

12.6 Website Validation Processes
12.6.1 Operational Testing
Individual test steps may involve a variety of checks on individual pages in the Website:     Page Quality. Is the entire page identical with a prior version? Are key parts of the text the same or different? Table, Form Quality. Are all of the parts of a table or form present? Correctly laid out? Can you confirm that selected texts are in the "right place". Page Relationships. Are all of the links a page mentions the same as before? Are there new or missing links? Performance, Response Times. Is the response time for a user action the same as it was (within a range)?

12.6.2 Test Suites
Typically you may have dozens or hundreds (or thousands?) of tests, and you may wish to run tests in a variety of modes:       Unattended Testing. Individual and/or groups of tests should be executable singly or in parallel from one or many workstations. Background Testing. Tests should be executable from multiple browsers running "in the background" [on an appropriately equipped workstation]. Distributed Testing. Independent parts of a test suite should be executable from separate workstations without conflict. Performance Testing. Timing in performance tests should be resolved to 1 millisecond levels; this gives a strong basis for averaging data. Random Testing. There should be a capability for randomizing certain parts of tests. Error Recovery. While browser failure due to user inputs is rare, test suites should have the capability of resynchronizing after an error.

12.6.3 Content Validation
Apart from how a Website responds dynamically, the content should be checkable either exactly or approximately. Here are some ways that should be possible:     Structural. All of the links and anchors match with prior "baseline" data. Images should be characterizable by byte-count and/or file type or other file properties. Checkpoints, Exact Reproduction. One or more text elements -- or even all text elements -- in a page should be markable as "required to match". Gross Statistics. Page statistics (e.g. line, word, byte-count, checksum, etc.). Selected Images/Fragments. The tester should have the option to rubber band sections of an image and require that the selection image match later during a subsequent rendition of it. This ought to be possible for several images or image fragments.

12.6.4 Load Simulation
Load analysis needs to proceed by having a special purpose browser act like a human user. This assures that the performance checking experiment indicates true performance -- not performance on simulated but unrealistic conditions. Sessions should be recorded live or edited from live recordings to assure faithful timing. There should be adjustable speed up and slow down ratios and intervals. Load generation should proceed from:  Single Browser. One session played on a browser with one or multiple responses. Timing data should be put in a file for separate analysis.





Multiple Independent Browsers. Multiple sessions played on multiple browsers with one or multiple responses. Timing data should be put in a file for separate analysis. Multivariate statistical methods may be needed for a complex but general performance model. Multiple Coordinated Browsers. This is the most-complex form -- two or more browsers behaving in a coordinated fashion. Special synchronization and control capabilities have to be available to support this.

12.7 Situation Summary
All of these needs and requirements impose constraints on the test automation tools used to confirm the quality and reliability of a Website. At the same time they present a real opportunity to amplify human tester/analyst capabilities. Better, more reliable Websites should be the result.

13.0
13.1 Introduction

Website Mapping:

Many properties of important Websites are only detectable when they are seen by the user from a browser. For example, on a Website with dynamically generated pages which are produced from several different servers and final-assembled just before delivery to the client browser, some parts might actually be missing or unavailable. The server may not see this, but the end user -- looking at the generated page from a browser -- certainly sees this as a problem. Also, how long a page takes to download -- a factor that is affected by server speed, the speed of the web, and the speed of the connection from the web to the client browser -- might check out OK from the servers' point of view, but might result in a too-slow page delivered to the user, who then clicks away to another site. These are just two of many reasons that point to the need to have detailed analysis of pages -- or of collections of pages, i.e. whole Websites -- analyzed from the perspective of a user, i.e. from a browser. Here are some browser-side questions that, having easy and quick and accurate answers to, could be very important:       Do the pages on the site download and render quickly enough? Are there broken images on the pages as they are rendered? Does the page contain links that go to unavailable resources? Are there obvious typos or misspellings on the page? Are there JavaScript mistakes on the page? How old are the pages?

Note that this list does not include "HTML Errors" because in most cases the browsers overcome most such errors by simply ignoring incorrect HTML. HTML correction is a different topic entirely and, it may be important to emphasize, a high fraction of perfectly OK Website pages score very low on the "HTML Perfection" scale!

13.2 Site Analysis Requirements
An automated Site Analysis engine has to meet some basic components:  A way of deciding where a search is to begin, how it is to run, and when to stop it.  A way to record (for completeness checking) which pages really were visited after the first page.  A way to decide what to do with (i.e. how to analyze) pages as they are selected and displayed.  A method for reporting problems found when tests made of the pages show up problems. The nature of a search engine based within a browser is actually similar to the commonly used idea of a spider program, because it would start at some point and

then create an internal worklist based on what it has just found as it recursively descends through a series of links. But for the goals of automatic Website analysis, you really don't want the search to drive all over the web. You really want the search focused like this:  From a specified starting Website page,  For all pages that are linked to that page (below that page),  And continuing until some limit is hit [see below]. In other words, you want your analyses to be constrained and controlled searches of a Website or a sub-Website, where you can easily control the search criteria. You don't want your search to run forever, but you do want the search to be over a large enough spans of pages so that the results of doing the analysis are interesting and valuable.

13.3 Site Analysis Engine Architecture
Here are the general requirements on the Site Analysis Engine that support the above goals:  Search Size Specification: From a user specified starting page, i.e. where the browser is currently positioned. To all links on that page out to a specified depth of search. With and without visiting pages that are not part of the current Website under test. Less than a specified total maximum number of pages to be visited. Less than a specified maximum amount of search and/or download time.  Search Type Specification: You should be able to specify both the types of links to search, and the types of protocols to search. Link type specifications should be by type of file, e.g. *.HTML or *.phtml or *.jpg, or by type of protocol, e.g. HTTP or HTTPS or FPT.  Search Inclusions/Exclusions: You should be able to provide a list of specific URL to exclude during a search (e.g. you might do this to prevent going to a page that logs your analysis session out after having logged in. You should also be able to indicate URLs that you wish to add to the search tables if they happen to be encountered (e.g. other sub-Websites).  Search Modes: You should have the option of seeing everything happen in the current browser (this will require it to have focus), or you should have the option to run in the background. (There may be some performance savings possible, if less than full analysis is done in background mode analyses.)  Cache Control: With no cache at all, with cache cleared before starting the search, or with cache fully operational.

13.4 Reporting
Certain standard reports are always generated by the site analysis process. These include all of the following (some may be optional but some are needed as evidence of the completion of the search work).

13.4.1 Sitemap Report
This Sitemap report is the record of the pages visited that also shows the way in which the search engine came to visit the page. This report is generated as a result of the process of doing the search. Even if there are no filters running, the Sitemap report is an important record of what ULRs were and were not reached. Two kinds of reports that are of particular interest:  Full URL Report. This reports shows every page (by its URL) and for that page the set of pages that depend on it (i.e. have links on it). There would be no need to show all the details of each page, but for completeness all of the pages below a page would need to be shown.  Unique URLs Report. This reports shows every page (by its URL) and for that page shows only the pages that were actually visited. You could expect that this would be a much smaller list, particularly for a Website that has a lot of connectivity.

13.4.2 Filter Reports
The outputs of the filters need to be very simple, easy to read, and generated in real time, as the search is going on. (This gives the user the information as soon as possible and prevents generating reports that contain too much information.) Given the ability of the site analysis process to systematically visit every page in a Website -- subject to the limits imposed by the user -- it is pretty easy to imagine the kinds of reports it would be interesting to generate from page by page analysis as each page is presented.  Pages Loading Slower Than Report: This uses the browser timing capability to measure the actual exact download time of the page (including all of its components and images). Downloading time can be expected to be correlated with page size, but not necessarily. The best use of the data is to include in the final list only those pages that are slower than a threshold, and then to sort these pages by URL in reverse time order.  Pages Larger Than Report: If a page is too big, it is a candidate for revision. Page size can be expected to be correlated with downloading time, but not necessarily. The report should show total byte count of pages only if the size exceeds a user-specified thresholds. The reported pages should be sorted in decreasing size order.  Pages Older Than Report: If pages are old, they may represent out of date information. For pages that are dynamically generated, of course, the page age would be essentially zero. But older pages on a site may be a signal that something important may be wrong.  Broken or Unavailable pages Report: From the browser's point of view, a page could be broken or unavailable for a wide range of reasons. It could be, of course, actually missing (a type 404 error). Or it could be from a server that is temporarily unavailable or cannot be reached right now. All pages that the search shows as unavailable should be marked with an indication of what caused the failure.  Off-site Pages Report: Many Websites reference a lot of off-Website pages, and it may be useful to know which links are to off-site pages. This filter lists them in the order of discovery.

 Pages Matching Search Criteria Report: This may be the most powerful kind of page by page analysis to have available. Every page would be searched, at varying levels of detail, for a match (or a non-match) on a specified string or strings. The report would show the pages that match (or don't match) the search criteria in the order in which the pages are visited. One possible application of this feature is as a security check mechanism. To confirm there was no unauthorized manipulation of pages, the server could place a particular code on every page in a hidden field. The site analysis engine could search for pages that do NOT have this code -- thus revealing any pages that are not "original". The level of detail of the scanning of individual pages needs to be controlled. At one end of the spectrum you might want to look at everything in the page, including JavaScript and HTML and hidden fields -- everying. On the other hand you may be concerned only about what the user sees on the page -- the visible text. Another option might be to search only in the META text fields on the page.

14.0

Testing System Characteristics:

Considering all of these disparate requirements, it seems evident that a single product that supports all of these goals will not be possible. However, there is one common theme: the majority of the work seems to be based on "...what does it [the Website] look like from the point of view of the user?" That is, from the point of view of someone using a browser to look at the Website. This observation led our group to conclude that it would be worthwhile trying to build certain test features into a "test enabled web browser", which we called CAPBAK/Web in the expectation that this approach would let us do the majority of the Website quality control functions using that engine as a base. Browser Based Solution - With this as a starting point, we determined that the browser based solution had to meet these additional requirements:   Commonly Available Technology Base. The browser had to be based on a well known base (there appear to be only two or three choices). Some Browser Features Must Be Deletable. At the same time, certain requirements imposed limitations on what was to be built. For example, if we were going to have accurate timing data we had to be able to disable caching because otherwise we are measuring response times within the client machine rather than "across the web." Extensibility Assured. To permit meaningful experiments, the product had to be extensible enough to permit timings, static analysis, and other information to be extracted.



Taking these requirements into account, and after investigation of W3C's Amaya Browser and the open-architecture Mozilla/Netscape Browser we chose the IE Browser as our initial base for our implementation of CAPBAK/Web. User Interface - How the user interacts with the product is very important, in part because in some cases the user will be someone very familiar with Website browsing and not necessarily a testing expert. The design we implemented takes this reality into account.   "C" Scripting. We use interpreted "C" language as the control language because the syntax is well known, the language is fully expressive of most of the needed logic, and because it interfaces well with other products. Files Interface. We implemented a set of dialogs to capture critical information and made each of them recordable in a text file. The dialogs are associated with files that are kept in parallel with each browser invocation: Keysave File: This is the file that is being created the file is shown line by line during script recording as the user moves around the candidate Website. Timing File: Results of timings are shown and saved in this file. Messages File: Any error messages encountered are delivered to this file. For example, if a file can't be downloaded within the user-specified maximum

o o 

 

 





 

time an error message is issued and the playback continues. (This helps preserve the utility of tests that are partially unsuccessful.) Event File: This file contains a complete log of recording and playback activities that are useful primarily to debug a test recording session or to better understand what actually went on during playback. Operational Features. Based on prior experience, the user interface for CAPBAK/Web had to provide for several kinds of capabilities already known to be critical for a testing system. Many of these are critically important for automated testing because they assure an optimal combination of test script reliability and robustness. Script Capture/Replay: We had to be able to capture a user's actual behavior online, and be able to create scripts by hand. Object Mode: The recording and playback had to support pure-Object Mode operation. This was achieved by using internal information structures in a way that lets the scripts (either recorded or constructed) refer to objects that are meaningful in the browser context. [Adjustable] True-Time Mode: We assured realistic behavior of the product by providing for recording of user-delays and for efficient handling of delays by incorporating a continuously variable "playback delay multiplier" that can be set by the user. Playback Synchronization: For tests to be robust there must be a built-in mode that assures synchronization so that Web-dependent delays don't interfere with proper Website checking. CAPBAK/Web does this using a proprietary playback synchronization method that waits for download completion (except if a specified maximum wait time is exceeded). Timer Capability: To make accurate on-line performance checks we built in a 1 millisecond resolution timer that could be read and reset from the playback script. Validate Selected Text Capability: A key need for Website content checking, as described above, is the ability to capture an element of text from an image so that it can be compared with a baseline value. This feature was implemented by digging into the browser data structures in a novel way (see below for an illustration). The user highlights a selected passage of the web page and clicks on the "Validate Selected Text" menu item.

Figure 1. Illustration of CAPBAK/Web Validate Selected Text Feature.

What results is a recorded line that includes the ASCII text of what was selected, plus some other information that locates the text fragment in the page. During playback if the same text is not found at the same location an error message is generated. o Multiple-playback: We confirmed that multiple playback was possible by running separate copies of the browser in parallel. This solved the problem of how to multiply a single test session into a number of test sessions to simulate multiple users each acting realistically. Test Wizards - In most cases manual scripting is too laborious to use and making a recording to achieve a certain result is equally unacceptable. We built in several test wizards that mechanize some of the most common scriptwriting chores. Link Wizard: This wizard creates a script based on the current Web page that visits every link in the page. Scripts created this way are the basis for "link checking" test suites that confirm the presence (but not necessarily the content) of URLs. FORM Wizard: For E-Commerce testing which involves FORMS we included in the system a FORM Wizard that generates a script that:  Initializes the form.  Presses each pushbutton by name.  Presses each radio button by name.  Types a pre-set script fragment into each text field.  Presses SUBMIT. Here is a sample of the output of this wizard, applied to our standard test page:



o

o

void name() { /* Produced by CAPBAK/Web [IE] Ver. 1.5 Form Wizard */ /* (c) Copyright 1999 by Software Research, Inc. */ WT_InitLink("http://www.testworks.com/Products/Web/CAPBAK/e xample1/"); WT_SubmitForm(FORM:0:12, "RESET FORM"); WT_SelectOneRadio(FORM:0:0, "now", "TRUE"); WT_SelectOneRadio(FORM:0:1, "next", "TRUE"); WT_SelectOneRadio(FORM:0:2, "look", "TRUE"); WT_SelectOneRadio(FORM:0:3, "no", "TRUE"); WT_SelectCheckBox(FORM:0:4, "concerned", "TRUE"); WT_SelectCheckBox(FORM:0:5, "info", "TRUE"); WT_SelectCheckBox(FORM:0:6, "evaluate", "TRUE"); WT_SelectCheckBox(FORM:0:7, "send", "TRUE"); WT_FormTextInput(FORM:0:8, "TestWorks"); WT_FormTextInput(FORM:0:9, "TestWorks"); WT_FormTextInput(FORM:0:10, "TestWorks"); WT_FormTextInput(FORM:0:11, "TestWorks"); WT_SubmitForm(FORM:0:13, "SUBMIT FORM"); }

Figure 2. Sample of Output of FORM Test Wizard. The idea is that this script can be processed automatically to produce the result of varying combinations of pushing buttons. As is clear, the wizard will have pushed all buttons, but only the last-applied one in a set of radio buttons will be left in the TRUE state. o Text Wizard: For detailed content validation this wizard yields up a script that includes in confirmation of the entire text of the candidate page. This script is used to confirm that the content of a page has not changed (in effect, the entire text content of the subject is recorded in the script).

14.1 Example Uses
Early applications of the CAPBAK/Web system have been very effective in producing experiments and collecting data that is very useful for Website checking. While we expect CAPBAK/Web to be the main engine for a range of Website quality control and testing activities, we've chosen two of the most typical and most important applications to illustrate how CAPBAK/Web can be used.  Performance Testing Illustration - To illustrate how CAPBAK/Web measures timing we have built a set of Public Portal Performance Profile TestSuites that have these features: Top 20 Web Portals: We selected 20 commonly available Websites on which to measure response times. These are called the "P4" suites. User Recording: We recorded one user's excursion through these suites and saved that keysave file (playback script). User Recording: We played back the scripts on a 56 kbps modem so that we had a realistic comparison of how long it would take to make this very-full visit to our selected 20 portals. P4 Timings: We measured the elapsed time it took for this script to execute at various times during the day. The results from one typical day's executions showed a playback time range of from 457 secs. to 758 secs (i.e. from -19% of the average to +36% of the average playback time). Second Layer Added: We added to the base script a set of links to each page referenced on the same set of 20 Websites. This yielded the P4+ suite that visits some 1573 separate pages, or around 78 per Website. The testsuite takes around 20,764 secs (~5 Hrs 45 mins) to execute, or an average of 1038 secs per Website. Lessons Learned: It is relatively easy to configure a sophisticated test script that visits many links in a realistic way, and provides realistic user-perceived timing data.

o o o

o

o

o



E-Commerce Illustration - This example shows a typical E-Commerce product ordering situation. The script automatically places an order and uses the Validate Selected Text sequence to confirm that the order was processed correctly. In a real-world example this is the equivalent of (i) selecting an

o

o

o

o

o

item for the shopping basket, (ii) ordering it, and (iii) examining the confirmation page's order code to assure that the transaction was successful. (The final validation step of confirming that the ordered item was actually delivered to a specific address is not part of what CAPBAK/Web can do.) Typical Order Form: We based this script on a typical order form that collects customer information including, for example, a code number (a credit card number). Type-In with Code Number: Starting with the FORM Wizard generated script, we modify it to include only the parts we want, and include the code number 8889999 (See the Figure 1 above). Response File: Once the playback presses the SUBMIT button the WebServer response page shows up displaying the response to the code number. We use the Validate Selected Text (See Figure 1) to capture the response number text. Error Message Generated: If the Cgi-Bin scripts make a mistake this will be caught during playback because the expected exact text 8889999 will not be present, it will be something else. Completed TestScript: Here is the complete testscript for CAPBAK/Web that illustrates this sequence of activities.

void name() { /* Recording by CAPBAK/Web [IE] Ver. 1.5 (c) Copyright 1999 by Software Research, Inc. */ WT_InitLink("http://www.soft.com/Products/Web/CAPBAK/example1/example1broken.html"); WT_SelectOneRadio(FORM:1:0, "buying-now", "TRUE"); WT_SelectOneRadio(FORM:1:1, "next-month", "FALSE"); WT_SelectOneRadio(FORM:1:2, "just-looking", "FALSE"); WT_SelectOneRadio(FORM:1:3, "no-interest", "FALSE"); WT_SelectOneRadio(FORM:1:4, "Yes", "TRUE"); WT_SelectOneRadio(FORM:1:5, "Yes", "TRUE"); WT_SelectOneRadio(FORM:1:6, "Yes", "TRUE"); WT_SelectOneRadio(FORM:1:7, "Yes", "TRUE"); WT_FormTextInput(FORM:1:8, "Mr. Software"); WT_FormTextInput(FORM:1:9, "415-550-3020"); WT_FormTextInput(FORM:1:10, "info@soft.com"); WT_FormTextInput(FORM:1:11, "8889999"); WT_SubmitForm(FORM:1:13, "SUBMIT FORM"); WT_Wait(3425); WT_ValidateText(12, 143, "88899999"); } o Figure 3. Script for E-Commerce Test Loop.

o

Lessons Learned: this example illustrates how it is possible to automatically validate a website using CAPBAK/Web by detecting when an artificial order is mis-processed. Summary: All of these needs and requirements impose constraints on the test automation tools used to confirm the quality and reliability of a Website. The CAPBAK/Web approach offers some significant benefits and technical advantages when dealing with complicated Websites. Better, more reliable Websites should be the result.




				
DOCUMENT INFO
Shared By:
Stats:
views:1256
posted:11/27/2009
language:English
pages:46