Glossary General Terms: Actual Result – The output that occurred at the time the test step was executed. Completion Criteria – Completion of Testing is determined by the execution and review of all test steps for a given set of requirements. Once all known severity 1 defects have been corrected and retested, the test is acceptable. Client Test Plan – A test plan written by the Quality Control department for execution by any clients in order to test software changes during the acceptance-testing phase of the new software feature or correction. This test is also referred to as an Acceptance Test. Defect – Item describing output that deviates from the expected result described in the test step or product behavior that does not conform to product requirements, also known as an anomaly or bug. Defect Metrics – Codes used to categorize defects found during software testing. Domain Expert – Software tester with extensive product knowledge of the application under test, often including past work experience in the actual domain or environment where clients of the software would use the product. Enhancement(s) – New Software feature(s), which may include additions or changes to existing functionality or may create entirely new functionality. Expected Result – Correct output for a test step as determined by associated product documentation and presumption based on the expertise of Quality Control personnel. Software Correction – Programming change intended to correct a software defect. The nature of the change does not affect the product’s intended use or original design specification. Test – A grouping of test steps written by the Quality Control department for execution by Quality Control personnel to exercise a new software feature or correction during the QC testing phase of the new software feature or correction. A test is a detailed, step-bystep set of instructions used as a plan of action and as a means of evaluating the behavior of the application under test (AUT). Test Step – A series of one or more tasks or inputs designed to exercise a specific software feature and produce output for evaluation. All steps will have an associated expected result. Steps are not written for instruction. Implied knowledge of the functionality is often assumed. It is not necessary to record each keystroke to accomplish a test step.
Test Set – A defined group of tests. Each test set is a series of tests to achieve a specific goal; i.e., ignition tests, security tests, regression tests, boundary tests. Test sets may be reusable or non-reusable, e.g. for testing a single bug project. Test Script – A programming script that automates the execution of a test. QC Test Engineers perform scripting with various automation tools, including WinRunner and SmarTerm. Testing Types 1. Component Testing to test newly introduced or replaced components (eg dlls) to determine if they cause problems with code function. This will require knowledge of the software module(s) which utilize the components in order to run functional regression tests. Unit Testing to test within the development organization to determine that it works properly and meets marketing and design specifications. Installation test to test that the system can be installed smoothly using installation procedures and instructions. Have all of the components been included in the compiled, distribution software? Do the installation scripts install the software in the correct directories and complete without error on all supported platforms? Sanity/Ignition test to check the entire system at a basic level (breadth, rather than depth) to see that it is functional and stable on all supported platforms. Have all of the components been included in the compiled, distribution software? This should include basic-level functional tests containing mostly positive checks exercising all major menu categories and functions. This set may also contain medium-level tests with both positive and negative checks. Integration test to verify that a software module works with all suite components that it may interact with. Integration tests ensure that the relevant system functions and interfaces are properly implemented This should include basiclevel functional tests that exercise the intercommunication or interdependencies between software modules. This set may contain basic or medium-level functional tests. Security tests to test access to the system with and without authorization. Does the log on procedure prevent unauthorized users from entering the database? Does the software disallow users access to certain code paths based on their security level? Functional (black box) tests to test the functionality within the application. Does the user interface respond to input as expected? Checks the entire system functionality in depth and breath. The objective of this testing is to cover all code paths. a) terminal emulation interface b) GUI interface c) HL7 interface d) API interface Boundary tests to exercise code at the data and system boundaries. Does the software allow or prevent certain data types or precision levels? What is the maximum/minimum length or precision allowed for data input/output? Is data entry mandatory or not?
2. 3.
4.
5.
6.
7.
8.
9.
10.
11. 12.
13.
Stress tests to test how the software reacts under repeated execution of the same operations. Can the application handle the stress of repeated operations? Does repeated execution result in the same result each time? Identifies the peak load conditions at which the system fails. Performance and load tests to test the performance of the system under normal and load conditions. Does the application respond to input at the expected speed? Helps to determine where bottlenecks occur…client? Server? Network? System software? Application software? This type of testing requires some investigation about user statistics in order to recreate reasonable scenarios. This type of testing can be used to benchmark hardware/software configurations and provide expected response times, Database validation tests to verify that data entry is correctly stored in the appropriate database. Verifies that data is correctly retrieved from the database. Regression tests are functional tests to evaluate maintenance builds. The goal of this type of test is to verify that a change to one part of the software did not break the rest of the application. A regression test includes functional tests with breadth and depth. System tests verify and/or validate the system to its initial objectives. Their purpose is to find those cases in which the system does not work as intended, regardless of the specifications. System test categories include: load/stress volume configuration compatibility security performance installability reliability/availability recovery serviceability human factors
Testing Phases: Unit Testing – Testing of individual programs or lines of code, usually performed by the programmer and may include white-box and/or black box tests. This testing is usually less formal and structured than other forms of testing performed in an organization. Integration Testing – An orderly progression of testing in which software and/or hardware elements are combined and tested until the entire system has been integrated. The purpose of this testing is to verify that each software unit interfaces correctly with the other software units. System Testing – Testing of all modules of a system, usually performed by a professional test group and may include white-box and/or black box tests. This testing is usually performed in a formal and structured manner with well-defined processes and may include functional tests, performance tests, security tests, stress tests, and regression tests as well as other types of tests.
Testing Types: Black-Box Testing – Testing based on external specifications without knowledge of how the system is constructed. Boundary Test– A validation test written to exercise the limits or boundaries of a given field or GUI object. Functional Test – A test written to exercise business requirements, e.g. what the software is supposed to do. The emphasis is on the results of processing, rather than how processing occurs. Ignition Test–A test written to exercise a narrow spectrum of functionality with an emphasis on sanity checks, usually used on new media for a current release. Performance Test–A test written to demonstrate that a system functions in accordance with its performance requirements regarding acceptable response times. Regression Test – A test written to selectively exercise parts of an integrated system to verify that modifications have not caused unintended adverse side effects or to verify that a modified system still meets requirements. This testing is often performed before releasing a new build (group of code changes) to the client population. Stress Testing–Often using performance tests, stress testing is the process of running client machines in high-volume scenarios to see when and where the application breaks under pressure. White-Box Testing – Testing based on knowledge of internal code structure and logic.
Uni t Te sti n g
Un it Testin g is the first
P a r a l l e l / A u di t T e s t i n g
In P a r a l l e l T e s t i n g , t h e u s e r recon cil e s th e ou tp u t of th e n ew system t o th e ou tp u t of verify th at th e n ew system is op erat in g correctl y.
Func ti onal Te s ti n g
Th i s i s a c a t e g o r y o f B l a c k b ox testin g , g ear ed to fu n ction al req u iremen ts of an testin g team
l evel of d yn amic te stin g an d is first th e resp on sib il ity of t h e d e ve l o p e r s a n d t h e n o f th e testers.
th e cu rren t system in ord er to ap p l ication , p erformed b y th e
Usa bi l i t y Te s ti ng
Usab il ity p erformed Te s t i n g to for of is the the
I n t e gr a t i o n T e s t i n g
Up on testin g , comp l etion in teg ration wh ite of u n it testin g b o x) d istin ct of stil l to work in
Sys te m Te sti n g
Af t e r i n t e g r a t i o n t e s t i n g , t h e Te s t i n g system categ ory Te a m te stin g , of wil l wh ich b l ack b eg in is a b ox
ch eck
'u ser-frien d l in ess'
(b asical l y, commen ces. to comp on en ts ap p l ication accord an ce req u iremen ts.
ap p l ication . Cl e arl y th is is s u b j e c t i ve a n d w i l l d e p e n d on th e targ eted en d u ser or cu stomer.
Th e p u r p o s e i s
en su re
t e s t i n g . Th e c o m p l e t e s y s t e m en viron men t accu racy in to val id ate the its
th e is con fig u red in a con trol l ed cu stomer an d comp l eten ess fun ction s
p erformin g
as d esig n ed .
End -t o- En d Te s ti ng
Simil ar to Syst em Test in g , E n d - t o - E n d Te s t i n g i n vo l ve s testin g of a in th at u se, such c omp l ete an m imics ap p l ication en viron men t real -w orl d u sin g commu n icati on s, in teract in g h ard ware, with ap p l ication s,
R e gr e s s i o n T e s t i n g
Th e o b j e c t i v e o f R e g r e s s i o n Te s t i n g i s t o e n s u r e s o f t w a r e rema in s in tact. A b asel in e set of d ata an d th at n ot scrip ts wil l e xe c u t e d the be to main tain ed an d
Sani ty Te s ti ng
San ity Te s t i n g wil l be
p erformed wh en e ver cu rsory t e s t i n g i s s u f f i c i e n t t o p r o ve th at the ap p l ication accord in g a su b set is to of fu n ction in g testin g is
as verify h ave
ch an g es an y
s p e c i f i c a t i o n s . Th i s l e ve l o f Reg ress ion Testin g .
in teract ion w ith a d atab ase, n etwork or oth er or
in trod u ced d u rin g th e rel ease “mad e u n in tend ed mod ificati on s ” in th e cod e.
systems wh er e ap p rop riat e.
Pe rf or mance Te sti n g
Al t h o u g h testin g can be is p erforman ce d escr ib ed reg ard ed l e ve l of te stin g as as a a wil l
Loa d Te s ti ng
Load with the Testin g testin g testin g a is an of a of con cern ed app l ication web l oad s site to
Instal l a ti on Te s ti n g
In s t a l l a t i o n fu l l, Th e with p artial , in stal l ati on wil l the be t estin g or is
c o n c e r n e d w i t h ve r i f y i n g t h e u p g rad e p rocesses. test for a of c on d u cted in stal l /u n in stall rel eas e
p a r t o f S y s t e m Te s t i n g , i t d istin ct
u nd er h eavy l oad s, su ch as ran g e
test in g . u nd er
Perform an ce
d etermin e at wh at p oin t th e systems' r esp on se t ime
verify th e l oad , vol u me, an d
ob j ecti ve
resp on se time s as d efin ed b y req u iremen ts.
d eg rad es or fail s.
d emon stratin g read in ess.
p rod u ction
Se curi ty /Pe ne t ra ti on Te sti n g
R e cove ry /E r ro r Te s ti n g
Th i s t e s t i n g i s c a r r i e d o u t t o
C o m pa t i bi l i t y T e s t i n g
Comp atib il ity Te s t i n g is
S e c u r i t y Te s t i n g v e r i f i e s a s to h ow wel l the p rotects e xt e r n a l itsel f ac cess, ag a in st or wil l fu l
ch eck
h ow
wel l f rom
a or
system c rash es, oth er
c o n c e r n e d w i t h ve r i f y i n g h o w wel l softwa re p erf orms in a p articu l ar h ard ware/s oftwar e/op er atin g system/n etwo rk/etc. en viron men t.
system reco vers h ard ware
fa il u res,
u n a u th o r i z e d d amag e, etc.
in tern al
catastr op h ic p rob l ems.
Co m pa ri s on Te s ti n g
Th i s Te s t i n g to comp ares an d comp etin g
A c c e pt a n c e T e s t i n g
Ac c e p t a n c e T e s t i n g , w h i c h i s b l ack verify fu n ction al ity p rior to the b ox test in g , the an d syst em p r o vi d e s system u sab il ity b ein g th e cl ien t th e op p ortu n ity to
softwar e stren g th s p rod u cts.
we akn esse s
m o ve d t o p r o d u c t i o n .