VII. Conformance Testing Methodology

Document Sample
VII. Conformance Testing Methodology Powered By Docstoc
					521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Outline
General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process

VII. Conformance Testing Methodology

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-1

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-2

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Outline
General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process

What is Conformance Testing?
The process of testing the extent to which implementations of protocol entities adhere to the requirements stated in the relevant standard or specification
Concerned with external behavior (black box) Prerequisites for interoperability
The primary objective of conformance testing is to increase the probability that different product implementations actually interoperate.

Testing of implementations y y But NOT with performance, reliability, fault tolerance, efficiency, etc.

ISO 9646:
A conforming implementation is one which satisfies both static and dynamic conformance requirements, consistent with the capabilities stated in the Protocol Implementation Conformance Statement (PICS)
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-3 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-4 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Standard
ISO/IEC 9646 (ITU-T X.290)
Conformance Testing Methodology and Framework (CTMF) for the Implementations of OSI and ITU Protocols

Conformance Testing Limitations
Aspects of protocol testing not considered in ISO 9646
Interoperability testing, performance testing and robustness testing The Th realization of service primitives li ti f i i iti Internal aspects of the system implementation The system environment and testing tools Procurement procedures Test certificates, and accreditation of test labs

7 part standard
ISO 9646-1. General Concepts ISO 9646-2. Abstract Test Suite Specification ISO 9646-3. Tree and Tabular Combined Notation (TTCN) ISO 9646-4. Test Realization ISO 9646-5. Conformance Assessment Process 9646 5 ISO 9646-6. Protocol Profile Test Specification ISO 9646-7. Implementation Conformance Statement

Conformance testing will always be limited, for technical and theoretical as well as for economic reasons
No guarantee can be given that an implementation absolutely conforms to the relevant protocol standards Conformance testing with positive test results will increase the degree of confidence in a protocol implementation
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-6 Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-5

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Basic Standard vs. Profile
Base standard is a standard relative to a single protocol in a given OSI layer A profile is a set of one or more base standards necessary for accomplishing a particular function
a multilayer stack of several adjacent base standards Function might be a type of application, or a transport service based on specific telecommunications techniques A protocol can be a base standard by itself, or can belong to a profile The purpose of defining a profile is to facilitate interoperability

Conformance Testing Process (1/2)
Test generation
To generate the abstract test suites (ATSs) manually or automatically Development of proformas for implementation conformance statements (ICS) and implementation extra information for testing (IXIT)

Test realization
The realization of test tools and means of testing, capable of executing particular test suites To generate the executable test suites (ETSs) with ATSs
Including both selection and parameterization process

Profile is twofold P fil i t f ld
Multiplicity: a profile may retain options in a set of base specifications Refinement: a profile restricts the choices of options in the base specifications. It may also specify additional conformance requirements.

Test execution
Perform test to an implementation with executable test case selected

Conformance assessment process
Performed at a test laboratory for a client, with a protocol conformance test report (PCTR) as the ultimate result
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-8 Dr. Junzhao Sun

We will mainly concentrate on the conformance testing of protocol
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-7 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Conformance Testing Process (2/2)
Standard Test generation protocol specification

Test Types
Test types in testing process
Basic interconnection tests (BIT)
To determine whether interconnection is possible at all as an initial stage To establish main features of the protocol are implemented Basic interconnection tests are standardized but may be omitted

Abstract test suite

TTCN

Test realization Protocol implementation Executable test suite Test execution

Capability tests
To validate that all the the observable external static capabilities of the IUT are valid with respect to the static conformance requirements expressed in the PICS Capabilities tests are standardized and mandatory

System Under Test IUT

Behavior tests
To check dynamic conformance of an IUT with respect to the dynamic conformance requirements specified in the protocol standard Behaviour tests are standardized and mandatory

Tester

Verdict: (result) • accept • failure • inconclusive • error (in test HW)
7-9 Dr. Junzhao Sun

Conformance resolution tests
Special tests for purposes beyond ordinary behavior testing e.g. exception Conformance resolution tests are not standardized and not mandatory
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-10 Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Test Types

Test Results
Test results
Pass

Behavior tests
Capability tests
Basic interactio i i n tests

Observed test outcome shows that the test purpose has been fulfilled and the IUT has displayed only valid behaviors as specified in protocol standards

Fail
Test outcome proves that some conformance requirement has been violated

Inconclusive
Neither pass nor fail can be assigned to the test outcome Happens when underlying provider signals problems e.g. disconnect or reset

Documentation of test results
Conformance resolution tests SCTR = system conformance test report PCTR = protocol conformance test reports

Requirements on test results
Repeatable, comparable and auditable to a reasonable degree

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-11

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-12

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Outline
General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process

Test Generation

Standard St nd rd Standardized Test Purposes (in English)

TPs
Standardized Abstract Test Suite (in TTCN)

ATS

Standardized (Protocol) Implementation Conformance Statement (tables)

ICS
Standardized (Protocol) Implementation eXtra Information for Testing (tables)

IXIT

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-13

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-14

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Test Specification to Be Produced

Conformance Requirements
Systems are called conforming if their externally visible behavior fulfils conformance requirements stated in protocol specification
Mandatory requirements, always h M d t i t l have t b observed to be b d Conditional requirements, have to be observed under conditions defined
Dependence on previous choices of parameter values and options

Optional requirements, can be freely selected

Static conformance requirements
Specify permitted combinations of capabilities implemented in real system E.g. E g value ranges for size parameters or timers as in ICS timers,

Dynamic conformance requirements
Specify the observable behavior of implementation permitted by standard Define actual protocol e.g. PDU exchanges, timing, format of PDUs.

Profile is based on base specifications with additional constraints
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-15 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-16 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

ICS (1/2)
ICS = Implementation Conformance Statement A checklist of the capabilities supported by the IUT
A questionnaires completed b client t present conformance requirements ti i l t d by li t to t f i t Produced by filling out an ICS proforma with static requirements Provides an overview of the features and options that are implemented Used to select and parameterise test cases and as an indicator for basic interoperability between different products

ICS (2/2)
Standardization organizations
Base Base Base specification Base specification specification specification PICSPICSPICSproforma PICSproforma proforma proforma

RL = Requirements List

Client Profile ICS
PICS PICS PICS PICS

PICS = Protocol Implementation Conformance Statement
IUT realizes a single protocol

Profile ICS
IUT contains a set of interrelated protocols, e.g. a stack of protocols

ICS is not just for testing
Used by specification writer to clarify the intent of standards Used by product implementer for selection of features
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-17 Dr. Junzhao Sun

Profile specification

Profile RL Profile specific ICS proforma

Profile RL Profile specific ICS

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-18

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

ICS Proforma – Protocol Capability
Status of capability in protocol specification
Mandatory, optional , conditional prohibited, excluded hibit d l d d out-of scope, not applicable

ICS Proforma – Structure
ICS proforma corrigenda (version information) ICS proforma instructions, abbreviations and terms Identification of the implementation, supplier and client Identification of the protocol, information object or profile Protocol, information object or profile corrigenda Major capabilities table Supported Supported PDUs Table Supported PDUs Table PDUs Table Supported parameters Supported parameters table in PDU Supported parameters table in PDU table in PDU PICS proforma tables

Capabilities can be grouped into categories
Major capabilities Supported PDUs and fields within PDUs Supported timers Encoding variations E di i ti Negotiation mechanisms Error procedures

Supported timer tables Supported protocol error handling table

Supported negotiation capabilities table Multi-specification dependencies table

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-19

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-20

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

ICS Proforma – ISDN Example
Index Protocol capability Conditions Status Reference Support Does the implementation support… for status M1 M2 M3 M4 M41 3.1kHz audio calls Data calls Packet mode calls Interworking with non-ISDN networks Calling line identification presentation M1 not M1 M1 not M1 O.1 O.1 O M M M N/A M N/A
7-21

IXIT, SCS
IXIT = Implementation eXtra Information for Testing Contains additional information beyond specification as ICS
An A IXIT proforma fill d out b t t l b t f filled t by test laboratory and client d li t With both protocol IXIT and profile IXIT, as in ICS types Specify additional testing information related to an specification
Physical setup, connection of test, and testing environment SUT hardware, socket, telephone numbers, and tester SAP addresses Test-specific timer or parameter values

6.1.2 6.1.1 6.3 8.4 8.4 8.4 8.4

[ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No

SCS = System Conformance Statement
List Li t references t specifications of th IUT related ICS related t t f to ifi ti f the IUT, l t d ICSs, l t d test reports, together with configuration information of SUT, I.e. an index

M421 Calling line identification restriction M422 Advice of charge

Conformance of an IUT: An IUT is said to be conforming if it satisfied the static and dynamic requirements of the relevant standards referenced in SCS and capabilities in ICSs.
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-22 Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

IXIT Proforma – ISDN Example
#PC_BASIC #PC_POINT_TO_POINT #PC POINT TO POINT #PC_POINT_TO_MULTIPOINT #PC_EN_BLOC_RECEIVING #PC_OVERLAP_RECEIVING #PX_CDPN_OCTET3 #PX_CPN_PTC1 #PX_LCPN_PTC2 #PX_CPN_PTC2 TRUE FALSE TRUE TRUE TRUE ’81’O ‘323830313531’O ‘07’O ‘323830313532’O

TSS & TP
TSS = Test Suite Structure TP = Test Purposes TSS & TP are derived from the relevant base standards.
Provide an informal, easy-to-read description of each test case Concentrate on the meaning of the test rather than detailing how it may be achieved TP is a textual description of a single requirement or a set of related requirements which should be tested Test Purposes are grouped into a logical Test Suite Structure according to suitable criteria, e.g., basic interconnection, error handling, functionality etc.

Will be introduced later in ATS section

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-23

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-24

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

ATS & ETS
ATS = Abstract Test Suite
The entire collection of Test Cases (ABSTRACT) Each Test Case specifies th d t il d coding of th T t P E hT tC ifi the detailed di f the Test Purposes, written itt in the standardized test specification language TTCN Can be generated manually from protocol and service standard Can be generated automatically from formal specification of protocol
CATG = Computer-Aided Test Case Generation Some tools are commercially available to generate test suite from e.g. SDL

Test Suites & Specification
Tests apply to a single system under test (SUT), and one protocol standards Exhaustive test of standards specifications
Must cover all features – mandatory & optional Must cover valid, invalid, & inopportune protocol exchanges

Use of standardized test specification may
Improve transparency and increase of the objectiveness for the test process Provide comparability of test results As the criteria for accreditation bodies to assess test laboratories Common test specifications are the basis for harmonized test services Provide a common starting point to suppliers, users, test laboratories and certification bodies

Will be studied later in more detail

ETS = Executable Test Suite
Test cases for truly test execution Can be quickly and easily implemented from the ATS using the TTCN compilers available on most modern test tool platforms
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-25 Dr. Junzhao Sun

Test specification language – TTCN
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-26 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Test Specification Development in ETSI

Typical Test Suites
GSM, GPRS, 3G (UMTS in 3GPP) Broadband Radio Access Networks (HiperLAN) DECT(Digital Enhanced Cordless Telecommunications) Bluetooth Local Loop, Digital Subscriber Line (DSL) IN, INAP (Intelligent Network Application Protocol) ISDN, B-ISDN SMS in PSTN/ISDN i SIP (Session Initiation Protocol), IPv6 IP Cablecom XML (Extensible Markup Language)

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-27

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-28

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Outline
General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process

ATM
ATM = Abstract Test Methods
NOT Automatic Teller Machine NOT Asynchronous Transfer Mode

A set of ATMs are defined by OSI testing methodology, describing
Abstract test configuration Possible types of observation and control Providing for test coordination

ATMs are determined by
Configuration of IUT with respect to SUT Entire testing configuration, I.e. test system + SUT + service provider Where tester can control and observe the IUT Difference degrees of precision of the test coordination procedures

Common test architecture and 4 test methods
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-29 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-30 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

OSI System & IUT
Complete system Partial system Application relay system Network relay system

OSI Terminology
Layer S Service provided SP Layer T PDU Service used Layer T Transport layer Layer S Session layer SP SAP

Application Presentation Session Transport Network Data link Physical

Application Presentation Session Transport Network Data link Physical

Application Presentation Session Transport Network Data link Physical

Application Presentation Session Transport Network Data link
Layer N

Layer N

Physical

Network layer

IUT implement one or a combination of OSI protocols from one layer or several adjacent layers
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-31 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab

SP : Service Primitive SAP : Service Access Point PDU : Protocol Data Unit
7-32 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Conformance Testing Terminology
Upper Tester
ASP S ASP PCO

Conceptual Test Architecture (1/3)
Test operator interface Product operator interface

Test System (TS) ASP: Abstract Service Primitive PCO: Point of Control and Observation Lower Tester Control Function
TCP

System Under Test (SUT) Test Co-ordination Procedures (TCP) Upper Tester Upper Tester Upper Tester
ASPs PCO

PDU

IUT
PCO ASP ASP

IUT:

Implementation Under Test

PDU: Protocol Data Unit

Lower Tester Lower Tester Lower Tester
ASPs

PDU IUT

Lower Tester A PCO maps to a SAP (Service Access Point) in the OSI reference model PCO

The PCO has two FIFO queues: • Send (from tester to IUT) • Receive (by tester from IUT )

Lower level service provider There can be several LTs and UTs being simultaneously used

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-33

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-34

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Conceptual Test Architecture (2/3)
Testing contexts
Single-party testing is for testing an IUT which communicates with exactly one real open system represented by a single lower tester system, Multi-party testing is for testing an IUT communicating with multiple real open systems, represented by more than one lower tester
Configuration for IUT components can be homogeneous or heterogeneous

Conceptual Test Architecture (3/3)
Upper tester (UT) controls and observes IUT’s upper service boundary, by operator access, API, or hardware interface
In i l I single-party context, UT behaves as a user of IUT t t t b h f In multi-party context, UTs working in parallel act as users of IUT

Lower tester (LT) controls and observes the IUT’s lower service boundary, indirectly, via the underlying service provider
In single-party testing, behaves as the p g p y g, peer entity to IUT y In multi-party testing, act as peer entities working in parallel

Test coordination procedures (TCPs) are used to ensure cooperation between the UTs and LTs
How tester shall respond Passing (preliminary) results Synchronisation S h i ti TCP is NOT Transport Control Protocol, as in TCP/IP

Lower tester control function (LTCF) coordinating all LTs
Assign the test case verdicts Mandatory in multi-party context, inapplicable in single-party context
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-35 Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-36

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

ATM Classification
ATMs for multi-party testing
Several parallel upper and lower testers In I complex situation a upper tester control function (UTCF) i needed l it ti t t t lf ti is d d Special cases include only one upper tester, or even no upper tester at all

Test Case

Lower Tester

IUT

Upper Tester

ATMs for single-party testing
Local Test Method (L)
Upper Tester and Lower Tester in Test System

ConnectRequest “CR” ConnectConfirm “CC”

ConnectIndication “CONind” ConnectResponse “CONresp”

Distributed Test Method (D)
Upper Tester in SUT, Lower Tester in Test System SUT

Co-ordinated Test Method (C)
As above, uses Test Management Protocol

Recall service primitives
Request Indication Response Confirm
7-37 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-38 Dr. Junzhao Sun

Remote Test Method (R)
Lower Tester in Test System, no Upper Tester
Dept. Electrical & Information Engineering, Computer Engineering Lab

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Local Test Method
UT ASPs TCP coordination N-layer entity under test ASPs LT
• • • • • • Upper Tester is located in Test System Requires an upper interface on IUT IUT is “built in” the tester No ATSs for this method Good for the testing of a hardware component Example: Ethernet driver
7-39 Dr. Junzhao Sun

Local Test Method

System Under Test (SUT)
2) CONind 3) CONresp

Test System (TS) Upper Tester
TCPs

ASPs PCO

Lower Tester IUT
ASPs PCO
1) CR 4) CC

Dept. Electrical & Information Engineering, Computer Engineering Lab

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-40

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Distributed Test Method
TCP UT LT PDUs IUT ASPs Service provider
• • • • UT in SUT, LT remote Requires synchronisation Suitable for upper layer protocols / protocols offering API Example: socket communication
7-41 Dr. Junzhao Sun

Distributed Test Method

System Under Test (SUT)

Test System (TS) Test Co-ordination Procedures (TCP) Lower Tester
ASPs

ASPs

Upper Tester
2) CONind 3) CONresp

PCO

IUT
PCO
1) CR 4) CC

Dept. Electrical & Information Engineering, Computer Engineering Lab

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-42

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Co-ordinated Test Method
TM-PDUs UT LT PDUs IUT ASPs Service provider
• • • • • UT in SUT but no access, LT remote No assumption of upper interface to the IUT Use only one PCO below the LT Uses Test Management Protocol (TMP) embedded in ASPs Suitable for mid-layer protocols
7-43 Dr. Junzhao Sun

Co-ordinated Test Method

System Under Test (SUT)
2) CONind 3) CONresp

Test System (TS) Test Management Protocol

Upper Tester IUT

Lower Tester
ASPs

PCO
1) CR 4) CC

Dept. Electrical & Information Engineering, Computer Engineering Lab

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-44

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Remote Test Method

Remote Test Method

TCP UT LT PDUs IUT ASPs Service provider

System Under Test (SUT)

Test System (TS)

IUT

Lower Tester

ASPs

PCO

• No Upper Tester • Upper Tester can be native application or a user accessible interface • Manual co-ordination • Limited, but easy to use
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-45 Dr. Junzhao Sun

1) CR 2) CC

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-46

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

ATMs Put Together

Testing Configuration Examples – MHS
MHS = Message Handling Systems
Multi-user in single-party context RTS = Reliable T R li bl Transfer S t f System

TCP UT-2 UT-1 LT MHSPDUs IUT MHS-ASPs

RTSASPs

RT service provider
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-47 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-48 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Testing Configuration Examples – Multiplexing
Multiplexing happens in transport or network layer protocols where more than one connection is multiplexed over a single lower connection, e g transport connections over network one connection e.g. TCP UT-2 UT-1 LT NWNW PDUs Service provider
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-49 Dr. Junzhao Sun

Testing Configuration Examples – Relay
Relay systems’ testing should be in multi-party context, without using UTs

TCP TransportASPs LT –1 PDUs IUT ASPs Service provider
Dept. Electrical & Information Engineering, Computer Engineering Lab

LT –2 PDUs ASPs Service provider
7-50 Dr. Junzhao Sun

NetworkASPs

IUT

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

A Example Test System Architecture
Protocol Tester Interface TC Editor TC Translator TC Library PICS,PIXIT Editor Test Select Test Execute Transmit Receive Service Provider
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-51 Dr. Junzhao Sun

Outline
SUT

Result Analysis&report IUT

General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-52

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

ATS Development Process
ATS = Abstract Test Suites
1. Determine the testable conformance requirements from the relevant specifications, specifications ICS and related documents 2. Determine the test groups & objectives that cover these requirements 3. Develop test purposes covering requirements and test group objectives 4. Choose testing context (single-/multi-party) and an ATM (configuration)
A test suite is always dedicated to a single test method

Recall the Test Generation Process

Standard St nd rd Standardized Test Purposes (in English)

TPs
Standardized Abstract Test Suite (in TTCN)

ATS

5. Specify ATS, including test steps, using standardized notation like TTCN p y parameterisation criteria with respect to the ICS p 6. Specify the selection and p and IXIT proformas

Standardized (Protocol) Implementation Conformance Statement (tables)

ATS should be used to derive ETS by compilation and adaptation to the means of testing (MOT)
MOT is the combination of test devices, equipments and procedures
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-53 Dr. Junzhao Sun

ICS
Standardized (Protocol) Implementation eXtra Information for Testing (tables)

IXIT

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-54

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

ATS Structure (1/2)
Test suite Test T t group Test group Test group Test case Test case Test case Test case Test step Test step Test case Test case Test case Test event Test case Test case Test event Test case Test case Test case Test t T t step Test step Test case Test case Test case Test event
7-55

ATS Structure (2/2)
A test suite consists of test cases with no execution order
Test cases are logically grouped into test group according to test purposes
Each t t E h test case has an individual t t purpose t d h i di id l test to demonstrate a certain behavior t t t i b h i

Test group Test case Test case Test case Test case Test case Test case Test case Test case

A test group objective may be provided for each test group The nested hierarchy should be created in a top-down fashion
Ensure systematic derivation of test purposes Achieve good coverage of all protocol aspects and conformance requirements

A test case may be modularized into nesting test steps
Modularization facilitates test case development and maintenance Group commonly used test steps together into test step libraries
Economical that they need not be specified repeatedly

Test event is the elementary unit within a test case
Test case Test case Test case Test event
Dr. Junzhao Sun

Usually the exchange of a PDU or an ASP at a PCO
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-56 Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

TSS = Test Suite Structure
“which tests are performed on which aspects of the reference specification” Adopt a top down approach deriving TSS from base specification when no technical constraint forbids it, structure TSS as a tree
1. The name of base specification should be placed at the top of the TSS 2. Test groups related to the "major functions", or "roles" of the base specification 3. Pre-defined groups of tests according to the "nature" of the tests:
Basic interconnection tests; capability tests; Valid behaviour t t i V lid b h i tests; invalid b h i lid behaviour t t i tests; inopportune b h i t behaviour t t tests;

TSS FTAM Example
FTAM = File Transfer Access and Management

4. Pre-defined groups of tests according to the "functional aspect" tested:
State event transitions; parameter variations; parameter combinations; timers

5. Groupings relevant to the base specification
E.g. name of functional units, then of PDUs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-57 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-58 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Test Purposes (1/2)
Each test case with one test purpose for meaning description Test purposes focus on one or more related conformance requirements, derived from OSI specification Test purposes and their hierarchical grouping are specified in a TSS & TP (Test suite structure and test purposes) standard
Structure chosen depends largely on personal experience and style

Test Purposes (2/2)
The expression of a TP shall not be limited to checking a state or a state transition of a protocol state machine
Example TP "Check th t th IUT sends a COMMON CONTROL ACK after E l TP: "Ch k that the d ft receipt of a COMMON CONTROL message containing ... , as a result of state transition S1 → S2.".

Test purposes for a test suite have to cover aspects of protocol
Capabilities, in particular mandatory and optional ones Behavior tests, with the tester displaying valid/invalid behavior PDUs sent to/received from IUT, and relationship between the PDUs Protocol phases and protocol machine states Timing, encoding, individual parameters and parameter combinations
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-59 Dr. Junzhao Sun

A clear explanation of the relation between TPs and the conformance requirements in the reference specification shall be provided The TPs shall be designated by the same numbering and referencing scheme as the TSS

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-60

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Combined TPs (CTPs)
The final cost of a test campaign directly depends on the size of the ATS The objective of CTP is to reduce the number of test cases, but to keep a good coverage

TP Combination Principle

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-61

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-62

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

TSS for CTP

Test Purpose Component

Initial condition should define precisely the expected state of the IUT at the beginning of the test
Example: Ensure that the IUT is in state connection established (reception of a connect confirm on the LT on response to a connect request).

Check(s) to be performed
Example: Ensure that the IUT, on reception of a valid SETUP message with the Sending complete information element sends any of a CALL PROCEEDING, ALERTING or CONNECT message.

Verdict criteria should explicitly describe the observable events which will be used to assess the result of the test.
Example: Check that the IUT in state S1, on receipt of a valid connect request, enters state S2 (sends a valid connect confirm).
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-63 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-64 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Testing States
Testing state is a combination of the states of SUT (IUT), test system, and underlying provider which can occur during testing
Stable t t St bl state can b maintained b t be i t i d between t t case without t t activities test ith t tester ti iti
Idle state is that no connection is established, independent of previous tests

Test Case Scheme
A test case is conceptually composed of 3 parts
Test body, the combination of test events for achieving test purpose
Starts i St t in a initial t ti state i iti l testing t t

Test state is the one that test case is truly performed to check test purpose
Initial state the start point of the execution of a test case

Test preamble, takes the test from stable states to initial state Test postamble, takes the test from end states to one stable state
If end state is not unique, it has to be checked by a verification step

End state is the stop state of a test
One test case may stop at several end states

Postamble Stable State Preamble Test Test Body State End State (Test Body) Verification End State (Verification)

The outcome for each test case should be assigned a verdict
Calculated from verdicts assigned to test events in the outcome
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-65 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-66 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Outline
General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process

Test Realization
Test realization is the process of producing a means of testing (MOT) for conformance to protocol standards, by reference to an ATS standard. standard
MOT is a real test system used to execute a test suite on IUTs
Combination of equipment and procedures that derive, select, parameterize and execute the test cases of a given ATS, together with produce conformance log

Does not mean the actual execution of test cases

ETS derivation is to convert the ATS into final ETS
Selection of applicable test cases (abstract or executable)
Performed mainly on the basis of ICS (Imple. Conformance Statement)

Parameterization of selected test cases (abstract or executable)
Performed mainly on the basis of IXIT (Imple. eXtra Info for Testing)

Derivation process, see next
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-67 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-68 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Derivation Process
One-to-one correspondence between selected abstract & executable test cases Correspondence in all respects e.g. behavior, test purpose, verdicts, relation to ICS/IXIT entries, selectability, independency, etc. May include some MOT y specific selection Independent from and compatible with the selection & parameterization processes ATS Selection SATS Parameterization PATS Derivation Derivation Derivation ETS Selection SETS Parameterization PETS

MOT Functions
Realization of configured ATM (abstract test method)
Realize the LTs (lower testers) and LTCF (LT control function)
LT’s LT’ encoder/decoder realization: simple b t not efficient d /d d li ti i l but t ffi i t LT’s reference protocol implementation: efficient but difficult

Specify or realize UTs (upper testers)
UT can be dedicated software, human operator, existing protocol layers, etc.

Specify or realize TCP (test coordination procedures)

ATS to ETS
Contain derived executable test cases Provide means for selection and parameterization Provide means for executing PETS

Derivation & execution
7-69

Execution

Conformance log generation
Must be human readable and permit verification of test verdicts assigned

Dept. Electrical & Information Engineering, Computer Engineering Lab

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-70

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Outline
General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process

Conformance Assessment Process (1/2)
Conformance assessment process is the entire activities that are necessary to validate and document the protocol conformance
Test T t campaign i th actual execution of th t t suite i is the t l ti f the test it

3 phases for conformance assessment process
1. Preparation for testing
Production of SCS, ICS & IXIT Choice of ATM & ATS Preparation of SUT & MOT

2. 2 Test operations
Static conformance review of ICS Test realization (selection & parameterization) Test campaigns

3. Test report production
Provides clients with documentation
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-71 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-72 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Conformance Assessment Process (2/2)
OSI specifications, proformas, test specifications ifi i Preparation for testing •administrative activities •completion of documents l i fd •choice of test method and ATS •choice of test specifications •SUT and MOT preparation Test Operation •static conformance review •test selection & parameterization •test campaigns t t i

Test Laboratories and Test Clients
Test Laboratory is responsible for conducting the conformance assessment of IUTs and can be classified according to their role
First-party i operated b th supplier of IUT, I i l Fi t t is t d by the li f IUT I.e. implementer, vendor t d Second-party is affiliated to the user of IUT who is willing to test it Third-party is independent from either of the two whose business is the testing of products

SCS, IXIT, ICS, PTS, ATS

static conformance review results, test results, test log

Clients will frequently be implementers or vendors of protocol implementation who want to have their own products tested and certified, but can also be important user of IUT

PCTR SCTR
Dept. Electrical & Information Engineering, Computer Engineering Lab

Test Report Production

7-73

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-74

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Preparations for Testing (1/2)
Production of SCS, ICS
By client filling proformas

Preparations for Testing (2/2)

Production of IXIT
By lab & client filling proformas

ICS
IXIT
SUT decs Extra decs

Choice of ATM & ATS
Agreement between client & lab

SUT config.

MOT config.

Preparation of SUT
By client

Preparation of MOT
By lab

ATS

ICS pro.

IXIT pro.
7-76 Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-75

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Testing Operations (1/3)
Static conformance review
Checks ICS for consistency with static conformance requirements

Testing Operations (2/3)
Test campaign
Assignment of verdicts = [pass, fail, inconclusive]
Test l b has to T t lab h t assign exactly one t t verdict per executed test case i tl test di t t dt t Abnormal test termination or error leads to “not run” in report “Inconclusive” verdict have to be repeated at least once

Test selection
Select a suitable subset of test cases from ATS resulting SATS

Test parameterization
Parameters are assigned values as specified in ICS & IXITs resulting PATS Transform PATS into PETS (parameterized executable test suite)

Conformance log
Outcome of all executed test cases to document for client

Negotiated exit
Time when client and test lab jointly decide to stop the test campaign No formal test report will be produced

Verification of TCPs
To verity MOT & SUT are able to use TCP (Test Coordination Procedure)

Test campaign, see next slide
Execution of PETS and production of conformance log

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-77

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-78

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Testing Operations (3/3) ETS ICS
IXIT
Derivation SUT decs Extra decs Parameterization

Test Report Production (1/2) ETS ICS
SUT config. MOT config. IXIT
Derivation SUT decs Extra decs Parameterization

Selection

Selection SUT config. MOT config.

SETS

SETS

ATS

ICS pro.

ATS
IXIT pro.

ICS pro.

PETS

Execution
7-79 Dr. Junzhao Sun

IXIT pro.

PETS

Execution
7-80

Comp. Report
Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

Dept. Electrical & Information Engineering, Computer Engineering Lab

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Test Report Production (2/2)
SCRT = System Conformance Test Report
Gives an overall summary of the conformance status to the SUT Contains C t i a summary of achieved t t results & administrative i f f hi d test lt d i i t ti information ti

SCTR Proforma
1. Identification summary
Administrative information for identification of SCTR Information about test laboratory, test client, SUT, profile, etc. I f ti b tt tl b t t t li t SUT fil t

PCTR = Protocol Conformance Test Report
One for each protocol under test Administrative and technical information A summary of all achieved test results Individual detailed presentation of each test case

2. System report summary
One section for each tested protocol is required Document reference
IUT definition, protocol specification, ICSs, IXITs, PCTR, ATS, ATM, etc.

Conformance status
Whether static and/or dynamic conformance error

Test cases run
Number of passed, failed, and inconclusive test cases

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-81

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-82

Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

PCTR Proforma
1. Identification summary
Information about PCTR, IUT, and testing environment

Test Conducting
Formal tests
Done by test laboratories that will issue official test certificate

2. 2 Protocol conformance summary
IUT conformance status
“This IUT has/has not been shown by conformance assessment to be nonconforming to the referenced base specification(s)”

Informal tests
In-house or private lab product test Tests at trade shows, bake-off meets Remote test – access to tester through Internet (e.g. on Web server)

Static conformance summary
“The ICSs for this IUT is/is not consistent with the static conformance requirements in the referenced base specification”

Quality of Tests
Consistency and repeatability Standard ATSs & approved ETSs Standard of test laboratory operations

Dynamic conformance summary y y
The test campaign did/did not reveal errors in the IUT

3. Protocol conformance report
Static conformance review report Test campaign report, for each test case in ATS
ATS reference, selected or not, run or not, verdict, and observations
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-83 Dr. Junzhao Sun

Test Report for procurement and regulatory bodies National and International recognition of test results
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-84 Dr. Junzhao Sun

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Detailed Testing Process

Whole Information Flow ETS ICS
IXIT
Derivation SUT decs Extra decs Parameterization

Selection SUT config. MOT config.

SETS

ATS

ICS pro.

IXIT pro.

PETS

Execution
7-86

Comp. Report
Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

7-85

Dr. Junzhao Sun

Dept. Electrical & Information Engineering, Computer Engineering Lab

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

521265A Telecommunication Software

Ch7 – Conformance Testing Methodology

Summary
Conformance testing
Black-box approach of testing protocol implementations against protocol specification

References of This Chapter
Bernd Baumgarten and Alfred Giessler, OSI Conformance Testing Methodology and TTCN, Elsevier Science, 1994

Three phases for conformance assessment process
1. Preparation for testing
Test generation: from ICS+IXIT & TPs, get ATS
• Document proforma

Configuration of ATM architecture
• 4 methods

2. Test operations
Test realization: from ATS, through selection & parameterization, get PETS Test execution

3. Test report production
SCTR & PCTRs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-87 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-88 Dr. Junzhao Sun