Get documents about "VII. Conformance Testing Methodology
Document Sample
521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Outline General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process VII. Conformance Testing Methodology Dept. Electrical & Information Engineering, Computer Engineering Lab 7-1 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-2 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Outline General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process What is Conformance Testing? The process of testing the extent to which implementations of protocol entities adhere to the requirements stated in the relevant standard or specification Concerned with external behavior (black box) Prerequisites for interoperability The primary objective of conformance testing is to increase the probability that different product implementations actually interoperate. Testing of implementations y y But NOT with performance, reliability, fault tolerance, efficiency, etc. ISO 9646: A conforming implementation is one which satisfies both static and dynamic conformance requirements, consistent with the capabilities stated in the Protocol Implementation Conformance Statement (PICS) Dept. Electrical & Information Engineering, Computer Engineering Lab 7-3 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-4 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Standard ISO/IEC 9646 (ITU-T X.290) Conformance Testing Methodology and Framework (CTMF) for the Implementations of OSI and ITU Protocols Conformance Testing Limitations Aspects of protocol testing not considered in ISO 9646 Interoperability testing, performance testing and robustness testing The Th realization of service primitives li ti f i i iti Internal aspects of the system implementation The system environment and testing tools Procurement procedures Test certificates, and accreditation of test labs 7 part standard ISO 9646-1. General Concepts ISO 9646-2. Abstract Test Suite Specification ISO 9646-3. Tree and Tabular Combined Notation (TTCN) ISO 9646-4. Test Realization ISO 9646-5. Conformance Assessment Process 9646 5 ISO 9646-6. Protocol Profile Test Specification ISO 9646-7. Implementation Conformance Statement Conformance testing will always be limited, for technical and theoretical as well as for economic reasons No guarantee can be given that an implementation absolutely conforms to the relevant protocol standards Conformance testing with positive test results will increase the degree of confidence in a protocol implementation Dept. Electrical & Information Engineering, Computer Engineering Lab 7-6 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-5 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Basic Standard vs. Profile Base standard is a standard relative to a single protocol in a given OSI layer A profile is a set of one or more base standards necessary for accomplishing a particular function a multilayer stack of several adjacent base standards Function might be a type of application, or a transport service based on specific telecommunications techniques A protocol can be a base standard by itself, or can belong to a profile The purpose of defining a profile is to facilitate interoperability Conformance Testing Process (1/2) Test generation To generate the abstract test suites (ATSs) manually or automatically Development of proformas for implementation conformance statements (ICS) and implementation extra information for testing (IXIT) Test realization The realization of test tools and means of testing, capable of executing particular test suites To generate the executable test suites (ETSs) with ATSs Including both selection and parameterization process Profile is twofold P fil i t f ld Multiplicity: a profile may retain options in a set of base specifications Refinement: a profile restricts the choices of options in the base specifications. It may also specify additional conformance requirements. Test execution Perform test to an implementation with executable test case selected Conformance assessment process Performed at a test laboratory for a client, with a protocol conformance test report (PCTR) as the ultimate result Dept. Electrical & Information Engineering, Computer Engineering Lab 7-8 Dr. Junzhao Sun We will mainly concentrate on the conformance testing of protocol Dept. Electrical & Information Engineering, Computer Engineering Lab 7-7 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Conformance Testing Process (2/2) Standard Test generation protocol specification Test Types Test types in testing process Basic interconnection tests (BIT) To determine whether interconnection is possible at all as an initial stage To establish main features of the protocol are implemented Basic interconnection tests are standardized but may be omitted Abstract test suite TTCN Test realization Protocol implementation Executable test suite Test execution Capability tests To validate that all the the observable external static capabilities of the IUT are valid with respect to the static conformance requirements expressed in the PICS Capabilities tests are standardized and mandatory System Under Test IUT Behavior tests To check dynamic conformance of an IUT with respect to the dynamic conformance requirements specified in the protocol standard Behaviour tests are standardized and mandatory Tester Verdict: (result) • accept • failure • inconclusive • error (in test HW) 7-9 Dr. Junzhao Sun Conformance resolution tests Special tests for purposes beyond ordinary behavior testing e.g. exception Conformance resolution tests are not standardized and not mandatory Dept. Electrical & Information Engineering, Computer Engineering Lab 7-10 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Test Types Test Results Test results Pass Behavior tests Capability tests Basic interactio i i n tests Observed test outcome shows that the test purpose has been fulfilled and the IUT has displayed only valid behaviors as specified in protocol standards Fail Test outcome proves that some conformance requirement has been violated Inconclusive Neither pass nor fail can be assigned to the test outcome Happens when underlying provider signals problems e.g. disconnect or reset Documentation of test results Conformance resolution tests SCTR = system conformance test report PCTR = protocol conformance test reports Requirements on test results Repeatable, comparable and auditable to a reasonable degree Dept. Electrical & Information Engineering, Computer Engineering Lab 7-11 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-12 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Outline General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process Test Generation Standard St nd rd Standardized Test Purposes (in English) TPs Standardized Abstract Test Suite (in TTCN) ATS Standardized (Protocol) Implementation Conformance Statement (tables) ICS Standardized (Protocol) Implementation eXtra Information for Testing (tables) IXIT Dept. Electrical & Information Engineering, Computer Engineering Lab 7-13 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-14 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Test Specification to Be Produced Conformance Requirements Systems are called conforming if their externally visible behavior fulfils conformance requirements stated in protocol specification Mandatory requirements, always h M d t i t l have t b observed to be b d Conditional requirements, have to be observed under conditions defined Dependence on previous choices of parameter values and options Optional requirements, can be freely selected Static conformance requirements Specify permitted combinations of capabilities implemented in real system E.g. E g value ranges for size parameters or timers as in ICS timers, Dynamic conformance requirements Specify the observable behavior of implementation permitted by standard Define actual protocol e.g. PDU exchanges, timing, format of PDUs. Profile is based on base specifications with additional constraints Dept. Electrical & Information Engineering, Computer Engineering Lab 7-15 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-16 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology ICS (1/2) ICS = Implementation Conformance Statement A checklist of the capabilities supported by the IUT A questionnaires completed b client t present conformance requirements ti i l t d by li t to t f i t Produced by filling out an ICS proforma with static requirements Provides an overview of the features and options that are implemented Used to select and parameterise test cases and as an indicator for basic interoperability between different products ICS (2/2) Standardization organizations Base Base Base specification Base specification specification specification PICSPICSPICSproforma PICSproforma proforma proforma RL = Requirements List Client Profile ICS PICS PICS PICS PICS PICS = Protocol Implementation Conformance Statement IUT realizes a single protocol Profile ICS IUT contains a set of interrelated protocols, e.g. a stack of protocols ICS is not just for testing Used by specification writer to clarify the intent of standards Used by product implementer for selection of features Dept. Electrical & Information Engineering, Computer Engineering Lab 7-17 Dr. Junzhao Sun Profile specification Profile RL Profile specific ICS proforma Profile RL Profile specific ICS Dept. Electrical & Information Engineering, Computer Engineering Lab 7-18 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology ICS Proforma – Protocol Capability Status of capability in protocol specification Mandatory, optional , conditional prohibited, excluded hibit d l d d out-of scope, not applicable ICS Proforma – Structure ICS proforma corrigenda (version information) ICS proforma instructions, abbreviations and terms Identification of the implementation, supplier and client Identification of the protocol, information object or profile Protocol, information object or profile corrigenda Major capabilities table Supported Supported PDUs Table Supported PDUs Table PDUs Table Supported parameters Supported parameters table in PDU Supported parameters table in PDU table in PDU PICS proforma tables Capabilities can be grouped into categories Major capabilities Supported PDUs and fields within PDUs Supported timers Encoding variations E di i ti Negotiation mechanisms Error procedures Supported timer tables Supported protocol error handling table Supported negotiation capabilities table Multi-specification dependencies table Dept. Electrical & Information Engineering, Computer Engineering Lab 7-19 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-20 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology ICS Proforma – ISDN Example Index Protocol capability Conditions Status Reference Support Does the implementation support… for status M1 M2 M3 M4 M41 3.1kHz audio calls Data calls Packet mode calls Interworking with non-ISDN networks Calling line identification presentation M1 not M1 M1 not M1 O.1 O.1 O M M M N/A M N/A 7-21 IXIT, SCS IXIT = Implementation eXtra Information for Testing Contains additional information beyond specification as ICS An A IXIT proforma fill d out b t t l b t f filled t by test laboratory and client d li t With both protocol IXIT and profile IXIT, as in ICS types Specify additional testing information related to an specification Physical setup, connection of test, and testing environment SUT hardware, socket, telephone numbers, and tester SAP addresses Test-specific timer or parameter values 6.1.2 6.1.1 6.3 8.4 8.4 8.4 8.4 [ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No [ ] Yes [ ] No SCS = System Conformance Statement List Li t references t specifications of th IUT related ICS related t t f to ifi ti f the IUT, l t d ICSs, l t d test reports, together with configuration information of SUT, I.e. an index M421 Calling line identification restriction M422 Advice of charge Conformance of an IUT: An IUT is said to be conforming if it satisfied the static and dynamic requirements of the relevant standards referenced in SCS and capabilities in ICSs. Dept. Electrical & Information Engineering, Computer Engineering Lab 7-22 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology IXIT Proforma – ISDN Example #PC_BASIC #PC_POINT_TO_POINT #PC POINT TO POINT #PC_POINT_TO_MULTIPOINT #PC_EN_BLOC_RECEIVING #PC_OVERLAP_RECEIVING #PX_CDPN_OCTET3 #PX_CPN_PTC1 #PX_LCPN_PTC2 #PX_CPN_PTC2 TRUE FALSE TRUE TRUE TRUE ’81’O ‘323830313531’O ‘07’O ‘323830313532’O TSS & TP TSS = Test Suite Structure TP = Test Purposes TSS & TP are derived from the relevant base standards. Provide an informal, easy-to-read description of each test case Concentrate on the meaning of the test rather than detailing how it may be achieved TP is a textual description of a single requirement or a set of related requirements which should be tested Test Purposes are grouped into a logical Test Suite Structure according to suitable criteria, e.g., basic interconnection, error handling, functionality etc. Will be introduced later in ATS section Dept. Electrical & Information Engineering, Computer Engineering Lab 7-23 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-24 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology ATS & ETS ATS = Abstract Test Suite The entire collection of Test Cases (ABSTRACT) Each Test Case specifies th d t il d coding of th T t P E hT tC ifi the detailed di f the Test Purposes, written itt in the standardized test specification language TTCN Can be generated manually from protocol and service standard Can be generated automatically from formal specification of protocol CATG = Computer-Aided Test Case Generation Some tools are commercially available to generate test suite from e.g. SDL Test Suites & Specification Tests apply to a single system under test (SUT), and one protocol standards Exhaustive test of standards specifications Must cover all features – mandatory & optional Must cover valid, invalid, & inopportune protocol exchanges Use of standardized test specification may Improve transparency and increase of the objectiveness for the test process Provide comparability of test results As the criteria for accreditation bodies to assess test laboratories Common test specifications are the basis for harmonized test services Provide a common starting point to suppliers, users, test laboratories and certification bodies Will be studied later in more detail ETS = Executable Test Suite Test cases for truly test execution Can be quickly and easily implemented from the ATS using the TTCN compilers available on most modern test tool platforms Dept. Electrical & Information Engineering, Computer Engineering Lab 7-25 Dr. Junzhao Sun Test specification language – TTCN Dept. Electrical & Information Engineering, Computer Engineering Lab 7-26 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Test Specification Development in ETSI Typical Test Suites GSM, GPRS, 3G (UMTS in 3GPP) Broadband Radio Access Networks (HiperLAN) DECT(Digital Enhanced Cordless Telecommunications) Bluetooth Local Loop, Digital Subscriber Line (DSL) IN, INAP (Intelligent Network Application Protocol) ISDN, B-ISDN SMS in PSTN/ISDN i SIP (Session Initiation Protocol), IPv6 IP Cablecom XML (Extensible Markup Language) Dept. Electrical & Information Engineering, Computer Engineering Lab 7-27 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-28 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Outline General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process ATM ATM = Abstract Test Methods NOT Automatic Teller Machine NOT Asynchronous Transfer Mode A set of ATMs are defined by OSI testing methodology, describing Abstract test configuration Possible types of observation and control Providing for test coordination ATMs are determined by Configuration of IUT with respect to SUT Entire testing configuration, I.e. test system + SUT + service provider Where tester can control and observe the IUT Difference degrees of precision of the test coordination procedures Common test architecture and 4 test methods Dept. Electrical & Information Engineering, Computer Engineering Lab 7-29 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-30 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology OSI System & IUT Complete system Partial system Application relay system Network relay system OSI Terminology Layer S Service provided SP Layer T PDU Service used Layer T Transport layer Layer S Session layer SP SAP Application Presentation Session Transport Network Data link Physical Application Presentation Session Transport Network Data link Physical Application Presentation Session Transport Network Data link Physical Application Presentation Session Transport Network Data link Layer N Layer N Physical Network layer IUT implement one or a combination of OSI protocols from one layer or several adjacent layers Dept. Electrical & Information Engineering, Computer Engineering Lab 7-31 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab SP : Service Primitive SAP : Service Access Point PDU : Protocol Data Unit 7-32 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Conformance Testing Terminology Upper Tester ASP S ASP PCO Conceptual Test Architecture (1/3) Test operator interface Product operator interface Test System (TS) ASP: Abstract Service Primitive PCO: Point of Control and Observation Lower Tester Control Function TCP System Under Test (SUT) Test Co-ordination Procedures (TCP) Upper Tester Upper Tester Upper Tester ASPs PCO PDU IUT PCO ASP ASP IUT: Implementation Under Test PDU: Protocol Data Unit Lower Tester Lower Tester Lower Tester ASPs PDU IUT Lower Tester A PCO maps to a SAP (Service Access Point) in the OSI reference model PCO The PCO has two FIFO queues: • Send (from tester to IUT) • Receive (by tester from IUT ) Lower level service provider There can be several LTs and UTs being simultaneously used Dept. Electrical & Information Engineering, Computer Engineering Lab 7-33 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-34 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Conceptual Test Architecture (2/3) Testing contexts Single-party testing is for testing an IUT which communicates with exactly one real open system represented by a single lower tester system, Multi-party testing is for testing an IUT communicating with multiple real open systems, represented by more than one lower tester Configuration for IUT components can be homogeneous or heterogeneous Conceptual Test Architecture (3/3) Upper tester (UT) controls and observes IUT’s upper service boundary, by operator access, API, or hardware interface In i l I single-party context, UT behaves as a user of IUT t t t b h f In multi-party context, UTs working in parallel act as users of IUT Lower tester (LT) controls and observes the IUT’s lower service boundary, indirectly, via the underlying service provider In single-party testing, behaves as the p g p y g, peer entity to IUT y In multi-party testing, act as peer entities working in parallel Test coordination procedures (TCPs) are used to ensure cooperation between the UTs and LTs How tester shall respond Passing (preliminary) results Synchronisation S h i ti TCP is NOT Transport Control Protocol, as in TCP/IP Lower tester control function (LTCF) coordinating all LTs Assign the test case verdicts Mandatory in multi-party context, inapplicable in single-party context Dept. Electrical & Information Engineering, Computer Engineering Lab 7-35 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-36 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology ATM Classification ATMs for multi-party testing Several parallel upper and lower testers In I complex situation a upper tester control function (UTCF) i needed l it ti t t t lf ti is d d Special cases include only one upper tester, or even no upper tester at all Test Case Lower Tester IUT Upper Tester ATMs for single-party testing Local Test Method (L) Upper Tester and Lower Tester in Test System ConnectRequest “CR” ConnectConfirm “CC” ConnectIndication “CONind” ConnectResponse “CONresp” Distributed Test Method (D) Upper Tester in SUT, Lower Tester in Test System SUT Co-ordinated Test Method (C) As above, uses Test Management Protocol Recall service primitives Request Indication Response Confirm 7-37 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-38 Dr. Junzhao Sun Remote Test Method (R) Lower Tester in Test System, no Upper Tester Dept. Electrical & Information Engineering, Computer Engineering Lab 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Local Test Method UT ASPs TCP coordination N-layer entity under test ASPs LT • • • • • • Upper Tester is located in Test System Requires an upper interface on IUT IUT is “built in” the tester No ATSs for this method Good for the testing of a hardware component Example: Ethernet driver 7-39 Dr. Junzhao Sun Local Test Method System Under Test (SUT) 2) CONind 3) CONresp Test System (TS) Upper Tester TCPs ASPs PCO Lower Tester IUT ASPs PCO 1) CR 4) CC Dept. Electrical & Information Engineering, Computer Engineering Lab Dept. Electrical & Information Engineering, Computer Engineering Lab 7-40 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Distributed Test Method TCP UT LT PDUs IUT ASPs Service provider • • • • UT in SUT, LT remote Requires synchronisation Suitable for upper layer protocols / protocols offering API Example: socket communication 7-41 Dr. Junzhao Sun Distributed Test Method System Under Test (SUT) Test System (TS) Test Co-ordination Procedures (TCP) Lower Tester ASPs ASPs Upper Tester 2) CONind 3) CONresp PCO IUT PCO 1) CR 4) CC Dept. Electrical & Information Engineering, Computer Engineering Lab Dept. Electrical & Information Engineering, Computer Engineering Lab 7-42 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Co-ordinated Test Method TM-PDUs UT LT PDUs IUT ASPs Service provider • • • • • UT in SUT but no access, LT remote No assumption of upper interface to the IUT Use only one PCO below the LT Uses Test Management Protocol (TMP) embedded in ASPs Suitable for mid-layer protocols 7-43 Dr. Junzhao Sun Co-ordinated Test Method System Under Test (SUT) 2) CONind 3) CONresp Test System (TS) Test Management Protocol Upper Tester IUT Lower Tester ASPs PCO 1) CR 4) CC Dept. Electrical & Information Engineering, Computer Engineering Lab Dept. Electrical & Information Engineering, Computer Engineering Lab 7-44 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Remote Test Method Remote Test Method TCP UT LT PDUs IUT ASPs Service provider System Under Test (SUT) Test System (TS) IUT Lower Tester ASPs PCO • No Upper Tester • Upper Tester can be native application or a user accessible interface • Manual co-ordination • Limited, but easy to use Dept. Electrical & Information Engineering, Computer Engineering Lab 7-45 Dr. Junzhao Sun 1) CR 2) CC Dept. Electrical & Information Engineering, Computer Engineering Lab 7-46 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology ATMs Put Together Testing Configuration Examples – MHS MHS = Message Handling Systems Multi-user in single-party context RTS = Reliable T R li bl Transfer S t f System TCP UT-2 UT-1 LT MHSPDUs IUT MHS-ASPs RTSASPs RT service provider Dept. Electrical & Information Engineering, Computer Engineering Lab 7-47 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-48 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Testing Configuration Examples – Multiplexing Multiplexing happens in transport or network layer protocols where more than one connection is multiplexed over a single lower connection, e g transport connections over network one connection e.g. TCP UT-2 UT-1 LT NWNW PDUs Service provider Dept. Electrical & Information Engineering, Computer Engineering Lab 7-49 Dr. Junzhao Sun Testing Configuration Examples – Relay Relay systems’ testing should be in multi-party context, without using UTs TCP TransportASPs LT –1 PDUs IUT ASPs Service provider Dept. Electrical & Information Engineering, Computer Engineering Lab LT –2 PDUs ASPs Service provider 7-50 Dr. Junzhao Sun NetworkASPs IUT 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology A Example Test System Architecture Protocol Tester Interface TC Editor TC Translator TC Library PICS，PIXIT Editor Test Select Test Execute Transmit Receive Service Provider Dept. Electrical & Information Engineering, Computer Engineering Lab 7-51 Dr. Junzhao Sun Outline SUT Result Analysis&report IUT General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process Dept. Electrical & Information Engineering, Computer Engineering Lab 7-52 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology ATS Development Process ATS = Abstract Test Suites 1. Determine the testable conformance requirements from the relevant specifications, specifications ICS and related documents 2. Determine the test groups & objectives that cover these requirements 3. Develop test purposes covering requirements and test group objectives 4. Choose testing context (single-/multi-party) and an ATM (configuration) A test suite is always dedicated to a single test method Recall the Test Generation Process Standard St nd rd Standardized Test Purposes (in English) TPs Standardized Abstract Test Suite (in TTCN) ATS 5. Specify ATS, including test steps, using standardized notation like TTCN p y parameterisation criteria with respect to the ICS p 6. Specify the selection and p and IXIT proformas Standardized (Protocol) Implementation Conformance Statement (tables) ATS should be used to derive ETS by compilation and adaptation to the means of testing (MOT) MOT is the combination of test devices, equipments and procedures Dept. Electrical & Information Engineering, Computer Engineering Lab 7-53 Dr. Junzhao Sun ICS Standardized (Protocol) Implementation eXtra Information for Testing (tables) IXIT Dept. Electrical & Information Engineering, Computer Engineering Lab 7-54 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology ATS Structure (1/2) Test suite Test T t group Test group Test group Test case Test case Test case Test case Test step Test step Test case Test case Test case Test event Test case Test case Test event Test case Test case Test case Test t T t step Test step Test case Test case Test case Test event 7-55 ATS Structure (2/2) A test suite consists of test cases with no execution order Test cases are logically grouped into test group according to test purposes Each t t E h test case has an individual t t purpose t d h i di id l test to demonstrate a certain behavior t t t i b h i Test group Test case Test case Test case Test case Test case Test case Test case Test case A test group objective may be provided for each test group The nested hierarchy should be created in a top-down fashion Ensure systematic derivation of test purposes Achieve good coverage of all protocol aspects and conformance requirements A test case may be modularized into nesting test steps Modularization facilitates test case development and maintenance Group commonly used test steps together into test step libraries Economical that they need not be specified repeatedly Test event is the elementary unit within a test case Test case Test case Test case Test event Dr. Junzhao Sun Usually the exchange of a PDU or an ASP at a PCO Dept. Electrical & Information Engineering, Computer Engineering Lab 7-56 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology TSS = Test Suite Structure “which tests are performed on which aspects of the reference specification” Adopt a top down approach deriving TSS from base specification when no technical constraint forbids it, structure TSS as a tree 1. The name of base specification should be placed at the top of the TSS 2. Test groups related to the "major functions", or "roles" of the base specification 3. Pre-defined groups of tests according to the "nature" of the tests: Basic interconnection tests; capability tests; Valid behaviour t t i V lid b h i tests; invalid b h i lid behaviour t t i tests; inopportune b h i t behaviour t t tests; TSS FTAM Example FTAM = File Transfer Access and Management 4. Pre-defined groups of tests according to the "functional aspect" tested: State event transitions; parameter variations; parameter combinations; timers 5. Groupings relevant to the base specification E.g. name of functional units, then of PDUs Dept. Electrical & Information Engineering, Computer Engineering Lab 7-57 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-58 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Test Purposes (1/2) Each test case with one test purpose for meaning description Test purposes focus on one or more related conformance requirements, derived from OSI specification Test purposes and their hierarchical grouping are specified in a TSS & TP (Test suite structure and test purposes) standard Structure chosen depends largely on personal experience and style Test Purposes (2/2) The expression of a TP shall not be limited to checking a state or a state transition of a protocol state machine Example TP "Check th t th IUT sends a COMMON CONTROL ACK after E l TP: "Ch k that the d ft receipt of a COMMON CONTROL message containing ... , as a result of state transition S1 → S2.". Test purposes for a test suite have to cover aspects of protocol Capabilities, in particular mandatory and optional ones Behavior tests, with the tester displaying valid/invalid behavior PDUs sent to/received from IUT, and relationship between the PDUs Protocol phases and protocol machine states Timing, encoding, individual parameters and parameter combinations Dept. Electrical & Information Engineering, Computer Engineering Lab 7-59 Dr. Junzhao Sun A clear explanation of the relation between TPs and the conformance requirements in the reference specification shall be provided The TPs shall be designated by the same numbering and referencing scheme as the TSS Dept. Electrical & Information Engineering, Computer Engineering Lab 7-60 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Combined TPs (CTPs) The final cost of a test campaign directly depends on the size of the ATS The objective of CTP is to reduce the number of test cases, but to keep a good coverage TP Combination Principle Dept. Electrical & Information Engineering, Computer Engineering Lab 7-61 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-62 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology TSS for CTP Test Purpose Component Initial condition should define precisely the expected state of the IUT at the beginning of the test Example: Ensure that the IUT is in state connection established (reception of a connect confirm on the LT on response to a connect request). Check(s) to be performed Example: Ensure that the IUT, on reception of a valid SETUP message with the Sending complete information element sends any of a CALL PROCEEDING, ALERTING or CONNECT message. Verdict criteria should explicitly describe the observable events which will be used to assess the result of the test. Example: Check that the IUT in state S1, on receipt of a valid connect request, enters state S2 (sends a valid connect confirm). Dept. Electrical & Information Engineering, Computer Engineering Lab 7-63 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-64 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Testing States Testing state is a combination of the states of SUT (IUT), test system, and underlying provider which can occur during testing Stable t t St bl state can b maintained b t be i t i d between t t case without t t activities test ith t tester ti iti Idle state is that no connection is established, independent of previous tests Test Case Scheme A test case is conceptually composed of 3 parts Test body, the combination of test events for achieving test purpose Starts i St t in a initial t ti state i iti l testing t t Test state is the one that test case is truly performed to check test purpose Initial state the start point of the execution of a test case Test preamble, takes the test from stable states to initial state Test postamble, takes the test from end states to one stable state If end state is not unique, it has to be checked by a verification step End state is the stop state of a test One test case may stop at several end states Postamble Stable State Preamble Test Test Body State End State (Test Body) Verification End State (Verification) The outcome for each test case should be assigned a verdict Calculated from verdicts assigned to test events in the outcome Dept. Electrical & Information Engineering, Computer Engineering Lab 7-65 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-66 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Outline General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process Test Realization Test realization is the process of producing a means of testing (MOT) for conformance to protocol standards, by reference to an ATS standard. standard MOT is a real test system used to execute a test suite on IUTs Combination of equipment and procedures that derive, select, parameterize and execute the test cases of a given ATS, together with produce conformance log Does not mean the actual execution of test cases ETS derivation is to convert the ATS into final ETS Selection of applicable test cases (abstract or executable) Performed mainly on the basis of ICS (Imple. Conformance Statement) Parameterization of selected test cases (abstract or executable) Performed mainly on the basis of IXIT (Imple. eXtra Info for Testing) Derivation process, see next Dept. Electrical & Information Engineering, Computer Engineering Lab 7-67 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-68 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Derivation Process One-to-one correspondence between selected abstract & executable test cases Correspondence in all respects e.g. behavior, test purpose, verdicts, relation to ICS/IXIT entries, selectability, independency, etc. May include some MOT y specific selection Independent from and compatible with the selection & parameterization processes ATS Selection SATS Parameterization PATS Derivation Derivation Derivation ETS Selection SETS Parameterization PETS MOT Functions Realization of configured ATM (abstract test method) Realize the LTs (lower testers) and LTCF (LT control function) LT’s LT’ encoder/decoder realization: simple b t not efficient d /d d li ti i l but t ffi i t LT’s reference protocol implementation: efficient but difficult Specify or realize UTs (upper testers) UT can be dedicated software, human operator, existing protocol layers, etc. Specify or realize TCP (test coordination procedures) ATS to ETS Contain derived executable test cases Provide means for selection and parameterization Provide means for executing PETS Derivation & execution 7-69 Execution Conformance log generation Must be human readable and permit verification of test verdicts assigned Dept. Electrical & Information Engineering, Computer Engineering Lab Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-70 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Outline General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process Conformance Assessment Process (1/2) Conformance assessment process is the entire activities that are necessary to validate and document the protocol conformance Test T t campaign i th actual execution of th t t suite i is the t l ti f the test it 3 phases for conformance assessment process 1. Preparation for testing Production of SCS, ICS & IXIT Choice of ATM & ATS Preparation of SUT & MOT 2. 2 Test operations Static conformance review of ICS Test realization (selection & parameterization) Test campaigns 3. Test report production Provides clients with documentation Dept. Electrical & Information Engineering, Computer Engineering Lab 7-71 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-72 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Conformance Assessment Process (2/2) OSI specifications, proformas, test specifications ifi i Preparation for testing •administrative activities •completion of documents l i fd •choice of test method and ATS •choice of test specifications •SUT and MOT preparation Test Operation •static conformance review •test selection & parameterization •test campaigns t t i Test Laboratories and Test Clients Test Laboratory is responsible for conducting the conformance assessment of IUTs and can be classified according to their role First-party i operated b th supplier of IUT, I i l Fi t t is t d by the li f IUT I.e. implementer, vendor t d Second-party is affiliated to the user of IUT who is willing to test it Third-party is independent from either of the two whose business is the testing of products SCS, IXIT, ICS, PTS, ATS static conformance review results, test results, test log Clients will frequently be implementers or vendors of protocol implementation who want to have their own products tested and certified, but can also be important user of IUT PCTR SCTR Dept. Electrical & Information Engineering, Computer Engineering Lab Test Report Production 7-73 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-74 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Preparations for Testing (1/2) Production of SCS, ICS By client filling proformas Preparations for Testing (2/2) Production of IXIT By lab & client filling proformas ICS IXIT SUT decs Extra decs Choice of ATM & ATS Agreement between client & lab SUT config. MOT config. Preparation of SUT By client Preparation of MOT By lab ATS ICS pro. IXIT pro. 7-76 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-75 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Testing Operations (1/3) Static conformance review Checks ICS for consistency with static conformance requirements Testing Operations (2/3) Test campaign Assignment of verdicts = [pass, fail, inconclusive] Test l b has to T t lab h t assign exactly one t t verdict per executed test case i tl test di t t dt t Abnormal test termination or error leads to “not run” in report “Inconclusive” verdict have to be repeated at least once Test selection Select a suitable subset of test cases from ATS resulting SATS Test parameterization Parameters are assigned values as specified in ICS & IXITs resulting PATS Transform PATS into PETS (parameterized executable test suite) Conformance log Outcome of all executed test cases to document for client Negotiated exit Time when client and test lab jointly decide to stop the test campaign No formal test report will be produced Verification of TCPs To verity MOT & SUT are able to use TCP (Test Coordination Procedure) Test campaign, see next slide Execution of PETS and production of conformance log Dept. Electrical & Information Engineering, Computer Engineering Lab 7-77 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-78 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Testing Operations (3/3) ETS ICS IXIT Derivation SUT decs Extra decs Parameterization Test Report Production (1/2) ETS ICS SUT config. MOT config. IXIT Derivation SUT decs Extra decs Parameterization Selection Selection SUT config. MOT config. SETS SETS ATS ICS pro. ATS IXIT pro. ICS pro. PETS Execution 7-79 Dr. Junzhao Sun IXIT pro. PETS Execution 7-80 Comp. Report Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab Dept. Electrical & Information Engineering, Computer Engineering Lab 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Test Report Production (2/2) SCRT = System Conformance Test Report Gives an overall summary of the conformance status to the SUT Contains C t i a summary of achieved t t results & administrative i f f hi d test lt d i i t ti information ti SCTR Proforma 1. Identification summary Administrative information for identification of SCTR Information about test laboratory, test client, SUT, profile, etc. I f ti b tt tl b t t t li t SUT fil t PCTR = Protocol Conformance Test Report One for each protocol under test Administrative and technical information A summary of all achieved test results Individual detailed presentation of each test case 2. System report summary One section for each tested protocol is required Document reference IUT definition, protocol specification, ICSs, IXITs, PCTR, ATS, ATM, etc. Conformance status Whether static and/or dynamic conformance error Test cases run Number of passed, failed, and inconclusive test cases Dept. Electrical & Information Engineering, Computer Engineering Lab 7-81 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-82 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology PCTR Proforma 1. Identification summary Information about PCTR, IUT, and testing environment Test Conducting Formal tests Done by test laboratories that will issue official test certificate 2. 2 Protocol conformance summary IUT conformance status “This IUT has/has not been shown by conformance assessment to be nonconforming to the referenced base specification(s)” Informal tests In-house or private lab product test Tests at trade shows, bake-off meets Remote test – access to tester through Internet (e.g. on Web server) Static conformance summary “The ICSs for this IUT is/is not consistent with the static conformance requirements in the referenced base specification” Quality of Tests Consistency and repeatability Standard ATSs & approved ETSs Standard of test laboratory operations Dynamic conformance summary y y The test campaign did/did not reveal errors in the IUT 3. Protocol conformance report Static conformance review report Test campaign report, for each test case in ATS ATS reference, selected or not, run or not, verdict, and observations Dept. Electrical & Information Engineering, Computer Engineering Lab 7-83 Dr. Junzhao Sun Test Report for procurement and regulatory bodies National and International recognition of test results Dept. Electrical & Information Engineering, Computer Engineering Lab 7-84 Dr. Junzhao Sun 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Detailed Testing Process Whole Information Flow ETS ICS IXIT Derivation SUT decs Extra decs Parameterization Selection SUT config. MOT config. SETS ATS ICS pro. IXIT pro. PETS Execution 7-86 Comp. Report Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-85 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 521265A Telecommunication Software Ch7 – Conformance Testing Methodology 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Summary Conformance testing Black-box approach of testing protocol implementations against protocol specification References of This Chapter Bernd Baumgarten and Alfred Giessler, OSI Conformance Testing Methodology and TTCN, Elsevier Science, 1994 Three phases for conformance assessment process 1. Preparation for testing Test generation: from ICS+IXIT & TPs, get ATS • Document proforma Configuration of ATM architecture • 4 methods 2. Test operations Test realization: from ATS, through selection & parameterization, get PETS Test execution 3. Test report production SCTR & PCTRs Dept. Electrical & Information Engineering, Computer Engineering Lab 7-87 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-88 Dr. Junzhao Sun
Related docs
