conformance testing by keralaguest


									Conformance Testing
Martha Gray, Alan Goldfine, Lynne Rosenthal, Lisa Carnahan

National Institute of Standards and Technology

With any standard or specification, eventually the discussion turns to "how
will we know if an implementation or application conforms to our standard
or specification?" The following discussion defines conformance and
conformance testing as well as describes the components of a
conformance testing program.

1. Definitions - what is conformance (to a standard)?

Conformance is usually defined as testing to see if an implementation
faithfully meets the requirements of a standard or specification. There are
many types of testing including testing for performance, robustness,
behavior, functions and interoperability. Although conformance testing may
include some of these kinds of tests, it has one fundamental difference --
the requirements or criteria for conformance must be specified in the
standard or specification. This is usually in a conformance clause or
conformance statement, but sometimes some of the criteria can be found
in the body of the specification. Some standards have subsequent
documentation for the test methodology and assertions to be tested. If the
criteria or requirements for conformance are not specified, there can be no
conformance testing.

The general definition for conformance has changed over time and been
refined for specific standards. In 1991, ISO/IEC DIS 10641 defined
conformance testing as "test to evaluate the adherence or non-adherence
of a candidate implementation to a standard." ISO/IEC TR 13233 defined
conformance and conformity as "fulfillment by a product, process or
service of all relevant specified conformance requirements." In recent years,
the term conformity has gained international use and has generally
replaced the term conformance in ISO documents.
In 1996 ISO/IEC Guide 2 defined the three major terms used in this field.

      conformity - fulfillment of a product, process or service of specified requirements
      conformity assessment - any activity concerned with determining directly or
       indirectly that relevant requirements are fulfilled.
      conformity testing - conformity assessment by means of testing.

ISO/IEC Guide 2 also mentions that "Typical examples of conformity
assessment activities are sampling, testing and inspection; evaluation,
verification and assurance of conformity (supplier's declaration,
certification); registration, accreditation and approval as well as their

Conformance tests should be used by implementers early-on in the
development process, to improve the quality of their implementations and
by industry associations wishing to administer a testing and certification
program. Conformance tests are meant to provide the users of conforming
products some assurance or confidence that the product behaves as
expected, performs functions in a known manner, or has an interface or
format that is known. Conformance testing is NOT a way to judge if one
product is better than another. It is a neutral mechanism to judge a product
against the criteria of a standard or specification.

2. Testing for conformance?

Ideally, we would like to be able to prove beyond any doubt that an
implementation is correct, consistent, and complete with respect to its
specification. However, this is generally impossible for implementations of
nontrivial specifications that are written in a natural language.

The alternative is falsification testing, which subjects an implementation to
various combinations of legal and illegal inputs, and compares the
resulting output to a set of corresponding "expected results." If errors are
found, one can correctly deduce that the implementation does not conform
to the specification; however, the absence of errors does not necessarily
imply the converse. Falsification testing can only demonstrate non-
conformance. Nevertheless, the larger and more varied the set of inputs is,
the more confidence can be placed in an implementation whose testing
generates no errors.

3. Conformance clause and specification wording

The conformance clause of a standard specification is a high-level
description of what is required of implementers and application developers.
It, in turn, refers to other parts of the standard. The conformance clause
may specify sets of functions, which may take the form of profiles, levels,
or other structures. The conformance clause may specify minimal
requirements for certain functions and minimal requirements for
implementation-dependent values. Additionally it may specify the
permissibility of extensions, options, and alternative approaches and how
they are to be handled.

3.1 Profiles and/or levels

Applications often do not require all the features within a standard. It is
also possible that implementations may not be able to implement all the
features. In these cases, it may be desirable to partition the specifications
into subsets of functionality.

A profile is a subset of the overall specifications that includes all of the
functionality necessary to satisfy the requirements of a particular
community of users.

Levels are nested subsets of the specifications. Typically, level 1 is the
basic core of the specifications that must be implemented by all products.
Level 2 includes all of level 1 and also additional functionality. This nesting
continues until level n, which consists of the entire specification.

It is possible for a standard to have both profiles and levels defined for the
entire specification (i.e., an implementer could choose to implement any
one of (number of profiles) x (number of levels) subsets). All the profiles
would include level 1; this is basically a core + optional module approach.
Alternatively, each profile could have its own set of levels.
3.2 Extensions to the specifications

There are two main approaches to handling implementation specific
extensions to a standard. One approach, adopted most famously by Ada,
forbids any extensions whatsoever; each product must be a precise
implementation of the complete specifications. This is called strict

The other approach adopts the weaker overall requirement that, while
extensions are allowed, an implementation must perform all the
functionality in the specifications exactly as specified. This more common
approach usually includes some additional, more specific, requirements in
the conformance clause, along the lines of:

      Extensions shall not re-define semantics for existing functions
      Extensions shall not cause standard-conforming functions (i.e., functions that do
       not use the extensions) to execute incorrectly
      The mechanism for determining application conformance and the extensions shall
       be clearly described in the documentation, and the extensions shall be marked as
      Extensions shall follow the principles and guidelines of the standard they extend,
       that is, the specifications must be extended in a standard manner.

One approach that has been used successfully is to have a register for
extensions. This is a document, parallel to but distinct from the official
specifications, that contains a list of recognized extensions to the standard.
These extensions may eventually migrate into future versions of the

3.3 Options

A standard may classify features as "mandatory" vs. "optional," and provide
a table listing features so classified. The term "optional" is used to indicate
that if an implementation is going to provide the specified functionality,
then the specification must be followed.

3.4 Implementer defined values
Specifications sometimes need to address:

      Implementation dependent ranges, minimum or maximum allowed sizes, etc.
      Values that may be different for different conforming implementations of the
      The color model(s), if any, supported by the standard
      Features reserved for registration.

3.5 Alternate approaches

Specifications may describe several different ways to accomplish its
operation (e.g., a choice of file formats or codes). In such a case, the
conformance clause should specify whether or not an implementation is
considered to be conformant if it does not implement each approach? (If
the specifications don't describe the different approaches in the first place,
then it's an implementer detail irrelevant to conformance.)

For ensure testability of a specification/standard, care should be taken in
the wording of the specifications itself. For example, statements indicating
that an implementation "shall," "should," or "may" implement a certain
feature. The meaning of the words "shall," "should," and "may," in the
context of the given standard, could be defined in the conformance clause
or be as defined by organizations such as ISO.

Some recent standards include, as official parts of the standard, lists of
assertions. These assertions are statements of functionality or behavior
derived from the standard, and which are                        true for conforming
implementations. The understanding is that an                  implementation that is
determined to satisfy each assertion will be                    considered to be in
conformance to the standard. Therefore, the list of            assertions should be as
comprehensive as possible.

4. Conformance Testing Program

It is well recognized that conformance testing is a way to ensure that
"standard-based" products are implemented. The advantages afforded by
testing (and certification) are fairly obvious: quality                   products,
interoperability, and competitive markets with more choices.

Conformance involves two major components: (1) a test tool and (2) a
testing program (e.g., certification or branding). A testing program cannot
exist without a test tool or test suite, but a test tool/suite can exist without
a testing program. Not all specifications or standards need a testing
program. Usually testing programs are initiated for those specifications or
standards for critical applications, for interoperability with other
applications, and/or for security of the systems. The decision to establish
a program is based on the risk of nonconformance versus the costs of
creating and running a program.

A conformance testing program usually includes:

      Standard or specification
      Test tool (e.g., tool, suite, and/or reference implementation
      Procedures for testing
      Organization(s) to do testing, issue certificates of validation, and arbitrate

A specification/standard that includes a conformance clause and a test
tools are essential to defining and measuring conformance.

Generically, a conformance test suite is a collection of combinations of
legal and illegal inputs to the implementation being tested, together with a
corresponding collection of "expected results." If such a list is provided,
the starting point for the development of the test suite is the list of
assertions for the standard. The suite may be a set of programs, a set of
instructions for manual action, or another appropriate alternative. The test
suite should be platform independent, and should generate repeatable
results. Development of the test suite may well be the costliest part of the
conformance program.

A reference implementation is an implementation of a standard that is by
definition conformant to that standard. Such an implementation provides a
proof of concept of the standard and also provides a tool for the
developers of the conformance test suite (by generating expected values,
testing the test suite, etc.) A reference implementation has maximum value
in the early stages of a conformance program.

The conformance testing policies and procedures should be spelled out
before testing begins. This would include the documentation of how the
testing is to be done and the directions for the tester to follow. Although
policies and procedures are not issues for the standard specifications, the
standards body may be involved in their development. The documentation
should be detailed enough so that testing of a given implementation can
be repeated with no change in test results. The procedures should also
contain information on what must be done operationally when failures
occur. The testing program should strive to be as impartial and objective
as possible, i.e., to remove subjectivity of both the procedures and the
testing tools.

Finally, the testing policy and procedure should identify and define the
actions of the organization(s) responsible for conducting the tests,
handling disputes, and issuing certificates (or brands) of validation.

Who does the testing? Some standards have no official testing
organizations. They rely on self-assessment by the implementer (1st party
testing) and acceptance testing by buyers. Others have qualified third party
testing laboratories that apply the test suite according to the established
policies and procedures. A testing laboratory can be an organization or
individual, and can either be accredited from a formal accreditation
organization such as NIST's National Voluntary Laboratory Accreditation
Program (NVLAP) or recognized by the buyer, seller, and certificate issuer,
as qualified to perform the testing.

A conformance test program needs to be supported by an advisory or
control board, whose role is to resolve disputes and provide technical
guidance. This board should be a body of impartial experts. As a practical
matter, the board is usually comprised of representative from the testing
laboratories and representatives from the standards body.
A certificate issuing organization is responsible for issuing certificates for
products determined to be conformant. The decision to issue a certificate
is based on the testing results and established criteria for issuing
certificates. These criteria may or may not require implementations to pass
100% of the specifications. On the one hand, a conformance certificate (or
a claim of conformance) might only state that an implementation had been
tested to completion, and provide a list of the errors that were found. It
would then be up to a purchaser to decide the criteria (how many or what
kinds of errors) it wishes to use to make implementations eligible for
purchase. On the other hand, the policy might be that a certificate is
issued (or a claim of conformance is made) only if no errors are found.
Often a hybrid of these is appropriate -- i.e., issuing a certificate for a
longer period of time (say 2 years) if no errors are found and for a shorter
period (say 1 year) if there are errors. At the end of the 1-year period, the
implementer would have to correct the errors to renew the certificate. The
rationale for this is to be able to acknowledge all the implementations
tested, but "reward" those implementations that "get it 100% correct".
Another issue it whether or not a certificate expires (e.g., good for 2 years
or never expire). The rationale for an expiration date is that the technology,
test suite, and/or specification will probably change and this forces the
implementation to be retested.

While actual testing (1st or 3rd party testing) and certification (branding) can
be carried out by various organizations, it is essential that there be a
centralized sponsor or owner of the conformance testing program. The
sponsor has a fundamental interest in ensuring the success of the program.
Typically, the sponsor establishes and maintains the program. It assumes
responsibility for insuring that the components of the program are in place
and becomes the centralized source for information about the program.

Lisa J. Carnahan
National Institute of Standards and Technology
Information Technology Laboratory
Room 562, Bldg. 820
Gaithersburg, Md. 20899

Tel: (301) 975-3362
FAX: (301) 948-6213

GLOSSARY of Conformance Terminology

Although there is a general understanding of what is meant by
conformance, the nuances and meaning of terms associated with
conformance have been refined for specific standards and communities.
This glossary presents terminology consistent with several ISO/IEC and
IEEE standards.

The 2 major terms are:

      Conformance - fulfillment of an implementation of all requirements specified;
       adherence of an implementation to the requirements of one or more specific
       specifications or standards.
      Conformance Testing - testing to evaluate the adherence or non-adherence of an
       implementation to a specification.

Additional terms are:

      Certification (of conformance) a document attesting that an implementation
      Conformance Clause - clause in a specification that states all the requirements
       that shall be satisfied to claim conformance to that part of the specification.
      Conformity - see Conformance. ISO/IEC Guide 2 uses the term conformity rather
       than conformance.
      Falsification - test method that attempts to find errors in an implementation to
       determine if it correctly implements the requirements in a given specification.
       Falsifications testing can only demonstrate non-conformance. If errors are found,
       the implementation does not conform, the absence of errors does not necessarily
       imply the converse.
      Implementation - realization of a specification; it can be a software product,
       system, service, program, or webpage.
      Software validation (conformity assessment) - the process of testing software for
    compliance with applicable specifications or standards.
   Test laboratory - an organization that carries out conformance testing. This can be
    a third party, a user organization, or a recognized private operating organization or
    an identifiable part of a supplier organization.
   Test method - specified technical procedure to design a test suite for a given
   Test procedures - defines the procedures to be followed when applying a test
    suite to a product for the purposes of conformance testing.
   Test Suite - the combination of test software, test documentation, and test
    procedures that check an implementation for conformance to a standard
   Validation - testing for conformance, by using an official test suite in a prescribed

To top