Get documents about "ModSpecP3 Test Guide v 0 2
Document Sample
Modular Specifications Project Phase 3 – Certificate Discovery
The Office of the National Coordinator for Health Information Technology
Modular Specifications Phase 3: Certificate Discovery Test Guide
Version 0.2
March 28, 2012
Modular Specification Phase 3 Direct
Version 0.2
Reference Documents
Direct Certificate Discovery Specification -
https://docs.google.com/a/nitorgroup.com/document/d/1igDpIizm7CTfV-
fUw_1EnrCUGIljFEgLPRHpgK5iaec/edit
Direct Certificate Discovery RTM – TBD
Document Change History
Version Date Items Changed Since Previous Version Changed By
0.1 March 21, 2012 Initial draft created ONC Testing Team
0.2 March 28, 2012 Updates made based on feedback ONC Testing Team
received from internal reviews
- Updated diagrams so that the DNS
servers are part of either the Testing Tool
or the System under test’s infrastructure
March 28, 2012 Page 1
Modular Specification Phase 3 Direct
Version 0.2
Table of Contents
1 EXECUTIVE SUMMARY .......................................................................................................................4
2 PROJECT BACKGROUND: MODULAR SPECIFICATION PHASE 3 ......................................................................5
2.1 Document Audience & Sections ...................................................................................................... 5
3 TEST ENVIRONMENT .........................................................................................................................6
3.1 Introduction .................................................................................................................................... 6
3.2 Terminology .................................................................................................................................... 6
3.3 Components of the Test Environment ............................................................................................ 7
3.3.1 DNS CERT records .................................................................................................................... 7
3.3.2 LDAP ......................................................................................................................................... 8
4 TEST COVERAGE OVERVIEW .............................................................................................................. 10
4.1 Key Test Scope Decisions .............................................................................................................. 10
4.1.1 Out of scope: Testing all DNS resource records .................................................................... 10
4.1.2 Out of scope: Testing all LDAP attributes .............................................................................. 10
4.1.3 Out of scope: LDAP and DNS Message format ...................................................................... 10
4.1.4 Out of scope: Local search matching rules ............................................................................ 10
4.2 Test Coverage Areas ...................................................................................................................... 10
4.2.1 DNS CERT Records ................................................................................................................. 11
4.2.2 LDAP ....................................................................................................................................... 11
4.3 Testing Roadmap........................................................................................................................... 11
4.3.1 Initial connectivity testing...................................................................................................... 11
4.3.2 Interoperability testing .......................................................................................................... 12
5 TESTING PREP ............................................................................................................................... 13
5.1 Stand Up Testing Tool ................................................................................................................... 13
5.2 Identify relevant Test Cases .......................................................................................................... 13
5.3 Load Test Data ............................................................................................................................... 13
5.4 General Test Case Guidance.......................................................................................................... 13
6 TESTING MATERIALS ....................................................................................................................... 14
6.1 Test Packet Contents ..................................................................................................................... 14
6.2 Navigating the Test Cases Sheet ................................................................................................... 14
7 EXECUTING TESTING ....................................................................................................................... 17
7.1 Testing Steps ................................................................................................................................. 17
7.2 Analysis.......................................................................................................................................... 17
7.2.1 Reporting Test Case Outcomes .............................................................................................. 17
8 APPENDIX A – GLOSSARY OF TERMS ................................................................................................... 18
Table of Figures
Figure 1: DNS CERT Records Test Focus Actor diagram (System is initiator) ................................................ 7
Figure 2: DNS CERT Records Test Focus Actor diagram (System stores certificate) ..................................... 8
Figure 3: Actors in the LDAP Test Coverage area (System stores certificates).............................................. 9
Figure 4: Actors in the LDAP Test Coverage area (System stores certificate) ............................................... 9
March 28, 2012 Page 2
Modular Specification Phase 3 Direct
Version 0.2
Figure 5: Test Cases Sheet - Preconditions .................................................................................................. 15
Figure 6: Test Cases – multiple Testing Tools.............................................................................................. 16
Figure 7: Test Cases Sheet – Test Steps verification steps .......................................................................... 16
March 28, 2012 Page 3
Modular Specification Phase 3 Direct
Version 0.2
1 Executive Summary
This Test Guide is a companion artifact to the Direct Certificate Discovery Test Packet, which contains
Test Cases, Test Data, inspection Checklists, and a Conformance Questionnaire. Together, the Test Guide
and Test Packet comprise a self-testing kit to enable implementers to verify conformance to the Direct
Certificate Dscovery specifications. It is recommended that implementers review the Test Guide before
examining the Test Packet.
The Test Guide includes project background information, an overview of our test coverage, and
instructions for identifying Testing Tools and executing Test Cases. A detailed walk-through of the Test
Packet artifacts is also provided.
March 28, 2012 Page 4
Modular Specification Phase 3 Direct
Version 0.2
2 Project Background: Modular Specification Phase 3
This section provides an introduction to the Modular Specification Phase 3 project that drove the
production of the Direct Certificate Discovery Test Packet and this Testing Guide.
The overall goal of the Modular Specification Phase 3 project is to use the specifications currently used
for discovery of digital certificates for the Direct Project using a hybrid approach based on DNS and LDAP
and develop implementable, testable specifications along with a conformant test implementation and
high quality Test Cases that can be used to verify conformance.
Within this project are three collaborative workstreams and groups, working under direction from ONC:
1. The Specifications Team: Focused on converting the existing specifications into a matrix format,
designed to clearly delineate normative content into tracable and atomic requirements.
2. Test Implementation Team: Focused on standing up and expanding existing test
implementation code into a demonstration System.
3. Testing Team: Focused on creating vendor neutral Test Cases, data, and conformance checklists
to serve as a written test kit for verifying conformance and/or testing interoperability. The
Testing Team will also perform execution of the test cases on the Test Implementation.
This document was produced by the Testing Team and reviewed by the entire Project Team, and
represents the main project deliverable of the Testing Team’s workstream.
2.1 Document Audience & Sections
The Test Packet is meant to act as a stand-alone and platform-neutral test kit that can be used to test
systems implementing the Direct specifications. It targets three main audiences:
A test team intending to perform an IV&V task,
A product vendor or developer intending to supplement or enhance their internal testing
process, or
A pilot System going through a pre-validation or self-attestation model for joining some kind of
exchange network.
This document is intended to be used in tandem with the Test Packet, which contains the Test Cases,
Test Data, and Inspection Checklists.
March 28, 2012 Page 5
Modular Specification Phase 3 Direct
Version 0.2
3 Test Environment
This section describes the test environment in which the Test Cases are executed. It outlines the
components of the test environment, and some common variations of the testing platform.
3.1 Introduction
The following core work components are required for completion of any Test Cases:
1. Set up a test platform to simulate the user or System activity to be tested.
2. Perform the steps to simulate the type of user or System activity.
3. Note the results at each step and compare them against expected results and communicate the
delta to the appropriate audience (development team, product owners, governing body, etc.).
It is important to note that many people (including professional testers) are familiar with systems using
heavy graphical user interfaces (GUIs) as opposed to messaging systems or integration programming,
which generally have very light (or no) interface. One of the key differences in testing between these
types of systems is the degree of emphasis on the three work areas defined above.
For most GUI-heavy Systems, the weight is firmly in #2 above: running test procedures or
scenarios, which generally have a 1-to-1 relationship to #3 (one step, one comparison to
expected results). As such, when testing a typical shopping cart application there might be a
dozen steps – and each step would have an expected result for comparison.
For testing message or integration-biased Systems, there might be a single logical task (“Send a
message from gateway 1 to gateway 2 with this structure.”), which then might have 100+
specific inspection tasks to follow.
In addition, in messaging-biased testing the tester often has to perform additional work in #1
(adjusting the test platform) to simulate a particular test. For example, to perform a basic
negative test in a shopping cart the tester might simply enter an incorrect credit card number. In
a messaging System, they might have significant work (programming or configuration) to
simulate a similar basic negative Test Case (such as leaving out an security header element).
3.2 Terminology
The following terms are used in the test artifacts (additional terms are defined in Appendix A: Glossary
of Terms):
System, or SUT: the physical system under test.
Testing Tool, or TT: something that is capable of operating as the control end of the message
exchange with the system under test. The Testing Tool is also able to query for certificates in
either the System’s DNS CERT records or the System’s LDAP server(s).
Initiator: the initiator of a message exchange. In the context of the “push” protocols used in the
Direct project (Mail and XDR), this is equivalent to the Sender. Note that in Direct Certificate
March 28, 2012 Page 6
Modular Specification Phase 3 Direct
Version 0.2
Discovery, most of the behavior requirements (e.g. using DNS queries to discover certs) belong
to the Initiator.
Receiver: the receiver of a message exchange.
DNS Server (Aka Name Server) - The Domain Name System (DNS) is maintained by a distributed
database system, which uses the client-server model. The nodes of this database are the name
servers. Each domain has at least one authoritative DNS server that publishes information about
that domain and the name servers of any domains subordinate to it. Used by DIRECT to store
certificates and point to LDAP servers where certificates can be stored.
DNS Client (aka DNS resolver) - The client-side of the DNS is called a DNS resolver. It is
responsible for initiating and sequencing the queries that ultimately lead to a full resolution
(translation) of the resource sought, e.g., translation of a domain name into an IP address.
(from: http://en.wikipedia.org/wiki/Domain_Name_System)
LDAP Server – Stores any organized set of records, often with a hierarchical structure, such as a
corporate electronic mail directory. Provides authenticated LDAP clients search and retrieve
functionality
LDAP Client – Starts LDAP query sessions with the LDAP Server and sends requests to the LDAP
Server. (from: http://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol)
3.3 Components of the Test Environment
The components of the test environment vary depending on the “Test Focus” employed by a test case.
The sections below explain how the compoents differ based on the test focus of the test cases.
3.3.1 DNS CERT records
The first coverage area involves three actors. The System under test (aka “the System”), the Testing
Tool, and the domain name system (DNS) that stores the certificates. Figure 1 shows the actors and the
steps of the basic flow of this scenario. Figure 2 shows the System in the role of the receiver.
Figure 1: DNS CERT Records Test Focus Actor diagram (System is initiator)
March 28, 2012 Page 7
Modular Specification Phase 3 Direct
Version 0.2
Figure 2: DNS CERT Records Test Focus Actor diagram (System stores certificate)
Note that a System may choose to have an LDAP server for some of their certificates and also choose to
store other certificates in CERT resource records. In that case, there may be an LDAP instance referred
to by their DNS, but that actor does not come into play in the DNS CERT test focus tests.
3.3.2 LDAP
The second test coverage area adds another actor, the LDAP server (or service). This test coverage area
also adds two steps to the workflow. Figure 3 demonstrates this environment and has the System
playing the role of the initiator. Figure 4 is the opposite setup and has the the System in the role of the
receiver.
March 28, 2012 Page 8
Modular Specification Phase 3 Direct
Version 0.2
Figure 3: Actors in the LDAP Test Coverage area (System stores certificates)
Figure 4: Actors in the LDAP Test Coverage area (System stores certificate)
March 28, 2012 Page 9
Modular Specification Phase 3 Direct
Version 0.2
4 Test Coverage Overview
This section gives an overview of the coverage of the Test Cases, grouped into logical test coverage
areas. The descriptions in this section serve as a high level guide to the more detailed Test Cases, test
data, and inspection Checklists contained in the Test Cases spreadsheet.
4.1 Key Test Scope Decisions
The following are key decisions made on the scope of testing.
4.1.1 Out of scope: Testing all DNS resource records
The Modular Specifications Phase 3 project leverages the domain name system (DNS) to enable
discovering certificates. We will test the applicable DNS resource records, such as CERT, and SRV
Records, but we will not test all of a System’s DNS records for conformance to the DNS Specifications.
4.1.2 Out of scope: Testing all LDAP attributes
The Modular Specifications Phase 3 project leverages the lightweight directory access protocol (LDAP) to
enable discovering certificates. We will test the applicable LDAP attributes, such as mail and
userCertificate, but other attributes of the InetOrgPerson object class we will not be testing. Some
System’s may choose to extend or constrain their LDAP instances. We consider these modifications out
of scope.
4.1.3 Out of scope: LDAP and DNS Message format
Testing internal LDAP and DNS message conformance is considered out of scope because our tests verify
the expected behavior via end-to-end testing.
4.1.4 Out of scope: Local search matching rules
Some Systems may choose to implement LDAP’s approximate matching rules which can be used in cases
where a slight mispellng of a direct mail address will still return a match. These locally adopted rules are
considered out of scope for our testing.
4.2 Test Coverage Areas
We have identified the following areas for test coverage. We approach each area with a coverage
philosophy that includes testing both success and failure variants.
In the test packet, the Test Cases and Checklists that primarily support a given test coverage area will list
that area in the “Test Focus” column. Note that some areas are not tested directly, but rather through
the test cases of some related area. The following test coverage areas are addressed:
March 28, 2012 Page 10
Modular Specification Phase 3 Direct
Version 0.2
4.2.1 DNS CERT Records
In this test coverage area, we test a System’s ability to store and retrieve certificates using DNS CERT
resource records.
Where System is initiator:
o Variants of DNS CERT Records
X.509 and IPKIX
If the System’s DNS can handle switching from UDP to TCP (if necessary)
DNS queries work for both upper and lowercase
Conditional for Receivers who use CERT RRs to store their DIRECT certificates
4.2.2 LDAP
In this test coverage area, we explore the System’s ability to store and retrieve certificates using the
LDAP mechanism.
Where Systems is the initiator
o Variants of the LDAP search parameters
o Variants of SRV record priorities and weights
o Variants of servers that are available and servers that are not available.
Negative testing
o Applicant as Receiver rejects invalid binding
o Applicant as Initiator handles no matching records
Conditional for Receivers who use LDAP to store their DIRECT certificates
4.3 Testing Roadmap
While the Test Cases can be executed in any order, Testers may find that there is an advantage to
executing them within a larger roadmap, building up from simple messaging to interoperability:
First, perform initial connectivity testing
Next, perform functional testing by executing each of the applicable Test Cases against a single
complement of Testing Tools.
Finally, perform interoperability testing by repeating some or all of the Test Cases against other
systems acting in the role of the Testing Tool, varying the implementations and deployment
models of these other systems.
The first and last steps are outlined below.
4.3.1 Initial connectivity testing
Initial connectivity testing simply means exercising a subset of end-to-end test cases that exercise the
key functionality of the system. We have categorized each test case by “flow” as follows:
Basic success: this is a nominal “happy path” case, which tests the most simple and/or common
way to exercise the functionality.
March 28, 2012 Page 11
Modular Specification Phase 3 Direct
Version 0.2
Variant success: these are the tests that exercise all the variations in functionality, and still
expect a successful result.
Error: these are the tests that ensure the system under test can detect and/or handle error
conditions appropriately.
While testers may decide what constitutes initial connectivity differently, a good starting point would be
to exercise all Basic success cases that apply to the system under test.
4.3.2 Interoperability testing
The conformance Test Cases that accompany this Guide can be used to test interoperability between
systems using the following approach:
Test your system according to your business use cases and expected partners
Test your system against different deployment models
Test your system against different implementations (i.e. based on different COTS products)
First, determine the business use case(s) that require the systems to communicate. Interoperability
encompasses a spectrum of communication capabilities and the level of interoperability required by a
use case depends on its goals. For example, sending a PDF from one system to another for a human
being to print out and act on requires a less intensive level of system interoperability than sending a
medical record that is to be parsed by the receiving system, automatically analyzed for certain
conditions, then used to trigger clinical decisions affecting the well-being of the patient involved.
Based on the level of interoperability required for each use case, determine the corresponding technical
capabilities that must be expressed to demonstrate interoperability. Use the accompanying Test Cases
to create test scenarios that demonstrate those technical capabilities. Two disparate systems that
execute these scenarios successfully can be said to be interoperable at the required level.
Data exchange partners may use the accompanying Test Cases to set up their own test beds. These can
be used to vet the systems of new partners. Such test beds will be most effective if they host a set of
systems representative of those in production.
Finally, test your system against different implementations, as available. For example, test against a
system that is built from different components than yours: using different email clients, email servers,
messaging stacks, etc.
March 28, 2012 Page 12
Modular Specification Phase 3 Direct
Version 0.2
5 Testing Prep
This section describes the steps that must be taken in preparation for executing Test Cases. A Tester
must stand up a Testing Tool, identify relevant Test Cases, and compile Test Data for use in the Test
Cases.
5.1 Stand Up Testing Tool
The first step in preparing for testing is to stand up a Testing Tool as described in section 3.3. In some
Test Cases multiple Testing Tools are used. These are identified in the Test Cases themselves.
5.2 Identify relevant Test Cases
A Conformance Questionnaire is included with this Test Packet to assist Testers with identifying which
Test Cases you will need to execute. The Questionnaire elicits information about the System
implementation type as well as which optional functionality is supported and this information is mapped
to the Test Case IDs found in the Test Case spreadsheet.
Use the Questionnaire to determine which Test Cases are Required, or Optional for your system.
5.3 Load Test Data
The Testing Tool should be loaded with certificates for different test cases. The test data is mentioned in
the “Test Data Load Notes” Column of each test case.
5.4 General Test Case Guidance
The Direct specifications include requirements that indicate varying degrees of conformance precision,
such as MAY, SHOULD, or MUST. The following guidance provides some insight into how the Test Cases
were written to address these variants.
Test Cases may be based on a function that is RECOMMENDED or SHOULD be implemented, rather than
a function that MUST be implemented. This is reflected directly in the test optionality (i.e. required,
optional, or conditional). If your System supports the requirement, do execute those test cases. If your
System does not support the requirement, you may want to make a note of this when recording your
testing outcomes (see Section 7.2.1).
March 28, 2012 Page 13
Modular Specification Phase 3 Direct
Version 0.2
6 Testing Materials
This Section serves as a guide to each Test Packet artifact, which contains the Test Cases, Test Data
instructions, and message verification tools. The Test Cases spreadsheet is your starting point and the
Test Steps contained in each Test Case will direct you to the appropriate Test Data and verification tools.
6.1 Test Packet Contents
The Test Packet contains sheets entitled Cover Sheet, and Test Cases.
Test Packet Sheets
Sheet Name Description How It’s Used
Cover Sheet Cover page, revision history Cover sheet.
Test Cases
Test Cases Test Cases for all Serves as a step-by-step guide for
implementation types. executing Test Cases. Each Test Case
directs the Tester to test data in the Test
Data Index or XDR Test Messages that is
used for that Test Case. Each Test Case
also directs the Tester to the appropriate
checklists for conformance verification.
6.2 Navigating the Test Cases Sheet
Each Test Case contains the following columns. Use the steps below to navigate through the Test Cases
and Test Data.
Test Cases Sheet Columns
Column Description
Prefix A unique prefix to be used for a group of test cases and/or checklist
rows. Combined with the ID, this yields a unique identifier. In this case,
we used DTS for Direct Transport and Security
ID Unique identifier for the Test Case. Note that this identifer is only used
for identification and traceability; it has no inherent meaning. For
example, consecutive numbering of a group of test cases should not be
assumed.
Test Focus Area of functionality being tested.
Flow The type of flow being tested. Can be one of: Basic success, Variant
success, or Error.
System is: Initiator/Receiver Whether the System under test is the message Initiator or Receiver
March 28, 2012 Page 14
Modular Specification Phase 3 Direct
Version 0.2
Purpose/Description Brief description of what is being tested.
Test Steps Test execution steps. May also contain Preconditions, reference to Test
Data, reference to Checklists and other verification instructions.
Required/Conditional Indicates whether this Test Case is Required or Conditional for Systems
that support the functionality under test.
Additional Info/Comments Additional information, descriptions, context.
RTM Traceability to the RTM
Underlying Spec Traceability to the underlying specification
Data Load Notes The data that the Testing Tool or the System needs to have loaded prior
to the Test being run
Request Contains… A description of what the DNS or LDAP message needs to contain
Expected Response Describes the expected response after the message was received by
either the Testing Tool or the System
6.2.1 By using the Questionnaire, you should already know the Test Cases that apply to your
system. By looking at the SUT, Source and Dest columns, you can identify the role(s) your
system will play (i.e. System under test or Testing Tool) in each test case.
6.2.2 Look in the Test Data Load Notes to identify the data that needs to be loaded on the Testing
Tool or the System prior to the test being run.
6.2.3 In some Test Cases, the Test Steps column defines Preconditions for the Test Data, System
or Testing Tool.
Figure 5: Test Cases Sheet - Preconditions
March 28, 2012 Page 15
Modular Specification Phase 3 Direct
Version 0.2
6.2.4 Some Test Cases require the use of two or more Testing Tools.
Figure 6: Test Cases – multiple Testing Tools
6.2.5 Some of the test cases are “Subflows” of another “basic flow” test case. These subflows
replace a step in the basic flow but then refer back to the basic flow for any remaining steps.
In the following case, Step 1 of DTS 8 would be replaced with Step 1 of this Test Case. After
step one, the test case would resume with Step 2 of DTS 8.
Figure 7: Test Cases Sheet – Test Steps verification steps
March 28, 2012 Page 16
Modular Specification Phase 3 Direct
Version 0.2
7 Executing Testing
Once you have completed the Testing Prep activities and are familiar with the contents of the Test
Packet, you are ready to execute Test Cases. This section provides step by step instructions for executing
Test Cases and conducting analysis of the resulting messages / outcomes.
7.1 Testing Steps
After completing the testing prep activities described in section 5, the Tester must execute the following
steps once for each Test Case.
1. Define which of your two endpoint nodes will act as the Initiator or Receiver for the case being
run. If required, identify additional Testing Tools as instructed by the Test Case.
2. Compile the correct Test Data or Message(s) for the Test Case being run, as described in Section
6.2.5.
3. Configure the System, Testing Tool, and Test Data to meet Preconditions, if required by the Test
Case.
4. Execute the test by following the Test Steps.
5. Collect logs or messages from the System or Testing Tool endpoint(s).
6. Perform verification inspection and analysis as directed by the Test Steps.
7.2 Analysis
7.2.1 Reporting Test Case Outcomes
It is recommended that Testers record the outcomes of the Test Cases that were executed. A suggested
method is to record the Test Case ID, the log file names, and any relevant notes for each Test Case in a
single report. Notes might include discrepancies in expected outcomes, failures, optional functionality
not supported, etc.
To resolve discrepancies in expected outcomes, a Tester might submit a question to the appropriate
specification body or to the maintainers of the test package, or modify the System(s) under test and
then rerun the Test Case.
March 28, 2012 Page 17
Modular Specification Phase 3 Direct
Version 0.2
8 Appendix A – Glossary of Terms
A brief list of terms referred to in this document or within the associated Test Cases follows.
Glossary of Terms
Term Description
Checklist Inspection criteria for verifying conformance of a Message to the
specifications. May contain data/fields, formats, values,
required/optional, etc.
Implementation Type Characterization of the type of Direct messages a System can create
and receive.
Test Implementation (TI) A benchmark System used to provide a definitive interpretation of the
specifications. It is developed in concert with specifications and the
test suite to collectively prove that the specifications are
implementable and clear.
Requirements Traceability The document that correlates the requirements, Test Cases, and
Matrix (RTM) functionality of a System. In this project’s context, it also contains a
flattened version of the requirements in one place (not merely a table
showing relationships).
Reviewer The person (or automated tool) looking at the resultant message from
an execution script, often performing inspections according to an
associated conformance checklist. The Test Steps direct the Reviewer
through the necessary inspection steps.
System System being tested by a Test Case.
Test Case One logical path through a System and/or Testing Tool pair. The Test
Case includes a description of its test goal(s), Testing Steps, test data,
and verification steps.
Tester The person (or automated tool) executing the Test Case.
Testing Tool An endpoint that is capable of operating as the control end of the
message exchange specified by a given Test Case, where the “System”
is the side being tested. The Testing Tool can be a copy of the System
under test, a separate System, or any other endpoint simulator.
March 28, 2012 Page 18
