Metrics Report Template by akp25756

VIEWS: 50 PAGES: 13

More Info
									Advancing Metrics
 on the Standards
      Track
        Lab Testing Results
draft-morton-ippm-advance-metrics-01
       Al Morton July 2010
 ACK: Len Ciavattone for NetProbe


                                       1
Outline
   Definition-centric metric advancement
   Possible Report Template
       Supports the Protocol Action Request
   Examples of the Definition-centric
    approach:
       Revised Lab test methods
       Results for one implementation
   Open call for participation in Lab tests

                                               2
Definition-Centric Process (revised)
     ,---.
   /      \
  ( Start )
   \      /    Implementations
    `-+-'         +-------+
       |         /|   1   `.
  +---+----+    / +-------+ `.-----------+      ,-------.
  | RFC     | /              |Check for |     ,' was RFC `. YES
  |         | /              |Equivalence..... clause x     -------+
  |         |/    +-------+ |under       |    `. clear? ,'         |
  | Metric \.....|    2   ....relevant   |     `---+---'      +----+---+
  | Metric |\     +-------+ |identical |        No |          |Report |
  | Metric | \               |network    |      +--+----+     |results+|
  | ...     | \              |conditions |      |Modify |     |Advance |
  |         |   \ +-------+ |            |      |Spec    +----+RFC     |
  +--------+     \|   n   |.'+-----------+      +-------+     |request |
                  +-------+                                   +--------+




                                                                           3
Key Points (the sub-points)
   Start with an RFC
       Focus on a specific clause
   Run test(s) with Implementations
       Test plan is customized to a specific clause
   Evaluate Measurements & Compare
       Expected measurement results are Clear
       Obvious place to take action if text is found to be
        ambiguous
   Final state is Report Dev. for Protocol Action Req.

                                                              4
Test Set-up (more details in I-D)
                         negotiated
                         100baseTx-FD
        NetProbe MS-1
                        10.201.0.1      eth2 10.201.0.254

        NetProbe MS-2                   eth3 10.202.0.254
                        10.202.0.1
                                             Network Emulator
                        10.203.0.1
        NetProbe MS-3                   eth4 10.203.0.254
                                        eth5 10.204.0.254
                        10.204.0.1
        NetProbe MS-4
Management                  Test LANs
LAN

                                                            5
NetProbe 5.8.5
   Runs on Solaris (and Linux, occasionally)
   Pre-dates *WAMP, functionally similar
   Software-based packet generator
   Provides performance measurements
    including Loss, Delay, PDV, Reordering,
    Duplication, burst loss, etc. in post-
    processing on stored packet records


                                                6
Report Template – Loss Threshold
   3.1. One-way Delay, Loss threshold, RFC
    2679 (procedure)
       3.1.1. NetProbe Lab results for Loss Threshold
       3.1.2. XXX Lab Results for Loss Threshold
            <your compliant implementation here>
       3.1.3. Conclusions on Lab Results for Loss
        Threshold



                                                         7
Section 3.1 – Loss Threshold
 See Section 3.5 of [RFC2679], 3rd bullet point and also
  Section 3.8.2 of [RFC2679].
 1. configure a path with 1 sec one-way constant delay
 2. measure (average) one-way delay with 2 or more
  implementations, using identical waiting time thresholds for
  loss set at 2 seconds
 3. configure the path with 3 sec one-way delay (or change
  the delay while test is in progress, measurements in step 2)
 4. repeat measurements
 5. observe that the increase measured in step 4 caused all
  packets to be declared lost, and that all packets that arrive
  successfully in step 2 are assigned a valid one-way delay.
Results: 22 of 38 packets were declared lost (post-process).


                                                                  8
Section 3.2: First-bit to Last-bit
See Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330].

    1. configure a path with 1000 ms one-way constant delay, and ideally
    including a low-speed link (10-baseT, FD)
    2. measure (average) one-way delay with 2 or more implementations,
    using identical options and equal size small packets (e.g., 32 octet IP
    payload)
    3. maintain the same path with 1000 ms one-way delay
    4. measure (average) one-way delay with 2 or more implementations,
    using identical options and equal size large packets (e.g., 1400 octet IP
    payload)
    5. observe that the increase measured in steps 2 and 4 is equivalent to
    the increase in ms expected due to the larger serialization time for each
    implementation. Most of the measurement errors in each system should
    cancel, if they are stationary.




                                                                                9
3.2: First-bit to Last-bit, NetProbe
 1400 byte payload   32 byte payload
 Delay for each mode   (one mode)                Delay Diff        Expected Diff
   microseconds        microseconds              microseconds      microseconds
   1001621                     1000356               1265             1094.4
   1002735                     1000356               2379             1094.4
                                      TxDelay(us)

        1003000

        1002500

        1002000

        1001500

        1001000

        1000500
                  1   4   7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58




                                                                                   10
3.2: First-bit to Last-bit, NetProbe
   Bi-modal delay behavior in the Network
    Emulator for UDP payload 96 octets & up
       Same for Poisson, Periodic, 1 sec, 10ms spacing
       Not evident in tests on the Management LAN
   Delay increased with larger payload, but more
    than expected (170 us for low mode)
   Suggestions to investigate further are welcome
Tests on Management LAN, 100baseTx-FD
  1400 byte payload   32 byte payload
     Ave Delay           Ave Delay         Diff      Expected Diff
    microseconds        microseconds    microseconds microseconds


        443.37           143.57          299.8        109.44         (190.36)
                                                                           11
Other Examples
   One-way Delay, RFC 2679
        This test is intended to evaluate measurements in sections 3 and 4 of
         [RFC2679].

            Average delays before/after 2 second increase

        Average pre-increase delay, microseconds                1000276.6
        Average post 2s additional, microseconds                3000282.6
        Difference (should be ~= Y = 2s)                        2000006

   Error Calibration, RFC 2679
        This is a simple check to determine if an implementation reports the error
         calibration as required in Section 4.8 of [RFC2679].




                                                                                      12
Summary
   Key clauses of RFC 2679 evaluated in Lab
   One Implementation: NetProbe
   Other volunteers?
   Comments on approach to reporting?
       draft-morton-ippm-advance-metrics-01

   What other RFCs should we target first?



                                               13

								
To top