Docstoc

PhD thesis

Document Sample
PhD thesis Powered By Docstoc
					To my family
Acknowledgements

                                              o
I would like to thank my supervisor Mats Bj¨ rkman for seeing potential in me
and for hiring me as his PhD student. Mats, you have been my great guide and
companion during both private and work related ups and downs. I respect and
admire you very much for that!
    I would like to thank my supervisor Bob Melander. Without your guid-
ance and support I would never have come this far. You have taught me that
everything is possible if you just try hard enough!
    I owe my deepest gratitude to Svante Ekelin, Christofer Flinta, Erik Har-
                      a
tikainen, Jan-Erik M˚ ngs, Martin Nilsson and my two supervisors, all involved
in the EvaluNet project group. As you know, our team work has been very
rewarding for me, and I must admit that it is difficult to find occasions I have
been happier at work than after interesting and challenging discussions with
you!
    I would also like to thank Jonas Neander and Ewa Hansen, both part of
                                      a
the data communication group at M¨ lardalen University, for their support. You
have really enriched my time as a PhD student! I also want to send my great
thanks to the rest of the personal at my department for interesting discussions,
coffee break laughs and in general for making my department a creative and
fun place to work at.
    I wish to thank the CoRe-group at Uppsala University for giving me the
opportunity of using their facilities when working in Uppsala. I particularly
want to thank Christian Rohner for giving me feedback on my PhD thesis and
on papers I have written earlier. Also, input and comments on papers from
Henrik Lundgren has been rewarding!
    I would also like to thank Annikki Welin and Rikard Holm for creative
discussions and for helping me with many of the measurements performed
during the first year of my PhD studies.
    To my mom Elly and dad Lennart, thank you for providing me with chem-

                                      iii
iv


istry boxes, astronomical telescopes, computers and other fun stuff, it really
pointed out the direction of my future! My parents as well as my brother Mat-
tias and sister Jenny have always supported me and encouraged me during my
PhD studies. Thank you!
    Finally, my deepest thanks goes to Maaret for always motivating and be-
lieving in me and making my life outside work enjoyable.



   I also wish to thank my sponsors. The Swedish Research Council gave me
economical support during my first three years as a PhD student. Then Vin-
nova and the KK Foundation supported my work through the EvaluNet project.
Financial support for travels and alike was granted from ARTES++.

                                                           Andreas Johnsson
                                                         a a
                                                        V¨ ster˚ s, April, 2007
Contents

    List of Publications                                                         1


I      Thesis                                                                   3

1 Introduction                                                                   5
  1.1 Background and motivation . . . . . . . . . . . . . . . . .       .   .    5
  1.2 Outline of the thesis . . . . . . . . . . . . . . . . . . . . .   .   .    7
  1.3 Research area . . . . . . . . . . . . . . . . . . . . . . . .     .   .    7
       1.3.1 Network model . . . . . . . . . . . . . . . . . . .        .   .    7
       1.3.2 Link capacity, available bandwidth and cross traffic        .   .    8
       1.3.3 Measuring the available bandwidth . . . . . . . .          .   .   10
       1.3.4 Research challenges . . . . . . . . . . . . . . . .        .   .   11
  1.4 Research methodology . . . . . . . . . . . . . . . . . . .        .   .   12
  1.5 My contributions . . . . . . . . . . . . . . . . . . . . . .      .   .   13

2 Summary of the papers                                                         15
  2.1 Paper A: On the Analysis of Packet-Train Probing Schemes . .              15
  2.2 Paper B: An Analysis of Active End-to-end Bandwidth Mea-
      surements in Wireless Networks . . . . . . . . . . . . . . . .            16
  2.3 Paper C: Real-time Measurement of End-to-End Available Band-
      width Using Kalman Filtering . . . . . . . . . . . . . . . . .            17
  2.4 Paper D: Measuring the Impact of Active Probing on TCP . .                18
  2.5 Paper E: On measuring the available bandwidth in wireless
      802.11b networks . . . . . . . . . . . . . . . . . . . . . . . .          18

                                      v
vi    Contents


3 Related work                                                                     21
  3.1 Methods and tools . . . . . . . . . . . . . . . . . . . . . . . .            22
       3.1.1 Link capacity estimation . . . . . . . . . . . . . . . .              26
  3.2 Theoretical work . . . . . . . . . . . . . . . . . . . . . . . .             27

4 Conclusions and future research challenges                                       29

Bibliography                                                                       31


II     Included Papers                                                             35
5 Paper A:
  On the Analysis of Packet-Train Probing Schemes                                  37
  5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . .   .   .   .   .   39
  5.2 Description of patterns . . . . . . . . . . . . . . . . .    .   .   .   .   40
        5.2.1 A multiple-hop model for route delay variation       .   .   .   .   40
        5.2.2 Mirror pattern . . . . . . . . . . . . . . . . . .   .   .   .   .   42
        5.2.3 Chain pattern . . . . . . . . . . . . . . . . . .    .   .   .   .   44
        5.2.4 Quantification pattern . . . . . . . . . . . . . .    .   .   .   .   45
  5.3 Testbed setup . . . . . . . . . . . . . . . . . . . . . .    .   .   .   .   47
  5.4 Signatures . . . . . . . . . . . . . . . . . . . . . . . .   .   .   .   .   48
        5.4.1 The independence signature . . . . . . . . . .       .   .   .   .   48
        5.4.2 The mirror signature . . . . . . . . . . . . . .     .   .   .   .   48
        5.4.3 The rate signature . . . . . . . . . . . . . . . .   .   .   .   .   49
        5.4.4 Quantification signature . . . . . . . . . . . .      .   .   .   .   50
  5.5 Mean and median analysis using patterns . . . . . . .        .   .   .   .   50
  5.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . .    .   .   .   .   52
  Bibliography . . . . . . . . . . . . . . . . . . . . . . . . .   .   .   .   .   52

6 Paper B:
  An Analysis of Active End-to-end Bandwidth Measurements in Wire-
  less Networks                                                          55
  6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
  6.2 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . 59
        6.2.1 Measurement model . . . . . . . . . . . . . . . . . . 59
        6.2.2 The testbed . . . . . . . . . . . . . . . . . . . . . . . 62
        6.2.3 Research question and experiments . . . . . . . . . . 62
  6.3 Experimental results . . . . . . . . . . . . . . . . . . . . . . 63
        6.3.1 Measurement results in wireless networks . . . . . . . 63
                                                             Contents       vii


   6.4   Analysis of experimental results . . . . . . . . . . . . . . . .   69
         6.4.1 The extended TOPP model . . . . . . . . . . . . . . .        69
         6.4.2 Estimated vs. physical link transmission capacity of a
                802.11 wireless link . . . . . . . . . . . . . . . . . .    71
   6.5 Other observations . . . . . . . . . . . . . . . . . . . . . . .     74
   6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . .     74
   Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   75

7 Paper C:
  Real-time Measurement of End-to-End Available Bandwidth Using
  Kalman Filtering                                                        79
  7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
        7.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . 81
        7.1.2 Related work . . . . . . . . . . . . . . . . . . . . . . 82
        7.1.3 Paper organization . . . . . . . . . . . . . . . . . . . 83
  7.2 Measuring available bandwidth . . . . . . . . . . . . . . . . . 83
        7.2.1 Definition of available bandwidth . . . . . . . . . . . 83
  7.3 Applying kalman filtering to a piecewise linear network model 85
        7.3.1 The network model . . . . . . . . . . . . . . . . . . . 85
        7.3.2 Estimation by filtering . . . . . . . . . . . . . . . . . 88
  7.4 Validation of the method . . . . . . . . . . . . . . . . . . . . 91
        7.4.1 Measurement testbed setup . . . . . . . . . . . . . . . 92
        7.4.2 Impact of probe packet size and probe train length . . 94
        7.4.3 Comparison of BART and pathChirp in the Testbed
              Environment . . . . . . . . . . . . . . . . . . . . . . 97
        7.4.4 Comparison of BART and pathChirp using an Internet
              Path . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
  7.5 Discussion and conclusions . . . . . . . . . . . . . . . . . . . 107
  7.6 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
  Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

8 Paper D:
  Measuring the Impact of Active Probing on TCP                             113
  8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . .    115
  8.2 Research questions and experimental setup . . . . . . . . . .         116
  8.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   118
      8.3.1 Initial results . . . . . . . . . . . . . . . . . . . . . .     118
      8.3.2 Measuring the impact of active end-to-end probing on
              TCP . . . . . . . . . . . . . . . . . . . . . . . . . . .     120
viii       Contents


       8.4 Network friendly probing . . . .     .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   126
       8.5 Tools with special flight patterns    .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   127
       8.6 Discussion . . . . . . . . . . . .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   129
       8.7 Conclusions . . . . . . . . . . .    .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   129
       Bibliography . . . . . . . . . . . . .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   130

9 Paper E:
  On measuring the available bandwidth in wireless 802.11b net-
  works                                                                   133
  9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
  9.2 Measurement model . . . . . . . . . . . . . . . . . . . . . . . 136
        9.2.1 The BART Kalman filter . . . . . . . . . . . . . . . . 136
        9.2.2 Tuning of the Kalman filter . . . . . . . . . . . . . . . 138
        9.2.3 The relation between estimates of the available band-
                 width and link capacity using BART in 802.11b networks139
  9.3 Research question and experimental setup . . . . . . . . . . . 140
        9.3.1 Research questions . . . . . . . . . . . . . . . . . . . 140
        9.3.2 Experimental setup . . . . . . . . . . . . . . . . . . . 141
  9.4 Measurements of the rate response curve in 802.11b networks
        and its relation to available bandwidth . . . . . . . . . . . . . 142
        9.4.1 Send rate problems . . . . . . . . . . . . . . . . . . . 142
        9.4.2 Does BART estimate correctly? . . . . . . . . . . . . 145
  9.5 Tuning the BART Kalman filter for qualitative estimation in
        802.11b networks . . . . . . . . . . . . . . . . . . . . . . . . 148
        9.5.1 Tuning of Q . . . . . . . . . . . . . . . . . . . . . . . 151
        9.5.2 Theoretical discussion of choice of Q . . . . . . . . . 153
  9.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
  Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
List of Publications

The following articles are included in this thesis:

  A. On the Analysis of Packet-Train Probing Schemes, Andreas Johnsson,
                               o
     Bob Melander and Mats Bj¨ rkman, In proceedings of the International
     Conference on Communication in Computing, Special Session on Net-
     work Simulation and Performance Analysis, Las Vegas, June 2004.
  B. An Analysis of Active End-to-end Bandwidth Measurement Methods in
                                                      o
     Wireless Networks, Andreas Johnsson, Mats Bj¨ rkman and Bob Me-
     lander, In proceedings of the 4th IEEE/IFIP End-to-end Monitoring Tech-
     niques and Services workshop, Vancouver, Canada, April 2006.

  C. Real-time Measurement of End-to-End Available Bandwidth Using Kalman
     Filtering, Svante Ekelin, Martin Nilsson, Erik Hartikainen, Andreas Johns-
                     a                                o
     son, Jan-Erik M˚ ngs, Bob Melander and Mats Bj¨ rkman, In proceedings
     of the 10th IEEE/IFIP Network Operations and Management Sympo-
     sium, Vancouver, Canada, April 2006.
  D. Measuring the Impact of Active Probing on TCP, Andreas Johnsson and
            o
     Mats Bj¨ rkman, In proceedings of the SCS International Symposium on
     Performance Evaluation of Computer and Telecommunication Systems,
     Calgary, Canada, July 2006.
  E. On measuring the available bandwidth in wireless 802.11b networks,
                                 o
     Andreas Johnsson and Mats Bj¨ rkman, Not yet published.




                                        1
2       LIST OF PUBLICATIONS


    Besides the above articles, I have (co-)authored the following peer-reviewed
scientific papers:

     I. On the Comparison of Packet-Pair and Packet-Train Measurements, An-
        dreas Johnsson, In Proceedings of the First Swedish National Computer
        Networking Workshop, Arlandastad, September 2003.
    II. Bandwidth Measurements from a Consumer Perspective - A Measure-
                                              o
        ment Infrastructure in Sweden, Mats Bj¨ rkman, Andreas Johnsson and
        Bob Melander, Presented at the Bandwidth Estimation (BEst) Workshop,
        San Diego, December 2003.
    III. A Study of Dispersion-based Measurement Methods in IEEE 802.11 Ad-
                                                   o
         hoc Networks, Andreas Johnsson, Mats Bj¨ rkman and Bob Melander,
         In proceedings of the International Conference on Communication in
         Computing, Special Session on Network Simulation and Performance
         Analysis, Las Vegas, June 2004.
    IV. DietTopp: A First Implementation and Evaluation of a Simplified Band-
        width Measurement Method, Andreas Johnsson, Bob Melander and Mats
          o
        Bj¨ rkman, In Proceedings of the Second Swedish National Computer
        Networking Workshop, Karlstad, November 2004.
     V. Bandwidth Measurement in Wired and Wireless Networks, Andreas Johns-
                                      o
        son, Bob Melander and Mats Bj¨ rkman, In Proceedings of the Fourth
        Annual Mediterranean Ad Hoc Networking Workshop, Porquerolles, France,
        June, 2005.
    VI. How does Available Bandwidth Measurement Methods Affect TCP?, An-
        dreas Johnsson, In Proceedings of the third Swedish National Computer
        Networking Workshop, Halmstad, November 2005.
  I

Thesis




  3
Chapter 1

Introduction

1.1 Background and motivation
Internet has gained much popularity among the public since the mid 1990’s and
is now an integrated part of our society. A large range of high-speed broadband
providers and the development of new and more efficient Internet applications
increase the possibilities to watch movies and live TV, use IP-telephony and
share files over the Internet. Such applications demand high data transmission
rates, which in turn consume network bandwidth. Since several users must
share the common bandwidth capacity on the Internet, there will be locations
in the network where the demand is higher than the capacity. This causes
network congestion, which has negative impact on both the data transmission
rate and transmission quality.
    If the status of the network path can be predicted, it is possible to use that
information in network applications. For example, the network status can be
used to adapt the traffic load in order to avoid congestion on a network path.
The network status information can also be used when making a decision on
whether to setup a new connection or not. For example, an IP-telephony ses-
sion requires that the latency and jitter between the two communicating nodes
are low. The IP-telephony application also requires that there is enough avail-
able bandwidth to transfer the encoded voice data over the network. If these
requirements are fulfilled, the connection can be established and the involved
people can start talking over the Internet.
    The status of a network such as the Internet must be measured. This is
because the Internet is a best-effort network that in the general case does not

                                        5
6     Chapter 1. Introduction


give users opportunities to make bandwidth reservations for their traffic.
    This thesis concerns methods for actively measuring the status of the net-
work path between two nodes. One important metric to measure is the end-to-
end available bandwidth of a network path. The available bandwidth is in this
thesis interpreted as the maximum transfer rate that can be used without caus-
ing congestion on any of the links between two nodes. The available bandwidth
measurement methods studied in this thesis inject so called probe packets into
the network path between the sender and the receiver. By conducting analysis
on the separation of the probe packets on the sender and on the receiver side,
an estimate of the available bandwidth is obtained.
    The main contributions of this thesis are: a model for measuring the avail-
able bandwidth in networks where the bottleneck is a wireless 802.11 link.
Further, a measurement method called BART is evaluated in both wired and
wireless networks. It is also studied how the injected probe packets affect other
network traffic.
    An alternative approach to measure the available bandwidth is to deploy
passive measurement methods. As opposed to active measurement methods,
passive measurement methods require access to routers in the network and can
only be used by network administrators and other persons with authorized ac-
cess to the network. Passive measurement methods listen to the network traffic
flowing through the network, rather than injecting probe packets. Passive mea-
surement methods are not within the scope of this thesis.
    There exist many examples of applications that would benefit from having
an updated view of the available bandwidth of the network path. For example,
the available bandwidth estimate can be used as input to the adaptive machinery
of streaming audio and video. If the available bandwidth is low, the streaming
application can reduce its send rate by decreasing the quality of the audio or
video, then less data needs to be sent per time unit. On the other hand, if
the available bandwidth is high, more data can be transmitted without causing
congestion and hence the quality of the audio and video can be increased.
    Another example is the following; suppose a user wants to download a
large file from a website on the Internet. By deploying active end-to-end mea-
surements he or she (or the web browser itself) can decide which mirror site to
use by estimating the available bandwidth to each site and then calculate the
minimum download time. Other examples of applications include network di-
agnosis, load balancing in routers and verification of service level agreements
in user-corporation or corporation-corporation relations.
                                                1.2 Outline of the thesis       7


1.2 Outline of the thesis
This thesis is divided into two parts, I and II. Part I provides an overview of
the research area, the contributions of this thesis, related work, conclusions
and future work. The second part of the thesis consists of five scientific papers
labeled A, B, C, D and E, respectively. Papers A - D are peer-reviewed while
paper E is not yet published.



1.3 Research area
This thesis focuses on how to actively measure end-to-end performance in IP
networks. Most methods discussed in this thesis require a probe-packet sender
and a receiver. The sender injects so called probe packets into the network
path with a pre-defined packet separation, corresponding to a probe rate. The
probe packets are affected by the current status of the network, and this may
change the probe-packet separation. This separation is measured at the re-
ceiver by time stamping each probe-packet arrival. Depending on the analysis
method, different performance metrics can be estimated, such as the available
bandwidth.
    In this section the research area of active end-to-end measurement methods
is introduced. First the network model assumed is briefly described. Then a
definition of link capacity, available bandwidth and cross traffic is given. A
high level description on how to measure the available bandwidth is presented.
More detailed descriptions of methods and tools for measuring available band-
width are given in Chapter 3. This section ends with a review of research
challenges addressed in this thesis.


1.3.1 Network model
In this section the network model is presented. Even though it is simplistic,
encompassing few parameters, it is shown in Part II of this thesis that the avail-
able bandwidth measurement methods studied produce accurate estimates in
both testbed and Internet scenarios.
    The networks that the methods studied in this thesis intend to measure are
best-effort packet-switched network. This means for example that no band-
width reservations can be made and that all network traffic is transmitted as
discrete packets.
8       Chapter 1. Introduction


    A network path consists of several consecutive links with different capaci-
ties C. The links in the network are inter-connected by routers and each router
has several incoming and outgoing links. The task of the router is to forward
incoming traffic to an outgoing link that leads towards the destination of the
data packets.
    A router is modeled as a set of queues, one for every outgoing link. If the
amount of network traffic that is forwarded to a specific outgoing link is larger
than the capacity of that link, the data packets are typically stored in a first-in-
first-out fashion. This leads to a delay as a result of the congestion. In case the
buffered traffic exceeds the size of the queue data packets are dropped. This
situation is called persistent congestion.

1.3.2 Link capacity, available bandwidth and cross traffic
The term available bandwidth has had varying definitions in the literature. In
this section, definitions of both link capacity and available bandwidth as well
as a short description of cross traffic are given with the thesis topic in mind,
which is the available bandwidth and link capacity at the IP layer in IP net-
works. The definitions of link capacity and available bandwidth are taken from
[1].
     Each link j in an end-to-end network path has a certain link capacity Cj ,
which is defined as the maximum transmission rate over that link (bit/s). The
maximum rate is determined by the link interfaces used by the nodes in the net-
work path. The link capacity is typically constant, but can shift in e.g. wireless
802.11 networks when the radio quality reaches some upper or lower thresh-
old. The link utilization for each link j varies over time due to cross traffic1 ,
as illustrated in Figure 1.1. A variety of parameters can be used to describe
the cross traffic, such as the intensity distribution, the packet size distribution
and the aggregation of cross-traffic flows. Measurement of these parameters
in different scenarios and networks is a research area in itself (e.g. in [2] the
packet-size distribution was studied).
     The intuitive definition of available bandwidth is the unused portion of the
link capacity during some time interval. The portion of the link capacity that is
in use during a measurement corresponds to the cross traffic Xj = Xj (t, ∆),
which is defined by

                                               1
                                       Xj =      Aj [t − ∆, t]               (1.1)
                                               ∆
    1 The   cross traffic is packets being transmitted over a network link.
                                                                     1.3 Research area           9


                        1


                       0.9


                       0.8


                       0.7


                       0.6
    Link utilization




                       0.5


                       0.4


                       0.3


                       0.2


                       0.1


                        0
                             100   200    300    400   500    600    700   800      900   1000
                                                       Time



                              Figure 1.1: Link utilization as a function of time.


where Aj [t − ∆, t] is the number of cross traffic bits over link j during a time
interval ∆. The available bandwidth Bj = Bj (t, ∆) is defined as

                                                Bj = Cj − Xj .                               (1.2)

  The available bandwidth of an end-to-end path is then defined as the mini-
mum available bandwidth for each link j. That is,

                                                 B = min Bj .                                (1.3)
                                                       j

The link corresponding to B is often referred to as the bottleneck link or the
tight link of the network path.
    This definition of available bandwidth can be interpreted as follows; if
sending packets at a rate below B, the packets will not cause congestion on
the network path. On the other hand, if sending packets at a higher rate than B,
the packets will be queued at the bottleneck link, and thus induce congestion.
10     Chapter 1. Introduction




Figure 1.2: High-level description on how to actively measure the end-to-end
available bandwidth.


1.3.3 Measuring the available bandwidth
Assuming that two nodes want to measure the available bandwidth on the end-
to-end path linking them together, then so called active probing is a feasible
approach. This section describes, at an abstract level, how the end-to-end avail-
able bandwidth can be measured using methods that rely on active probing.
    The principle of active probing is to inject so called probe packets into the
path to be measured. Assume that the sender in Figure 1.2 wants to measure the
available bandwidth of the path to the receiver. Then the sender injects probe
packets with initial mean packet separation ∆in . The probe packets traverse
the network path, and along the way the separation may change due to cross
traffic or link properties, such as the capacity of the links. At the receiver
side each probe packet is time stamped when it arrives. Then ∆out can be
calculated, which is the separation between probe packets at the receiver side.
If the mean input separation of several successive measurements is equal to
the output separation, it can be assumed that no congestion has occurred on
the path. That is, the probe rate (proportional to the inverse of the probe-
                                                     1.3 Research area        11


packet separation) is less than the available bandwidth B. On the other hand,
if the mean output separation is greater than the input separation, the probe
packets have experienced congestion and thus the separation has increased in
proportion to the amount of cross traffic on the path. The probe rate is in this
case above the available bandwidth. Most available bandwidth measurement
methods rely on this idea, which is called self-induced congestion. It originates
from the fact that the probe packets themselves induce congestion on the path
for a very short time interval in order to be able to measure how much cross
traffic there actually is, and then calculate the available bandwidth.
     Both ∆out and ∆in are fed into an analysis algorithm. The analysis is
based on a measurement model that is used to produce estimates of the avail-
able bandwidth using samples of ∆out and ∆in as input. (Different measure-
ment models are described in Chapter 3.) The resulting available bandwidth
estimate can then be used in an application such as a network monitoring tool
that monitors an end-to-end path and plots the estimates in a graph, as illus-
trated in Figure 1.2.
     The approach of self-induced congestion requires no previous knowledge
of the network topology. This is desirable since the link property information
is hard or impossible to retrieve for the average user on the Internet.
     In Chapter 3 a more comprehensive description of different methods and
tools in this area of research is presented. The scheme of how to inject the
probe packets and the analysis of the probe-packet time stamps vary depending
on the method.

1.3.4 Research challenges
The research challenges addressed in this thesis are summarized below.

   • Find and evaluate new and lightweight active end-to-end measurement
     methods that are able to obtain bandwidth estimates in reasonable time
     without consuming too much resources of the network or on the probe-
     packet sender and receiver nodes. It is also important that new methods
     minimize the time between the actual measurement and presentation of
     the bandwidth estimate to the application. Old estimates are outdated
     since the network conditions may have changed.
   • It is important to understand whether bandwidth estimation methods re-
     port correct values in wireless networks, such as 802.11 or 3G. For many
     of the bandwidth estimation methods it is assumed that the forwarding
     and queuing mechanism in the routers is first-in-first-out based. What
12      Chapter 1. Introduction


       happens in the case where packets are assigned priority based on source
       or destination address or application type? Do bandwidth estimation
       methods still report a valid estimate?

     • It is important to study how the probe packets injected by the avail-
       able bandwidth measurement methods that exist today affect other traffic
       flows on the network. Is there a difference between how the probe pack-
       ets affect UDP traffic flows compared to flows mainly consisting of TCP?
       Is it possible to inject the probe packets in a way that minimizes the im-
       pact on other traffic flows while at the same time keeps the accuracy of
       the measurement samples?

    Even though the active end-to-end bandwidth measurement research area
has matured over recent years there still exist many important research chal-
lenges to be addressed by researchers in the future. These are presented in
Chapter 4.


1.4 Research methodology
To produce the research results presented in this thesis two main research
methodologies have been used. Analytical methodology has been the primary
tool when modeling dependencies while experimental methodology has been
applied when analytical verification is infeasible due to the complexity of the
problem.
    The primary task of experimental research methodology is to study the re-
lation between cause and effect. A set of hypotheses regarding a system or
a phenomenon are tested by conducting experiments. Thereafter conclusions
are drawn which in turn will lead to new hypotheses or input to the analyti-
cal methodology. In experimental research the effect of single variables are
isolated and thus the impact of that variable can be understood and perhaps
modeled. In the available bandwidth measurement research area it is for ex-
ample interesting to study the impact of the cross-traffic distribution on the
available bandwidth estimate produced by different methods.
    The experiments presented and analyzed in this thesis have been performed
over real Internet paths, in testbed laboratories or in network simulation tools.
Experimental results obtained from laboratories and simulation tools are pos-
sible to verify since all variables are known. However, experiments performed
over real Internet paths are harder to verify due to the fact that it is a non-trivial
task for researchers to gain access to the core Internet.
                                               1.5 My contributions        13


    Another aspect of experimental research methodology is the reproducibil-
ity of the study. By conducting experiments in network simulations and in
testbeds, and then documenting all parameters, the experiments can be re-
peated with similar results. Experiments in real networks, such as the Inter-
net, are harder to reproduce since it is for example impossible to control and
re-produce the cross-traffic conditions at a given time.


1.5 My contributions
The main scientific contributions of the work presented in this thesis, con-
ducted by myself and in collaboration with others, are the following:

   • A model describing how bandwidth measurement methods are affected
     by the properties of wireless 802.11 networks. The model is an extension
     of the TOPP model [3] (an available bandwidth measurement model).
     The extended model describes the dependencies between available band-
     width, link capacity estimates, probe-packet size and cross-traffic rate.

   • The parameters in the Kalman filter used by BART [1] (a bandwidth
     measurement method) have been tuned to increase the accuracy of the
     estimates, when the bottleneck is a wireless 802.11 link. The parameter
     configuration is found to be different when measuring in wireless net-
     works compared to wired networks. Further, the BART method has also
     been evaluated in both wired testbed and Internet scenarios. The results
     show that BART reports accurate bandwidth estimates.

   • A study on how different active end-to-end available bandwidth mea-
     surement methods affect other traffic flows. In a simulation based study
     it is shown that the impact of probe packets on TCP flows is limited even
     though the probe packets may increase the round-trip time and the packet
     loss experienced by the TCP flows.

   Other results include:

   ◦ A framework describing the probe-packet train and cross-traffic packet
     interactions at the discrete packet level has been developed.

   ◦ Identification of problems concerning the send mechanism in Linux of
     probe packets when injected into a wireless 802.11b network.
14      Chapter 1. Introduction


     ◦ Implementation of two bandwidth measurement methods, DietTOPP and
       BART, in C/C++ for UNIX based systems.
     ◦ Construction of a testbed network consisting of both wired single-access
       and wireless multi-access links.
Chapter 2

Summary of the papers

The second part of this thesis is a collection of five papers; paper A, B, C, D and
E. Papers A - D have been published in peer-reviewed conference or workshop
proceedings while paper E has not yet been published. This chapter gives a
short summary of each paper along with editorial comments and comments on
my contribution. The editorial comments are intended as correction of errors
and to simplify for the reader of the papers.
    Papers A and D focus on the interaction between probe packets and cross-
traffic packets. Paper B reports on a study on how wireless 802.11 bottle-
neck link properties affect available bandwidth estimates. Paper C presents
a bandwidth measurement method named Bandwidth Available in Real Time
(BART), along with an evaluation of the method in various settings. Paper E is
a continuation of papers B and C. The foci of paper E are on general problems
concerning measurements of available bandwidth in wireless networks and on
tuning of parameters in BART in order to increase accuracy of the estimates,
when the bottleneck is a wireless 802.11 link.


2.1 Paper A: On the Analysis of Packet-Train Prob-
    ing Schemes
                                             o
Andreas Johnsson, Bob Melander and Mats Bj¨ rkman, ”On the Analysis of
Packet-Train Probing Schemes”, In proceedings to the International Confer-
ence on Communication in Computing, Special Session on Network Simula-
tion and Performance Analysis, Las Vegas, USA, June 2004.

                                       15
16     Chapter 2. Summary of the papers


    Summary: This paper describes probe packet and cross-traffic packet in-
teractions at the discrete packet level. Three main interaction-types are iden-
tified; mirror patterns, chain patterns and quantification patterns, respectively.
Using the pattern terminology, the difference between using the statistical mean
and median operators to reduce noise in the measurements are explored.
    My contribution: I have written most of the paper, co-developed the mea-
surement tool and performed all experiments.


2.2 Paper B: An Analysis of Active End-to-end Band-
    width Measurements in Wireless Networks
                             o
Andreas Johnsson, Mats Bj¨ rkman and Bob Melander, ”An Analysis of Active
End-to-end Bandwidth Measurements in Wireless Networks”, In proceedings
to the 4th IEEE/IFIP End-to-end Monitoring Techniques and Services work-
shop, Vancouver, Canada, April 2006.
    Summary: In this paper a study on how the properties of wireless 802.11
bottleneck links affect bandwidth estimates (i.e. both available bandwidth and
link capacity) produced by methods related to the TOPP model is presented.
The results from experiments performed in a testbed environment illustrate that
the probe-packet size is crucial to the estimate of the available bandwidth. The
estimates decrease with decreasing probe-packet size. From the experimen-
tal results it is also evident that the estimated link capacity, using the TOPP
model, is dependent on the probe-packet size and the cross-traffic rate. The
reason for the dependency between bandwidth estimates, probe-packet size and
cross-traffic rate is due to the link-layer mechanisms in 802.11 networks. This
observation was developed into an extension of the TOPP model that describes
the dependencies in detail.
    Editorial comments: In this paper there is no differentiation made be-
tween cross traffic sent by other nodes in the wireless network and cross traffic
injected by the probe-packet sender (which usually is zero during a measure-
ment session). This should however not affect the overall understanding of
the extended TOPP model. The cross traffic that affect the function Tk (x), in
Equation 6.8 in the paper, originates from other nodes using the same wireless
link while the ”single” x corresponds to the cross traffic generated at the node
injecting the probe packets. In paper E this is noted.
    My contribution: I have written most of the paper, co-analyzed the experi-
mental results, conducted all measurements and implemented the measurement
tool DietTopp.
           2.3 Paper C: Real-time Measurement of End-to-End Available
                               Bandwidth Using Kalman Filtering   17

2.3 Paper C: Real-time Measurement of End-to-
    End Available Bandwidth Using Kalman Fil-
    tering

Svante Ekelin, Martin Nilsson, Erik Hartikainen, Andreas Johnsson, Jan-Erik
  a                                o
M˚ ngs, Bob Melander and Mats Bj¨ rkman, ”Real-time Measurement of End-
to-End Available Bandwidth Using Kalman Filtering”, In proceedings to the
10th IEEE/IFIP Network Operations and Management Symposium, Vancou-
ver, Canada, April 2006.
    Summary: This paper presents the BART method for measuring and esti-
mating the system state of an end-to-end path. From the system state both the
available bandwidth and the link capacity of the bottleneck link can be derived.
BART, which is an offspring of the TOPP method, makes use of a Kalman filter
in order to track system state changes in real time. The measurement model
BART makes use of is piecewise linear. Since the Kalman filters require a
linear model, it is also discussed in the paper how to overcome this limitation.
    In the paper an evaluation of the BART method in testbed networks as well
as over real Internet paths is presented. The evaluation shows that BART is
very accurate in most cases. Since BART estimates the available bandwidth in
real time, results from experiments where the available bandwidth varies are
also discussed. BART reacts quickly to the changes in available bandwidth.
The accuracy of BART is also compared to results obtained by pathChirp. The
impact of the probe-packet size and the probe-packet train length is briefly
studied. Large probe packets in combination with long probe-packet trains is
the best choice in order to get accurate results when performing the experi-
ments discussed in this paper.
    Editorial comments: BART make use of the TOPP model, which de-
scribes the relation between the offered probe rate o (probe rate at the sender
side) and the measured probe rate m (probe rate measured at the receiver). In
this paper the variable names has changed to u for the offered rate and r for the
measured rate at the receiver.
    My contribution: I have written parts of the evaluation section, partici-
pated in discussions on how to evaluate BART, co-designed evaluation scenar-
ios and participated in conducting measurements. I have also implemented the
measurement method into a tool used in the evaluation process.
18     Chapter 2. Summary of the papers


2.4 Paper D: Measuring the Impact of Active Prob-
    ing on TCP
                                 o
Andreas Johnsson and Mats Bj¨ rkman, ”Measuring the Impact of Active Prob-
ing on TCP”, In proceedings to the SCS International Symposium on Perfor-
mance Evaluation of Computer and Telecommunication Systems (SPECTS’06),
Calgary, Canada, July 2006.
    Summary: This paper discusses the impact of active probing, such as the
impact of probe packets injected by available bandwidth measurement meth-
ods, on TCP. In the paper, simulation results show that the available bandwidth
on a link in an end-to-end path actually can increase when injecting probe
packets into a path even though the probe packets add traffic to the network,
if the cross traffic consists of TCP flows. This is because the probe-packet
trains may cause packet losses that in turn trigger the TCP backoff mecha-
nism. Thus, the overall TCP throughput is reduced. To investigate whether it
is possible to lower the impact of the probe packets on the TCP flows, different
probe-packet train lengths were used when measuring the available bandwidth
in order to study the impact on the TCP flows. The number of samples, that is
the number of probe-packet separations per time unit, was however constant.
The simulation results show that probe-packet trains of ”medium length” is
preferable. Short probe-packet trains increase the TCP round-trip time esti-
mate while long probe-packet trains increase TCP loss. Both have negative
effect on TCP throughput.
    In the paper it is noted that the overall impact of the probe packets on TCP
throughput is rather low.
    My contribution: I have written most of the paper, co-analyzed the simu-
lation results, constructed the simulation setup and performed all simulations.


2.5 Paper E: On measuring the available band-
    width in wireless 802.11b networks
                               o
Andreas Johnsson and Mats Bj¨ rkman, ”On measuring the available bandwidth
in wireless 802.11b networks”, Not yet published.
    Summary: In this paper it is further studied how the properties of wire-
less 802.11 networks affect bandwidth measurement methods. Two problems
related to the operating system drivers are identified. These problems can bias
the measurement accuracy of available bandwidth and link capacity. In the pa-
 2.5 Paper E: On measuring the available bandwidth in wireless 802.11b
                                                      networks      19


per the accuracy of the BART method is also investigated, when measuring in
wireless 802.11 networks.
    It is possible to tune the Kalman filter used by BART and thereby to give
different weight to the new measurement sample on the current estimate of
the available bandwidth. In this paper the BART method is tuned in order
to increase the accuracy when measuring in networks where the bottleneck is
a wireless 802.11 network. It is shown that the configuration of the tunable
parameters is different from the configuration used in wired single-access net-
works.
    My contribution: I have written most of the paper, co-analyzed the exper-
imental results and performed all measurements.
Chapter 3

Related work

Much work has been published in the active end-to-end bandwidth estimation
research area in recent years, and the area has started to mature. Many dif-
ferent available bandwidth estimation methods and tools have been developed
and each new method tries to address new aspects of measuring available band-
width. Hence, each new method contributes to the common understanding on
how to perform active end-to-end available bandwidth measurements.

    The problem of measuring the end-to-end available bandwidth is complex
because there are many theoretical issues as well as real-world problems that
must be considered, such as the impact of the cross-traffic distribution on the
probe packets, or clock tick resolution issues on the sender and receiver side.
The methods differ in the design, how the probe packets are injected and how
the analysis is performed, and thus makes different assumptions and address
different challenges regarding active end-to-end measurements. This leads to
the conclusion that there may not be one available bandwidth measurement
method that is optimal in all scenarios. Rather, the method to be applied de-
pends on the demands of the application using the measurement method or on
what is known about the characteristics of the end-to-end network path.

    This chapter reviews important literature related to this thesis. Several
methods have been developed for actively measuring the end-to-end available
bandwidth. These are mentioned in Section 3.1. In Section 3.2 work of more
theoretical nature is presented.

                                      21
22     Chapter 3. Related work


3.1 Methods and tools
Well known active end-to-end bandwidth measurement methods are for exam-
ple Pathchirp [4], Pathload [5], Spruce [6] and TOPP [3]. Among newly devel-
oped methods, ABget [7] and BART [1] can be mentioned. The above methods
can be classified in two categories: direct probing and iterative probing [8].
    In direct probing, each injection of probe packets (as pairs or in longer se-
quences as trains) gives a sample of the available bandwidth. Spruce is one
such method. As other methods that rely on direct probing, Spruce needs
knowledge of the bottleneck link capacity to obtain estimates of the available
bandwidth. By comparing the input rate and the output rate, the cross-traffic
rate can be measured. Then, if the link capacity is known the available band-
width is calculated as the link capacity minus the cross-traffic rate. The advan-
tage of direct probing is that one probe-packet train can be enough to determine
the available bandwidth.
    In iterative probing no knowledge of the end-to-end path is needed. Instead,
these methods only rely on self-induced congestion. The idea is the following:
inject a set of probe packets (e.g. in pairs or in trains) with a predefined initial
probe rate into a network. If the initial probe rate is higher than the available
bandwidth, the probe packets will be queued after each other at the bottleneck
and cause congestion for a short time interval. In such a case, the mean separa-
tion between the probe packets will increase, which is equivalent to a decrease
in the received probe rate. The increase in the probe-packet separation is pro-
portional to the amount of cross traffic. However, if the initial probe rate is
equal to the received probe rate it is assumed that the packets did not have to
queue and thus the end-to-end path is not congested. That is, the initial probe
rate is less than the available bandwidth. By sending several probe packets with
different input rates and then applying an analysis method, the available band-
width is located. There exist several methods that exploit this feature; such as
ABget, BART, Pathchirp, Pathload and TOPP.
    In the following paragraphs, descriptions of a selection of active end-to-end
bandwidth estimation methods are given. A more in-depth review of available
bandwidth measurement theory, methods and tools can be found in [9]. In
[10] the problems of comparing different available bandwidth measurement
methods are discussed. Further, in [11] a comparison of public end-to-end
available bandwidth estimation methods on high-speed links is presented.

Spruce Spruce [6] is an example of direct probing, which assumes that the
capacity of the bottleneck is known. Spruce injects probe packets with sepa-
                                                 3.1 Methods and tools         23




            Figure 3.1: The rate-response curve obtained by TOPP.


ration ∆in and measures the separation ∆out on the receiver side. Then the
rate of the cross traffic during the network path traversal of the probe packets is
∆out −∆in
    ∆in    C where C is the capacity of the bottleneck link. Then the available
bandwidth B is
                                    ∆out − ∆in
                          B = 1−                 C.                      (3.1)
                                        ∆in
   In [12] it was shown that Spruce and other methods that rely on direct
probing tend to underestimate the available bandwidth in the general case.

TOPP TOPP [3] injects probe packets at rate o in an interval omin to omax
and measures the received rate m at the receiver node. If plotting o/m versus o
the rate-response curve is obtained, see Figure 3.1. If o/m = 1 then the probe
packets did not cause congestion on the network path. Otherwise, if o/m > 1
congestion has been induced into the network path and the rate-response curve
deviates from o/m = 1 in a linear fashion.
    After the measurement session TOPP deploys linear regression in order to
find the equation of the sloping segment. The available bandwidth is defined
as the intersection of the line o/m = 1 and the sloping segment, seen in Figure
3.1. That is, the point corresponding to an offered rate that just not congest
the network path. Having the equation of the sloping segment, the available
bandwidth is easily found. TOPP is explained in more detail in Part II of this
thesis.
24      Chapter 3. Related work


Pathload Pathload [5] is an implementation of the Self-Loading Periodic
Streams (SLoPS) methodology [5][13]. Using this methodology the end-to-
end available bandwidth is measured. The basic idea of SLoPS is explained
below.
     A sender transmits a packet train with a pre-defined probe rate to a receiver.
The sender time stamps the send time and the receiver time stamps each probe
packet when it arrives. The time stamp difference, defined as the one way delay
(OWD) is calculated for each packet (D1 , D2 , ..., DK ).
     If the initial rate is higher than the available bandwidth, the length of the
router queue grows when additional probe packets in the train are received to
the router. Due to this fact, the OWD values will have an increasing trend
(i.e. DK > DK−1 > ... > D1 ). When the initial probe rate is less than the
available bandwidth, the router queues will not grow as a result of the injection
of the probe train. Hence, the OWD will be more or less stable. Statistical tests
are used to determine whether the OWD values are stable or increasing.
     By sending probe trains at different pre-defined probe rates, the available
bandwidth is found by binary search. If the initial probe rate causes an in-
creasing OWD trend, the probe rate is above the available bandwidth. The
probe-train rate for the next train is then decreased. On the other hand, if the
probe train does not cause an increasing OWD trend, the initial probe rate is
less than the available bandwidth. In this case the probe rate is increased. This
process is iterated until a satisfactory accuracy of the available bandwidth is
obtained.

Pathchirp Pathchirp [4] is another tool that measures the end-to-end avail-
able bandwidth and exploits the iterative method. Instead of sending probe-
packet trains it sends chirps of probe packets. A chirp is essentially a packet
train, but the separation between the probe packets are exponentially decreas-
ing, dn = T γ n , d(n−1) = T γ (n−1) , ..., d(2) = T γ 2 , d1 = T γ 1 , where di is the
separation and T and γ are constants. By using chirps the end-to-end path is
probed at different initial probe rates using just one chirp.
    In the analysis the probe packet send time is compared to the queuing de-
lay that arises when the initial probe rate is above the available bandwidth.
The queuing delays are derived from comparing probe packet send and receive
times. In Figure 3.2 an example is shown. It illustrates the measurement results
obtained from one chirp. When the queuing delay is zero, the probe packets
were transmitted without queuing. During the excursions, shown in the figure,
the cross traffic is more intensive and hence causes queuing. The first few ex-
cursions go back to zero because the cross traffic is bursty (i.e. sometimes the
                                                3.1 Methods and tools          25




     Figure 3.2: Packet sending time versus queuing delay by Pathchirp.


cross traffic is absent during parts of a chirp). The last excursion in the figure
does not return to zero. This is when the send probe rate has exceeded the
bottleneck link capacity.
    By analyzing the excursions from one chirp it is possible to estimate the
end-to-end available bandwidth. By injecting several chirps over time, changes
in the available bandwidth can be tracked.

BART BART [1] injects probe packets at random rates in the interval umin to
umax . A system state vector describes the sloping segment of the rate-response
curve, similar to the TOPP model, see Figure 3.3. In BART the system state
is updated for each measurement sample using a Kalman filter. This way an
estimate of the available bandwidth is obtained in real time. The Kalman filter
tells how much to trust the new sample compared to the current estimate of the
available bandwidth.
     The update of the system state is illustrated in Figure 3.3. The system state
describes the solid line which results in an available bandwidth estimate B
(compare to the TOPP method). A new sample is obtained by injecting probe
packets into the network path. The system state changes due to the new sample,
and a new available bandwidth estimate Bx is produced. BART is described in
more detail in the second part of this thesis.

ABget ABget [7] relies on the same type of analysis as Pathload. The novel
thing about ABget is that it does not require a probe-packet sender and receiver.
Instead it makes use of a fake TCP client which instructs an ordinary TCP
server to send packets according to a scheme similar to sending probe-packet
26     Chapter 3. Related work




                              Figure 3.3: BART.



trains. The actual analysis of the received probe packets on the client side
is done using the ideas of increasing one-way delay during congestion, taken
from Pathload.




3.1.1 Link capacity estimation

There also exist methods for estimating the link capacity (not just the available
bandwidth) of the bottleneck link. Pathrate [14], TOPP and BART are three
examples. Pathrate looks for a capacity mode obtained from packet-pair prob-
ing while TOPP and BART produce the link capacity as a side effect when
estimating the available bandwidth.
     Pathneck [15] is a tool that uses ICMP packets and ordinary probe packets
to find the actual location of a bottleneck link in an end-to-end path. However,
it cannot estimate the link capacity or the available bandwidth.
   There exist a variety of older tools that for example uses ICMP and variable
packet sizes in order to estimate the link capacity, but these methods fail in
many cases [16].
                                                 3.2 Theoretical work        27


3.2 Theoretical work
Much theoretical research has been produced throughout the years. In this
subsection a short review of the literature is given.
    In [14] it was studied how probe-packet pairs and trains are affected by
cross traffic. Depending on the scheme used, different modes are visible in
histograms of probe-packet time separations. In this study the authors defined
one of the modes as the asymptotic dispersion rate, which was called the pro-
portional share in [17]. This value corresponds to the proportion of the link
capacity in use by the probe packets. In [14] a study of the impact of probe
packet size was also investigated. The findings lead forward to the link capac-
ity estimation method Pathrate.
    In [17][3] discussion about the proportional share, among other things, lead
forward to the TOPP model to measure the end-to-end available bandwidth and
link capacity.
    In [18] a delay-variation model for packet-pair like methods was developed.
Using this model several histogram signatures could be identified. The modes
in the histograms corresponded to bottleneck link and secondary bottleneck
link capacities. A secondary bottleneck link is a link that is congested when
further increasing the probe-packet rate, even though the first bottleneck link
is congested. The impact of the probe-packet size to link capacity estimation
methods was also discussed.
    The effect of layer-2 store-and-forward devices on per-hop capacity estima-
tion was studied in [16]. In this paper it was shown why many older methods
for estimating bottleneck link capacity reported erroneous estimates.
    A description on how the one-way delay within a probe-packet train varies
depending on different parameters such as cross traffic and train length was pre-
sented in [13]. By investigating the one-way delay the authors could conclude
that the one-way delay increased within a probe-packet train if the send rate
was above the available bandwidth. Otherwise the one-way delay remained
constant. A method for detecting increasing trends was developed and resulted
in the tool Pathload for estimating available bandwidth.
    In [19][20][21] the rate response curve (seen in e.g. Figure 3.1) was in-
vestigated in-depth. In this work a more comprehensive mathematical model
for describing the cross-traffic effect on the rate response curve was presented.
However, it has been shown by experiments that simpler models (used by e.g.
TOPP and BART) for describing the rate response curve give good estimates
of the available bandwidth as well [1][22].
    Sometimes the available bandwidth itself is not sufficient for an applica-
28     Chapter 3. Related work


tion. In [23] a study of how to measure the available bandwidth variation range
is presented. The available bandwidth is a mean value over some time and the
variation range describes how much the cross traffic fluctuates during this in-
terval. The variation range is defined using second order statistics such as the
variance. That method was implemented in a tool called Pathvar.
    Initial work on how broadband access links, such as 802.11 and ADSL,
affect bandwidth measurements obtained from different measurement methods
was presented in [24]. Here they showed how the probe-packet size affects the
available bandwidth estimates obtained from a set of tools.
    The above literature survey, together with the presentation of a selection of
bandwidth measurement methods, covers much of the important research made
in this area. The papers included in part two of this thesis further develop the
research area as described in Section 1.5.
Chapter 4

Conclusions and future
research challenges

In this thesis methods for actively measuring the end-to-end available band-
width have been studied. An evaluation of BART has been presented along
with a detailed study of how 802.11 link properties affect available bandwidth
estimates produced by methods such as BART and TOPP. The estimates are
dependent on the probe-packet size, contrary to what is seen in wired single-
access networks.

    Further, the Kalman filter deployed by BART has been tuned to increase the
accuracy of link capacity and available bandwidth estimates when measuring
in wireless networks. The parameter configuration was found to be different
compared to the one used in single-access wired networks.

    In the thesis it has also been studied in detail how the probe packets and
the cross-traffic packets interact with each other. For example, it has been
investigated how the probe packets affect TCP flows that share a network path.
Depending on the probe-packet train length, either the TCP round-trip time or
the TCP loss rate increase, which both cause decreased TCP performance. By
simulations it was shown that the combined effect was minimized when the
probe-packet sender injected ”medium length” probe-packet trains.

                                     29
30      Chapter 4. Conclusions and future research challenges


    This thesis has given answers to some important research questions men-
tioned in Chapter 1, but many challenges remain. Some future research chal-
lenges not addressed in the thesis, or where more research is needed, are sum-
marized in the list below.

     • Today it is possible to measure the end-to-end available bandwidth. In
       the future it is likely that a more complete picture of the network status
       is needed. How are active network tomography measurement methods
       supposed to function? Can such methods be used for bandwidth predic-
       tion for a single path, including not only bandwidth estimation but also
       bandwidth prediction?
     • For many of the bandwidth estimation methods it is assumed that the
       forwarding mechanism in the routers is first-in-first-out based. What
       happens in the case where packets are assigned priorities based on source
       or destination address or application type? Do bandwidth estimation
       methods still report valid estimates? It is also important to understand
       whether bandwidth estimation methods report correct values in wireless
       networks, such as 802.11 or 3G.
     • A next step is also to integrate bandwidth estimation methods with suit-
       able applications such as streaming media tools, transport protocols or
       server selection algorithms.
     • It would be interesting to investigate whether it is possible to send pack-
       ets related to the application as probe packets. In that way the total load
       on the network is decreased. For example, can the packets from a stream-
       ing media application be injected in a measurement-like fashion?
Bibliography

                                                             a
[1] S. Ekelin, M. Nilsson, E. Hartikainen, A. Johnsson, J. M˚ ngs, B. Me-
                     o
    lander, and M. Bj¨ rkman. Real-time measurement of end-to-end avail-
    able bandwidth using kalman filtering. In Proceedings of the IEEE/IFIP
    Network Operations and Management Symposium, Vancouver, Canada,
    2006.

[2] Yongmin Choi, Heung-No Lee, and Anurag Garg. Measurement and
    analysis of wide area network (wan) traffic. In Proceedings of the SCS
    Symposium on Performance Evaluation of Computer and Telecommuni-
    cation Systems, Vancouver, Canada, July 2000.

                          o
[3] Bob Melander, Mats Bj¨ rkman, and Per Gunningberg. Regression-based
    available bandwidth measurements. In Proceedings of the SCS Sympo-
    sium on Performance Evaluation of Computer and Telecommunications
    Systems, San Diego, USA, July 2002.

[4] V. Ribeiro, R. Riedi, R. Baraniuk, J. Navratil, and L. Cottrel. pathChirp:
    Efficient available bandwidth estimation for network paths. In Proceed-
    ings of the Passive and Active Measurement Workshop, San Diego, USA,
    2003.

[5] Manish Jain and Constantinos Dovrolis. Pathload: a measurement tool
    for end-to-end available bandwidth. In Proceedings of the Passive and
    Active Measurement Workshop, Ft Collins, USA, March 2002.

[6] Strauss, Katabi, and Kaashoek. A measurement study of available band-
    width estimation tools. In Proceedings of the ACM SIGCOMM Internet
    Measurement Workshop, Miami, USA, 2003.

                                     31
32     Bibliography


 [7] Demetres Antoniades, Manos Athanatos, Antonis Padadogiannakis,
     Evangelos P. Markatos, and Constantine Dovrolis. Available bandwidth
     measurement as simple as running wget. In Proceedings of the Passive
     and active measurement workshop, Adelaide, Australia, 2006.

 [8] Manish Jain and Constantinos Dovrolis. Ten fallacies and pitfalls in end-
     to-end available bandwidth estimation. In Proceedings of the ACM SIG-
     COMM Internet Measurement Conference, Sicily, Italy, October 2004.

 [9] R.S. Prasad, M. Murray, C. Dovrolis, and K. Claffy. Bandwidth estima-
     tion: metrics, measurement techniques, and tools. IEEE Network Maga-
     zine, 2003.

[10] Frederico Montesino-Pouzols. Comparative analysis of active bandwidth
     estimation tools. In Proceedings of the Passive and Active Measurement
     Workshop, Antibes Juan-les-Pins, France, 2004.

[11] A. Shriram, M. Murray, Y. Hyun, N. Brownlee, A. Broido, M. Fomenkov,
     and K. Claffy. Comparison of public end-to-end bandwidth estimation
     tools on high-speed links. In Proceedings of the Passive and Active Mea-
     surement workshop, Boston, USA, 2005.

[12] Li Lao, Constantinos Dovrolis, and M.Y. Sanadidi. The probe gap model
     can underestimate the available bandwidth of multihop paths. In ACM
     SIGCOMM Computer Communications Review, October 2006.

[13] Manish Jain and Constantinos Dovrolis. End-to-end available bandwidth:
     Measurement methodology, dynamics, and relation with TCP throughput.
     In Proceedings of the ACM SIGCOMM conference, Pittsburg, USA, Au-
     gust 2002.

[14] Constantinos Dovrolis, Parameswaran Ramanathan, and David Moore.
     What do packet dispersion techniques measure? In Proceedings of the
     IEEE Infocom conference, Anchorage, USA, April 2001.

[15] Ningning Hu, Li (Erran) Li, Zhuoqing Morley Mao, Peter Steenkiste, and
     Jia Wang. Locating internet bottlenecks: algorithms, measurements, and
     implications. In ACM SIGCOMM: Proceedings of the 2004 conference
     on Applications, technologies, architectures, and protocols for computer
     communications, Portland, USA, 2004.
[16] Ravi S. Prasad, Constantinos Dovrolis, and Bruce A. Mah. The effect of
     layer-2 store-and-forward devices on per-hop capacity estimation. In Pro-
     ceedings of the IEEE Infocom conference, New York, USA, April 2002.
                           o
[17] Bob Melander, Mats Bj¨ rkman, and Per Gunningberg. A new end-to-end
     probing and analysis method for estimating bandwidth bottlenecks. In
     Proceedings of IEEE Globcom, San Francisco, November 2000.
              a
[18] Attila P´ sztor and Darryl Veitch. The packet size dependence of packet
     pair like methods. In Tenth International Workshop on Quality of Service,
     Miami Beach, USA, May 2002.
[19] Xiliang Liu, Kaliappa Ravindran, and Dmitri Loguinov. Single-hop prob-
     ing asymptotics in available bandwidth estimation: Sample-path analysis.
     In Proceedings of the ACM SIGCOMM Internet Measurement Confer-
     ence, Taormina, Italy, 2004.
[20] Xiliang Liu, Kaliappa Ravindran, and Dmitri Loguinov. Multi-hop prob-
     ing asymptotics in available bandwidth estimation: Stochastic analysis. In
     Proceedings of the ACM SIGCOMM Internet Measurement Conference,
     Berkeley, USA, 2005.
[21] Xiliang Liu, Kaliappa Ravindran, and Dmitri Loguinov. Measuring prob-
     ing response curves over the ron testbed. In Proceedings of the Passive
     and Active Measurement workshop, Rio de Janeiro, Brazil, 2006.
                                                 o
[22] Andreas Johnsson, Bob Melander, and Mats Bj¨ rkman. Diettopp: A first
     implementation and evaluation of a new bandwidth measurement tool. In
     Proceedings of the Swedish National Computer Networking Workshop,
     Karlstad, Sweden, 2004.
[23] Manish Jain and Constantinos Dovrolis. End-to-end estimation of the
     available bandwidth variation range. In Proceedings of the ACM SIG-
     METRICS conference, Banff, Canada, 2005.
[24] Karthik Lakshminarayanan, Venkata N. Padmanabhan, and Jitendra pad-
     hye. Bandwidth estimation in broadband access networks. In Proceedings
     of the Internet Measurement Conference, Taormina, Italy, 2004.
      II

Included Papers




       35
Chapter 5

Paper A:
On the Analysis of
Packet-Train Probing
Schemes

                                              o
Andreas Johnsson, Bob Melander and Mats Bj¨ kman
In proceedings of the International Conference on Communication in Comput-
ing, Special Session on Network Simulation and Performance Analysis, Las
Vegas, USA, June 2004.




                                   37
                                   Abstract

With a better understanding of how probe packets and cross-traffic packets
interact with each other, more accurate measurement methods based on ac-
tive probing can be developed. Several existing measurement methods rely on
packet-train probing schemes. In this article, we study and describe the inter-
actions between probe packets and cross-traffic packets.
    When one packet within a packet train is delayed, the dispersion (i.e. packet
separation) of at least two (and possibly more) probe packets will change. Fur-
thermore, the dispersions are not independent, which may bias calculations
based on statistical operations. Many methods use dispersion averages, such as
the mean, in the calculation of bandwidth estimates and predictions.
    We describe cross traffic effects on packet trains. The interaction results
in mirror, chain and quantification patterns. Experiments have been performed
in a testbed to explore these patterns. In histograms of delay variations for
adjacent probe packets, these patterns are manifested as different identifiable
signatures.
    Finally, we also discuss the effect of these patterns on the mean and median
operations.
                                                       5.1 Introduction        39


5.1 Introduction

Measurement of the end-to-end available bandwidth of a network path is get-
ting increasingly important in the Internet. Verification of service level agree-
ments, streaming of audio/video flows, and Quality-of-Service management
are all examples of Internet activities that need or can benefit from measure-
ments of available bandwidth.
    Many methods that attempt to measure end-to-end bandwidth actively probe
the network path by injecting probe packets in predetermined flight patterns.
Common flight patterns include pairs of probe packets, so called packet-pair
probing schemes and its extension into longer sequences of probe packets
[1, 2, 3, 4, 5, 6], which we will refer to as packet-train probing schemes.
    When the probe packets (independent of probing scheme) traverse the net-
work path, the dispersion between successive probe packets will change. This
is due to limited link capacity and interactions with other packets traversing the
same path (so called cross-traffic packets).
    To calculate an available bandwidth estimate an analysis of the dispersion
values is made. The analysis relies on the probe packet dispersion at the sender
side in combination with the probe packet dispersion at the receiver side.
    In this article we describe the probe packet and cross-traffic packet inter-
actions when using packet-train probing schemes. The interactions are mani-
fested as patters. Further, we illustrate these patterns as identifiable signatures
in histograms.
   We describe the effect of the packet interactions when using mean and
median operations to dispersion values obtained from packet-train probing
schemes.
    All dispersion-based measurements has been performed in a testbed, which
is described in the article.
    The rest of this article is organized as follows: Section 5.2 describes three
patterns that occur when probe and cross-traffic packets interact with each
other. Section 9.1 defines a testbed that we have used for all of our measure-
ments. Section 5.4 identifies four signatures that arise from the three patterns.
Section 5.5 describes the impact of mean and median filtering when perform-
ing analysis of dispersion values obtained from packet-train probing schemes.
The article ends with conclusions in Section 9.6.
40     Paper A


5.2 Description of patterns
This section describes three patterns that arise when probing a network path us-
ing packet-train probing schemes. The patterns are described using a multiple-
hop model for route delay variation [7]. The concepts of that model is described
in the following subsection, while the patterns are described in subsections
5.2.2 - 5.2.4.


5.2.1 A multiple-hop model for route delay variation
This subsection describes the concepts of a multiple-hop model for route de-
lay variation [7]. We use this model to describe the identified patterns in the
following subsections.
    In what follows, the definition of a hop is one router, its in-queue, and the
outgoing link used by the packets. Hence, the arrival time of an arbitrary packet
to hop h + 1 is equal to the departure time from the previous hop h.
    A packet Pi arrives to a hop h at time τi . After a queuing time wi ≥ 0
the packet begins its service time xi > 0. Packet Pi leaves the hop at time τi∗ .
Thus, the one-hop delay for packet Pi is

                        di ≡ τi∗ − τi = wi + xi + D,                        (5.1)

where D is the link propagation delay, which is equal for all equally-sized
packets traveling on the same link.
    From Equation (5.1), a set of equivalences to compare two adjacent packets
are derived:

                inter-packet arrival time: ti   ≡   τi − τi−1
             inter-packet departure time: t∗i   ≡   τi∗ − τi−1
                                                           ∗

                          delay variation: δi   ≡   di − di−1
                                                =   t∗ − ti
                                                     i
                                                =   (xi − xi−1 ) +
                                                    (wi − wi−1 ).

   The waiting time of a packet within an infinite FIFO buffer is described by
Lindley’s equation

                     wi = max(0, wi−1 + xi − ti ) + ci ,                    (5.2)
                                                      5.2 Description of patterns     41


ci corresponds to the waiting time caused by cross traffic entering the hop
between τi−1 and τi .
     A router queue can in principle operate in two states - busy and idle state.
The busy state implies that the router is constantly forwarding packets from its
in-queue, while in the idle state the in-queue is empty.
     Probe packets (i.e. packets used for obtaining dispersion values) can conse-
quently be divided into two categories, Initial and Busy probe packets (adapt-
ing to the notation in [7]). The first packet of a busy period is by definition an
initial probe packet. That is, an I probe packet is never queued behind another
probe packet (i.e. wi of Equation (5.2) is equal to 0 + ci ). B probe packets are
packets that are queued behind other probe packets.
     With this categorization of probe packets, the delay variation δi is defined
with respect to whether a probe packet Pi is I or B. From [7] we have

                   I:      δi     = (xi − xi−1 ) + (wi − wi−1 )                     (5.3)
                   B:      δi     = (xi − ti ) + ci                                 (5.4)

where Equation (5.4) is derived from Equations (5.2) and (5.3).
    Equations (5.3) and (5.4) are extended in [7] to describe multiple hops.
These extensions are based on the following statements: if a probe packet is
I or B at hop j and B at hop j + 1, δi is overwritten and replaced by δi from
Equation (5.4). On the other hand, if the probe packet is I at hop j + 1, the
right hand side of Equation (5.3) is added to the existing δi .
    Hence, a probe packet that traverses an H-hop path, being I at every hop,
has a delay variation
                           H                           H
                   δi =          (xh − xh ) +
                                   i    i−1
                                                              h    h
                                                            (wi − wi−1 ).           (5.5)
                          h=1                         h=1

   If a probe packet is B on at least one hop, it will be B for the last time at
some hop in the path. Denote this hop si . The delay variation for such a probe
packet is
                                                        H
              δi    =     (xsi
                            i    − ti +   csi )
                                           i      +           (xh − xh ) +
                                                                i    i−1
                                                      h=si +1
                                 H
                                       h    h
                          +          (wi − wi−1 ).                                  (5.6)
                              h=si +1
42     Paper A


5.2.2 Mirror pattern

In the following we will describe the characteristics of the mirror pattern.
Hence, consider a probe packet train, containing at least three probe packets
Pi−1 , Pi and Pi+1 that are all I at hop h. Further, assume that Pi−1 and Pi+1
are unaffected by cross traffic (i.e. wi−1 = wi+1 = 0). Then, if Pi is delayed,
wi > 0, its delay variation δi > 0. Hence, Pi will have a delay variation
δi = (xi − xi−1 ) + (wi − wi−1 ) = (wi − wi−1 ), under the assumption of fix
size probe packets.
   Since both Pi−1 and Pi+1 are unaffected by cross traffic, the following
holds

                            δi+1     =     wi+1 − wi
                                     =     −wi
                              δi     =     wi − wi−1
                                    =      wi
                                   =⇒
                            δi+1     =     −δi ,                             (5.7)


    which we define as a perfect mirror pattern. An example of this phe-
nomenon is shown in Figure 5.1. The vertical packets above the time line shows
when in time a probe packet (white box) or a cross-traffic packet (shaded box)
arrives to the hop. The arc indicates when in time all bits of the packet have
been received to the router. When all bits have been received, the router trans-
mit the packet on the outgoing link, if it is not delayed by another packet. The
transmission from the router is shown below the time line in the same manner
as above the time line. The horizontal packets describe the packet pattern on
the out-going link. Probe packet Pi is delayed wi time units, visualized by the
horizontal arrow next to Pi in Figure 5.1. Since Pi−1 and Pi+1 are unaffected
by cross traffic Equation (5.7) holds and we have a perfect mirror pattern.
     In addition to the fact that probe packet Pi can be delayed, there is a possi-
bility that one, or both of Pi−1 and Pi+1 are affected by cross traffic. This will
cause changes to the mirror pattern as described below. It is obvious that this
possibility grows with increasing cross traffic and/or increasing probe rate.
   Assume, for instance, that both Pi and Pi+1 are delayed by cross traffic,
while Pi−1 and Pi+2 are unaffected. Then the mirror pattern is divided in a
                                                                      5.2 Description of patterns                           43
    P i−1




                                                                                                    Packet arrival




                                                                               P i+1
                                            Pi




                                                                      w
                                                                          i
                                                                                                                     Time




                                                                                                     P i+1
                                                                 Pi
                      P i−1




   Packet departure



                              P i−1                                                    Pi                    P i+1

                                      t*                                                    t*
                                        i                                                     i+1


                                                 Packet pattern on outgoing link


Figure 5.1: Arrival and departure times for cross traffic (shaded boxes) and
probe packets (white boxes) entering a hop. The cross-traffic packet delays
probe Pi in such a way that a mirror pattern arises.



predictable way. That is,


                                             δi      =      wi − wi−1 = wi
                                       δi+1          =      wi+1 − wi


since wi+1 > 0. Now, the next packet in the train, Pi+2 , will have a delay
variation


                                            δi+2        =      wi+2 − wi+1
                                                        =      −wi+1


since Pi+2 has a waiting time wi+2 = 0. Hence, −δi = δi+2 + δi+1 .
    To generalize, assume that Pi and the following (n − 1) probe packets are
delayed by cross traffic (not necessarily at the same hop), then we have a chain
of divided mirror patterns. Their delay variations relate to each other in the
44     Paper A


following way:

              δi+n + · · · + δi+1     =    (wi+n − wi+(n−1) )
                                           + · · · + (wi+2 − wi+1 )
                                           +(wi+1 − wi )
                                      =    −wi
                                      =    −δi                               (5.8)
                                      =⇒
                           i+n
                                 δa   =    0                                 (5.9)
                           a=i

since wi+n = 0. That is, cross-traffic effects on packet trains will cancel out, to
a certain degree, and hence not affect the mean value of the packet dispersion
values (nor δ-values) obtained from the probe-packet train.
    Mirror patterns are erased if probe packets in the packet train are trans-
formed from I to B, as described by Equations (5.5) and (5.6).
    When a probe packet train traverses an H-hop path, the delay variation
of every probe packet is described by Equation (5.5) or (5.6) depending on
whether the probe packets ever become B. We have extended the model pre-
sented in [7] to describe the relation between delay variation values obtained
from packet-train probing.


5.2.3 Chain pattern
In this section we will describe the chain pattern. If a cross-traffic packet delays
a probe packet, Pi−1 , in such a way that at least Pi and Pi+1 are transformed
from I to B, and makes the involved probe packets Pi−1 , Pi and Pi+1 back-to-
back after the hop, a chain pattern is visible.
    This is the definition of a pure chain pattern. If other probe packets within
the scope of the chain pattern are delayed by cross traffic, a quantification pat-
tern will arise. Quantification patterns are described in Section 5.2.4.
    An example of a chain pattern is shown in Figure 5.2, which is similar to
Figure 5.1. When Pi−1 is received, it must wait for the cross-traffic packet to
complete its departure. The waiting time of Pi−1 is wi−1 , shown in Figure 5.2.
Pi−1 is transmitted back-to-back behind the cross-traffic packet. Pi−1 is in this
example by definition I since it does not have to queue behind any other probe
packet.
                                                             5.2 Description of patterns                           45
            P i−1




                                                                                           Packet arrival




                                                         P i+1
                             Pi




                                  w
                                      i−1
                                                         w
                                                             i
                                                                                 w                          Time
                                                                                     i+1




                                                                         P i+1
   Packet departure
                         P i−1




                                                    Pi




                                            P i−1                P   i                     P i+1
                                            Packet pattern on outgoing link


Figure 5.2: Arrival and departure times for cross traffic (shaded boxes) and
probe packets (white boxes) entering a hop. The cross-traffic packet delays
Pi−1 in such a way that a chain pattern arises.


    During the waiting time of Pi−1 , the next probe packet Pi enters the router.
Pi has to wait wi time units in the queue for Pi−1 to complete its departure,
and is therefore B. After the waiting time, Pi is sent back-to-back behind Pi−1 .
The same procedure is repeated for Pi+1 .
    After the service time of Pi+1 has elapsed, Pi−1 , Pi and Pi+1 travel back-
to-back after each other on the link. Also, Pi and Pi+1 have been transformed
from I to B since both packets had to queue behind other probe packets. Hence,
a chain pattern is visible after the hop.
    The relation between delay variation values from probe packets involved in
a chain pattern can be described by Equation (5.8), similarly to mirror patterns.
The difference is that the mirror pattern involves I probe packets while the
chain pattern involves probe packets that change from I to B.
    Chain patterns are preserved to some degree in an H-hop path if the hop
where the patterns arise is the last hop where the probe packets are B. Of
course, this pattern is blurred by mirror and quantification patterns if there
are hops downstream hop si with cross-traffic. This is described by Equations
(5.5) and (5.6).


5.2.4 Quantification pattern
The last pattern identified in this paper is the quantification pattern. This pattern
is described below.
46         Paper A


    Let us assume that the probe packet generator is sending probe packets at a
high rate (i.e. the probe packet dispersion is less than the service time xCT of a
large cross-traffic packet). When a cross-traffic packet enters the router queue
between the arrival time of two probe packets the probe packets will become
separated by the service time xCT of that cross-traffic packet. Hence there
is no idle time gap in the router between the probe packets. This separation is
hereafter referred to as a quantification pattern. The term quantification is used
since the traffic consists of discrete transmissions, rather than a continuous flow
of bits.
                                Pi
              P i−1




                                                                                      Packet arrival
                                                       P i+1




                                w                       w                        w                      Time
                                     i−1                       i                     i+1
                                                                         P i+1




                                             w
                                              ct
     Packet departure
                        P i−1




                                                   i
                                                   P




                                           P i−1                          Pi                    P i+1

                                            Packet pattern on outgoing link


Figure 5.3: Arrival and departure times of cross traffic (shaded boxes) and
probe packets (white boxes) entering a router. The upper left cross-traffic
packet causes a chain pattern. The smaller cross-traffic packet causes a quan-
tification pattern.

     An example of the quantification pattern is shown in Figure 5.3, which is
similar to Figure 5.1. In this example we see that the leftmost cross-traffic
packet creates a chain pattern (equivalent to Figure 5.2). The second cross-
traffic packet entering the hop between probe packet Pi−1 and Pi will be sent
directly after Pi−1 , while Pi is sent back-to-back with the second cross-traffic
packet. Hence, Pi−1 has to wait wi−1 time units (stemming from the large
cross-traffic packet), while Pi has to wait wi time units (corresponding to the
small cross-traffic packet and a portion of the the big cross-traffic packet). That
is, a quantification pattern has arisen.
     The delay variation relation of values obtained from probe packets involved
in a quantification pattern can be described by Equation (5.8), similarly to mir-
ror patterns.
     Quantification patterns are preserved to some degree in an H-hop path if
                                                       5.3 Testbed setup         47


the hop where the pattern arise is the last hop where the probe packets are B.
This pattern is blurred if there are hops downstream hop si with cross-traffic
noise. This is described by Equations (5.5) and (5.6).


5.3 Testbed setup
The testbed network (see Figure 5.4) used in the experiments consists of three
router nodes, one Black Diamond (BD) and two Torrent routers (T1 and T2)
[8]. There are also a probing generator (PG), a probing receiver (PR), a traffic
generator (TG) called IP Traf Gen (internal product of Ericsson, www.ericsson.com)
and a traffic receiver (TR) which is an IXIA 1600 Traffic Generator/Analyzer.
The links are all 100 Mbps except between the three routers, where the links
are limited to 10 Mbps.

                                        TR




                            10Mbps            10Mbps
     PG               BD                T1               T2                PR




                                        TG



                            Figure 5.4: Testbed setup

    Depending on the experiment setup the cross traffic can enter and leave the
router chain at different positions. The cross traffic flow can be one or several
of the following: TG → T1 → T2 → TR (flow 1), TG → BD → T1 → TR
(flow 2) and TG → BD → T1 → T2 → TR (flow 3).
    All cross traffic is exponentially inter-packet spaced. Four different cross
traffic sizes where used: 64, 148, 482 and 1518 bytes. In the experiments all of
these could be used at the same time or just a selection of them. When all sizes
are used, 46% is of size 64, 11% of 148, 11% of 482 and 32% of size 1518
bytes. This distribution of packet sizes has its origin from findings in [9].
    The cross traffic intensity is variable within the testbed, in steps of arbitrary
size.
    The probe traffic is sent through the path PG → BD → T1 → T2 → PR.
48     Paper A


The probe packet size is 1500 bytes and consists of 32 packets. Normally 5
trains are sent per test run.


5.4 Signatures
When cross traffic affects a packet train, the train will suffer from different
patterns, as discussed in Section 5.2. In this section we illustrate these patterns
using real packet trains which interact with cross traffic. The patterns, alone
or in combination with each other, correspond to different signatures. The
signatures are described and illustrated in delay variation histograms below.
    Four signatures are defined and discussed in this section. They are: inde-
pendence, mirror, rate and quantification signatures. In [7] similar signatures
have been identifies for packet-pair probing schemes. However, we have shown
that packet-pair and packet-train probing schemes are fundamentally different
[10]. Thus, it is important to identify the signatures for packet trains as well.
Examples from the described testbed are shown.
    The probe rate for each diagram in Figure 5.5 is 1 Mbps in the upper left
diagram, 2.9 Mbps, 4.8 Mbps and 6.7 Mbps in the bottom right diagram. The
probe packets are 1500 bytes, and are sent in 5 trains consisting of 32 packets
in each. The cross traffic rate is 5 Mbps, exponentially distributed and consist
of 4 different packet sizes. The cross traffic uses flow 1 as described in Section
9.1.


5.4.1 The independence signature
The independence signature is visible in scenarios where there is no or very
little cross traffic interfering with the packet train. That is, no or very few
probe and cross traffic packet interactions.
     The signature arises from the fact that most probe packets traverse the net-
work path unaffected by cross traffic, thus independent of other traffic. This
means that the delay variation δ for such packets are near 0 as seen in the upper
left diagram in Figure 5.5. The peak at δ = 0 is called the independence peak.


5.4.2 The mirror signature
The mirror signature is a signature that arises due to the mirror pattern. If
there is very little cross traffic and the probe rate is relatively low, there will
arise mirror patterns in the values obtained from packet-train probing. That is,
                                                                      5.4 Signatures                  49


     50                                                    40

                                                           35
     40
                                                           30

     30                                                    25

                                                           20
 #




                                                       #
     20                                                    15

                                                           10
     10
                                                           5

     0                                                     0
     −6   −4   −2        0          2   4          6       −4   −2     0           2       4          6
                 delay variation (s)           −3
                                            x 10                     delay variation (s)          −3
                                                                                               x 10
     14                                                    25

     12
                                                           20
     10

                                                           15
     8
 #




                                                       #




     6
                                                           10

     4
                                                           5
     2

     0                                                     0
     −2   −1    0        1          2   3          4       −1    0     1           2       3          4
                 delay variation (s)           −3
                                            x 10                     delay variation (s)          −3
                                                                                               x 10



Figure 5.5: Using 4 packet sizes and cross traffic flow 1, with a rate of 5 Mbps.
The probe rate increases from approximately 1 Mbps (upper left) to 6.7 Mbps
(lower right).


for each positive delay variation value there is a corresponding negative delay
variation value.
     The upper right diagram of Figure 5.5 shows a distribution of delay varia-
tion values around the diminished independence peak (compared to the upper
left diagram).


5.4.3 The rate signature
The rate signature is a peak that arises from the fact that several probe packets
traverse the network path back-to-back, because of the chain patterns. The rate
signature corresponds to the link rate of the link creating the chain pattern.
    The rate peak grows in size when the probe rate increases, since more probe
50               Paper A


packets will travel back-to-back in such cases.
    The rate peak is visible in the bottom left diagram of Figure 5.5. The rate
peak is the leftmost peak. The delay variation value for that peak can be con-
verted to the link rate.

5.4.4 Quantification signature
The quantification signature arises from the quantification pattern described
above. The quantification signature corresponds to the gap between probe
packets. In the bottom right diagram of Figure 5.5 there are 3 clearly sepa-
rated regions of peaks. The leftmost peak corresponds to the rate signature.
    The locations of the quantification peaks depend on the size of the cross
traffic packets and the number of packets of each size that are interfering.


5.5 Mean and median analysis using patterns
An important step in the analysis phase is to reduce noise and effects stemming
from cross-traffic burstiness. One way to do this is to mean or median filter
the dispersion samples. It is therefore important to understand the impact and
properties of these statistical operations.
    In a previous simulation-based study [10] we have shown that the patterns
described in Section 5.2 affect the mean and median. However, they are af-
fected in different ways. These differences can be illustrated and explained in
terms of the packet interactions described as patterns in Section 5.2.
        2                                                               2




       1.8                                                             1.8




       1.6                                                             1.6
 o/m




                                                                 o/m




       1.4                                                             1.4




       1.2                                                             1.2




        1                                                               1




       0.8                                                             0.8
             2    3   4    5     6     7    8      9   10   11               2   3   4   5     6     7    8      9   10   11
                               Probe rate (Mbps)                                             Probe rate (Mbps)



Figure 5.6: Mean offered / measured probe rate to the left. Median offered /
measured probe rate to the right. Cross traffic = 3 Mbps on a 10 Mbps link.
                                   5.5 Mean and median analysis using patterns                                               51


        2                                                             2




       1.8                                                           1.8




       1.6                                                           1.6
 o/m




                                                               o/m
       1.4                                                           1.4




       1.2                                                           1.2




        1                                                             1




       0.8                                                           0.8
             2   3   4   5     6     7    8      9   10   11               2   3   4   5     6     7    8      9   10   11
                             Probe rate (Mbps)                                             Probe rate (Mbps)



Figure 5.7: Mean offered / measured probe rate to the left. Median offered /
measured probe rate to the right. Cross traffic = 5 Mbps on a 10 Mbps link.

     Figures 5.6 and 5.7 show measurements from two different scenarios with
increasing amounts of cross traffic. The topology is described in Section 9.1
with only cross traffic flow 1 being active. Both graphs in each figure show the
ratio between offered and measured dispersion rate on the y-axis. The x-axis
is the probe rate. In the left side graphs mean filtering is used whereas median
filtering is used in the right hand graphs. A ratio close to 1 corresponds to an
underload situation. In the part where a curve deviates from 1, the slope using
mean filtering is inversely proportional to the link capacity [5].
     From the graphs it is apparent that the mean and median filtering give dif-
ferent results. When the probe rate increases the number of probe and cross-
traffic packet interactions will also increase. These interactions will occur as
mirror patterns followed in turn by chain and quantification patterns. In the
case of mean filtering, the asymmetry effects of the mirror and the chain pat-
terns will be cancelled. Hence, they will have no effect on the offered / mea-
sured ratio. The quantification patterns that arise when the link is full (i.e.
when the available bandwidth is reached) cause the slope of the curves seen in
the graphs regardless if mean or median filtering is used.
     The symmetry effect of the mirror pattern is also cancelled with respect to
median filtering. However, the asymmetry of chain patterns is not cancelled.
The reason is that number of back-to-back probe packets constituting the chain
pattern outnumbers the initial probe packet dispersion. Thus, the median is
shifted towards the dispersion of back-to-back packets. Since the back-to-back
chain corresponds to the measured probe rate whereas the offered rate corre-
spond to the initial probe packet dispersion there will be a dip near 7 Mbps in
the median ratio curve in Figure 5.6.
52     Paper A


     Using the mathematical definitions of chain patterns it can be shown that
chain patterns can not occur until probing at 7 Mbps on a 10 Mbps link with
an MTU of 1500 bytes.
     Figure 5.7 shows an even more striking behavior when median filtering is
applied. The median filtering curve begins to rise at 5 Mbps because of quan-
tification patterns that occur when the channel is full. At 7 Mbps there is a drop
in the ratio curve caused by chain patterns. As before, these chain patterns are
invisible when using mean filtering. When the probe rate is increased further
the quantification patterns will again dominate and hence cause a new rise of
the median curve.
     The two median curves has a chain saw shape near the available band-
width (7 Mbps in Figure 5.6 and 5 Mbps in 5.7) compared to the mean curves.
Exactly why this happens is ongoing research. We also study if there is a possi-
bility to combine the information from mean and median curves to make better
estimates of the available bandwidth.


5.6 Conclusions
In this article we have identified and described patterns that arise when probe
packets and cross-traffic packets interact with each other. Further, we have
identified signatures that originate from these patterns. We have studied how
these patterns and their corresponding signatures affect mean and median oper-
ations that are used in the analysis phase of available bandwidth measurement
methods. All measurements in this article has been made in a testbed.
    Our goal is to further study the patterns and the corresponding signatures
derived from dispersion values obtained from packet-train probing schemes.
We will study mean and median operations and especially the combination of
them to make better estimates of available bandwidth and link capacity on an
end-to-end path.
Bibliography

[1] Robert Carter and Mark Crovella. Measuring bottleneck link speed in
    packet-switched networks. Technical Report 1996-006, Boston Univer-
    sity Computer Science Department, Boston, MA, USA, March 1996.
[2] Constantinos Dovrolis, Parameswaran Ramanathan, and David Moore.
    What do packet dispersion techniques measure? In Proceedings of the
    IEEE Infocom conference, Anchorage, USA, April 2001.
[3] Srinivasan Keshav. A control-theoretic approach to flow control. In Pro-
                                                u
    ceedings of ACM SIGCOMM, pages 3–15, Z¨ rich, Switzerland, Septem-
    ber 1991.
[4] Kevin Lai and Mary Baker. Measuring bandwidth. In Proceedings of
    IEEE INFOCOM, pages 235–245, New York, NY, USA, March 1999.
                          o
[5] Bob Melander, Mats Bj¨ rkman, and Per Gunningberg. Regression-based
    available bandwidth measurements. In Proceedings of the SCS Sympo-
    sium on Performance Evaluation of Computer and Telecommunications
    Systems, San Diego, USA, July 2002.
[6] Manish Jain and Constantinos Dovrolis. End-to-end available bandwidth:
    Measurement methodology, dynamics, and relation with TCP throughput.
    In Proceedings of the ACM SIGCOMM conference, Pittsburg, USA, Au-
    gust 2002.
             a
[7] Attila P´ sztor and Darryl Veitch. The packet size dependence of packet
    pair like methods. In Tenth International Workshop on Quality of Service,
    Miami Beach, USA, May 2002.
[8] Rikard Holm. Probe-based load measurement. Master thesis, Ericsson
    Research, Sweden, 2003.

                                    53
 [9] Yongmin Choi, Heung-No Lee, and Anurag Garg. Measurement and
     analysis of wide area network (wan) traffic. In Proceedings of the SCS
     Symposium on Performance Evaluation of Computer and Telecommuni-
     cation Systems, Vancouver, Canada, July 2000.
                                                      o
[10] Andreas Johnsson, Bob Melander, and Mats Bj¨ rkman. Modeling of
     packet interactions in dispersion-based network probing schemes. Tech-
                    a
     nical report, M¨ lardalen University, 2004.
Chapter 6

Paper B:
An Analysis of Active
End-to-end Bandwidth
Measurements in Wireless
Networks

                           o
Andreas Johnsson, Mats Bj¨ rkman and Bob Melander
In proceedings of the 4th IEEE/IFIP End-to-end Monitoring Techniques and
Services workshop, Vancouver, Canada, April 2006.




                                  55
                                   Abstract

For active, probing-based bandwidth measurements performed on top of the
unifying IP layer, it may seem reasonable to expect the measurement problem
in wireless networks to be no different than the one in wired networks. How-
ever, in networks with 802.11 wireless bottleneck links we show that this is not
the case.
    The results from the experiments presented in this paper show that the mea-
sured available bandwidth is dependent on the probe packet size (contrary to
what is observed in wired networks). Another equally important finding is that
the measured link capacity, using the well known TOPP model, is dependent
on the probe packet size and on the cross-traffic intensity.
    The underlying reasons for the observed differences are analyzed by incor-
porating the characteristics of 802.11 wireless networks into the TOPP model.
The extended model is applicable to other end-to-end bandwidth measurement
methods as well, such as BART, Pathload and PTR.
                                                       6.1 Introduction        57


6.1 Introduction
Wireless networks, used when connecting to the Internet or when several nodes
want to communicate in an ad-hoc manner, are becoming more and more pop-
ular. Because of the increased dependence on wireless network technology, it
is important to ensure that methods and tools for network performance mea-
surement also perform well in wireless environments. In this paper, we focus
on active performance measurements in terms of network bandwidth, both link
capacity and the unused portion thereof; the available bandwidth.
     Measurement of network properties such as available bandwidth in a va-
riety of networks are important for network error diagnosis and performance
tuning but also as a part of the adaptive machinery of network applications such
as streaming audio and video.
     State-of-the-art active end-to-end bandwidth measurement methods are for
example BART [1], Pathchirp [2], Pathload [3], PTR [4], Spruce [5] and TOPP
[6][7]. The basic principle is to inject a set of measurement packets, so called
probe packets, into the network. The probe packets traverse the network path
to a receiver node, which time stamps each incoming probe packet. By analyz-
ing these time stamps, estimates of the link capacity (a physical property of a
link) and/or the available bandwidth (the unused portion of the link capacity)
can be made. For many end-to-end available bandwidth measurement meth-
ods no prior knowledge of the underlying network topology is needed. That
is, bandwidth estimation methods are well suited for end-to-end performance
measurements and monitoring in all sorts of networks. The existing methods
differ in how probe packet are sent (the flight patterns) and in the estimation al-
gorithms used. A good overview of available bandwidth measurement methods
and tools can be found in [8].
     In this paper we study how the properties of wireless 802.11 bottleneck
links affect estimates reported by active end-to-end bandwidth measurement
methods. We measure, analyze and describe how the characteristics of band-
width estimation change in wireless scenarios. By conducting experiments
where we vary the probe-packet size we show that the obtained bandwidth
estimates reported vary. Further, by varying the cross-traffic rate we illustrate
how the estimated link capacity obtained by using the well known TOPP model
is not constant.
     To understand the obtained bandwidth estimates reported we incorporate
the properties of 802.11 wireless networks into the TOPP model. Using the
extended model we describe why and how the probe-packet size affect the
bandwidth estimate. Further, we discuss the relation between the estimated
58     Paper B


link capacity (using TOPP) and the real link capacity of a wireless 802.11
bottleneck link.
    The extended TOPP model is also applicable to, and will thus explain why
estimates produced by other end-to-end bandwidth measurement methods (e.g.
BART, Pathload and PTR) vary with the probe-packet size in wireless 802.11
networks. We discuss this in detail in the paper.
    The bandwidth measurements have been performed in a testbed containing
both wireless and wired hops. Our testbed topology only consist of one wire-
less 802.11 bottleneck link, to be used as an access link, since that is a common
way to deploy wireless networks. To produce measurement results we have
used DietTopp, a tool that implements the TOPP model, which measures the
available bandwidth and link capacity of an end-to-end path. For comparisons
and to illustrate that our observations concerning the available bandwidth es-
timates are not tied to a certain measurement tool, we have also used the tool
Pathload, which also measure the available bandwidth of an end-to-end path,
in our experiments.
    Earlier work has touched upon the problem of end-to-end measurement
of bandwidth in wireless networks. In [9] we discuss the main problem ar-
eas when deploying existing bandwidth measurement methods in wireless net-
works. Further, measurement results presented in [10] indicate that the avail-
able bandwidth estimates is dependent of the probe packet size in multi-access
networks, such as wireless 802.11 networks. The authors are not restricted to
bandwidth measurements in 802.11 networks, but rather investigate a range of
access network techniques.
    This paper complement and extend the work made in [10]. In this paper
we develop an analytical model describing the measurement results. Further,
compared to [10] we use more complex testbed scenarios along with a different
set of bandwidth measurement methods.
    The rest of this paper is organized as follows. Section 6.2.1 describes the
measurement model. DietTopp, which is our implementation of a simplified
TOPP model, is also presented. Section 7.3 is a description of the testbed we
have used for the investigation of the bandwidth measurement problem in wire-
less networks. In Section 6.2.3 the experiments are described along with the
research question. Section 6.3 show measurement results from using DietTopp
in wired as well as in wireless networks. We discuss the results and compare
them to results obtained by Pathload. In Section 6.4 we extend the original
TOPP model to describe and explain the obtained measurement results. Fur-
ther, in Section 6.5 some important observations are made. The paper ends
with conclusions in Section 9.6.
                                                  6.2 Experimental setup     59
            oi / mi




                                              b


          y=1




                                                                o
                                     a                              i

            Figure 6.1: Plot of the ratio oi /mi as a function of oi .


6.2 Experimental setup
This section describes the experimental setup used in order to study the be-
havior of bandwidth measurement methods in wireless 802.11 networks. That
is, the measurement model and the tools used in the experiments. The testbed
setup and finally what kind of bandwidth measurements that have been per-
formed along with their relevance are also presented.

6.2.1 Measurement model
The TOPP model [6] is well suited to describe and explain how end-to-end
bandwidth measurement methods function [11]. Therefore we use that model
to study the impact of wireless bottleneck links on bandwidth estimates. The
TOPP model is described below.
    The available bandwidth on a single link can be described as the unused
portion of the link capacity during some time period. The end-to-end available
bandwidth is the minimum available bandwidth for each link in the path. That
link defines the bottleneck.
    The basic assumption is that the probe packets gain a proportional share of
the link capacity when traversing the bottleneck link. That is

                                     m        if m < a
                       m=          o                                       (6.1)
                                 (x+o)   ∗L   if m ≥ a
60     Paper B



                            X1
       Xw2      11Mbps                      100Mbps

                                  100Mbps


           11Mbps
                                      10/100                  10/100                  100Mbps

     Xw1                         R1                     R2                                      D
                                                                                 R3
                                                        100Mbps


                    11/100Mbps
       S                                       X2                      100Mbps




Figure 6.2: The testbed is constructed by one wireless link, three routers and
several cross-traffic generators (on both the wireless and the wired links)


    where m is the measured probe-packet rate, o the offered probe-packet
rate, x the cross-traffic rate on the link with capacity L and a is the available
bandwidth on the bottleneck link.
    Equation 6.1 can be converted into a linear form describing the quotient
o/m

                            o                  1             if m < a
                              =                a        o                                       (6.2)
                            m          (1 −    L)   +   L    if m ≥ a

    where a is the available bandwidth on the specific link. A common defini-
tion of a is a = (L − X)/δ where X is the amount of cross traffic that flows
through the link during time δ. If plotting Equation 6.2 in a diagram the result
will be much like the theoretical one in Figure 6.1. This theoretical curve is
in [11] called the rate response curve, that is, the relation between the offered
rate and the measured rate. We adopt that term in this paper. Further, in [11] it
is shown that the fluid-like analysis described above can give erroneous band-
width estimates when taking packet-level interactions in the router queues into
account. Especially if the cross traffic is bursty (e.g. Pareto distributed) or if
there exist several secondary bottlenecks [6]. By using longer probe-packet
trains instead of probe-packet pairs the obtained rate response curve asymptot-
ically moves towards the fluid curve shown in Figure 6.1. The tools used in the
experiments use long probe-packet trains. Further, we study the effect of one
wireless bottleneck link on the rate response curve. Thus, for the objectives in
this paper we believe that the original fluid model is sufficient.
                                               6.2 Experimental setup          61


    To study how wireless 802.11 networks affect available bandwidth esti-
mates (i.e. how the rate response curve in Figure 6.1 is affected) we have
implemented a tool called DietTopp [12]. (In [7] preliminary results obtained
from DietTopp in wireless networks is discussed along with the tool itself.) It
measures the end-to-end available bandwidth along with the link capacity of
the bottleneck link. Further, Pathload has been used to show that the results are
not tied to a certain tool.
    Both DietTopp and Pathload directly or indirectly use the information em-
bedded in rate response curve to estimate the available bandwidth. We describe
how below:
    DietTopp injects a set of probe-packet trains at an increasing rate in the
interval [omin , omax ]. On the receiver side each probe packet is time stamped
in order to calculate mi for each incoming probe-packet train i. The probe-
packet train rate increases for each successive train, hence the bottleneck link
will be congested at some point (corresponding to oi = a in Figure 6.1). When
all probe-packet trains have traversed the network path the quotient oi /mi can
be plotted. The rate response curve in Figure 6.1 is used as an example. Di-
etTopp uses linear regression to estimate the linear segment b. The end-to-end
available bandwidth is defined as the offered rate corresponding to the inter-
section of b and y = 1. Further, the slope of b corresponds to the bottleneck
link capacity according to [6].
    To speed up the probing phase of DietTopp it is desired to avoid measure-
ments with an offered rate o below a. That is, DietTopp wants to ensure that
omin > a. This is done by estimating mmax which is done by injecting a set
of probe packets at rate omax (could be the link capacity of the access link
for example) and then measure their separation at the receiver. According to
[6] mmax is greater than the available bandwidth (mmax is referred to as the
asymptotic dispersion rate in [13]). DietTopp also assumes only one bottle-
neck link between the end nodes contrary to the more computational expensive
TOPP model.
    Pathload is based on observations of the one-way delay of probe-packet
trains. If the offered probe-packet rate is above the available bandwidth the
one-way delay will show an increasing trend. On the other hand, if the one-way
delay doesn’t show an increasing trend the offered rate is below the available
bandwidth. To locate the available bandwidth, Pathload deploys binary search.
That is by varying the offered probe rate and investigating the one-way delay.
In [11] it is stated that there is a strong statistical correlation between a high
rate response and the increasing trend of the one-way delay within a probe-
packet train. Thus, Pathload is tied to the rate response curve. A change in the
62     Paper B


curve changes the estimates produced by Pathload.
    In [11] a description on how the rate response curve affect estimates pro-
duced by Pathload as well as other available bandwidth methods (e.g. Spruce
and PTR) is described in more detail. BART directly use the rate response
curve in order to estimate the available bandwidth. Thus, the study on how
the rate response curve change due to properties in wireless 802.11 networks
is very important.


6.2.2 The testbed
The testbed used in this work consists of 9 computers running Linux, shown in
Figure 9.1. The link speed for each link is shown in the figure. The links be-
tween Xw1, Xw2 and R1 are 802.11b wireless links (sharing the same chan-
nel) while the link between S and R1 either can be a 802.11b wireless link or
a 100 Mbps wired link.
    The bottleneck in the experiments is either the wireless 802.11b link or the
link between R1 and R2, depending on whether the link between S and R1 is
wired or wireless.
    The cross traffic in the testbed is generated at the nodes Xw1, Xw2 and
X1. The cross traffic consists of UDP packets and is generated by a modified
version of tg [14]. The cross traffic is either constant bit rate (CBR), expo-
nential or Pareto distributed (shape = 1.5). Further, the cross traffic consists of
60 (46% of the packets), 148 (11%), 500 (11%) and 1500 (32%) byte packets.
This distribution of packet sizes originates from findings in [15].


6.2.3 Research question and experiments
In this paper we want to identify and explain properties associated with band-
width measurements in wireless 802.11 networks.
    The measurements have been performed mainly using DietTopp. We elab-
orate on the impact of probe packet size, the cross-traffic distribution and on
the number of cross-traffic generators in the wireless network. We compare
our results regarding available bandwidth to results obtained from Pathload.
    The goal of the measurements performed is to understand the impact of the
variables described above on the rate response curve. That is, the impact on the
estimated available bandwidth (and link capacity) reported by many bandwidth
measurement methods.
                                                                   6.3 Experimental results               63


                           Available bandwidth and link capacity estimates in a wireless network
             7
                   Link capacity cbr0x
                   Available bandwidth cbr0x
                   Link capacity cbr250x
             6     Available bandwidth cbr250x
                   Link capacity cbr500x
                   Available bandwidth cbr500x


             5




             4
    (Mbps)




             3




             2




             1




             0
             250             500                 750                1000                1250       1500
                                                 Probe−packet size (byte)



Figure 6.3: Measured available bandwidth and link capacity using different
probe-packet sizes under the impact of 0, 250 Kbps and 500 Kbps cross-traffic
rates.


6.3 Experimental results
This section presents the results obtained from running DietTopp in different
experiment scenarios. We have used Pathload to compare the obtained mea-
surement results. In the diagrams all measurement results are shown with a
95% confidence interval.


6.3.1 Measurement results in wireless networks
This subsection presents the results from measurements using DietTopp and
Pathload where the bottleneck is a wireless link (the link between S and R1
in the testbed as described in subsection 7.3). Cross traffic is present on both
of the wired links R1 - R2 and R2 - R3, but the rate is limited to approxi-
mately 9% of the corresponding link capacity (100 Mbps in this case). That is,
64             Paper B


                         Available bandwidth and link capacity estimates in a wired network
              10


               9


               8


               7


               6
     (Mbps)




               5


               4


               3


               2


               1
                                                                                     Link capacity
                                                                                     Available bandwidth
               0
               250       500                 750                1000                 1250              1500
                                             Probe−packet size (byte)



Figure 6.4: Available bandwidth and link capacity measured by DietTopp in
a wired network using different probe packet sizes. The cross traffic is a 3.26
Mbps Pareto distributed stream on a 10 Mbps link.


the wireless link is the link that limits both the link capacity and the available
bandwidth. The cross traffic at the 100 Mbps links between R1, R2 and R3 is
Pareto distributed (with respect to cross-traffic packet arrival times) and con-
sists of 4 different packet sizes. The cross-traffic configuration on the wired
links is the same for each experiment presented in this section.
    The probe-packet size affects both the measured link capacity and the avail-
able bandwidth estimate obtained by DietTopp when the bottleneck on an end-
to-end path is a wireless 802.11 link. We illustrate and describe this phe-
nomenon in a set of diagrams below.
    The two upper curves in Figure 6.3 show the measured link capacity and
the measured available bandwidth when no cross traffic is present on the wire-
less link. Varying the probe packet size from 250 bytes up to 1500 bytes gives
increasing values of both the measured link capacity and the measured avail-
able bandwidth. It should be observed that the total number of bits remains
                                                                   6.3 Experimental results               65


                           Available bandwidth and link capacity estimates in a wireless network
             7
                   Link capacity exp0x
                   Available bandwidth exp0x
                   Link capacity exp250x
             6     Available bandwidth exp250x
                   Link capacity exp500x
                   Available bandwidth exp500x


             5




             4
    (Mbps)




             3




             2




             1




             0
             250             500                 750                1000                1250       1500
                                                 Probe−packet size (byte)



Figure 6.5: Available bandwidth and measured link capacity measured under
the impact of 0, 250 Kbps and 500 Kbps exponentially distributed cross-traffic.


constant independent of the probe packet size. The total amount of probe data
sent by DietTopp in these measurements is 1.2 Mbit. Each probe train consists
of 16 probe packets and we send 5 probe trains on each probe rate level. The
number of probe rate levels depends on the probe packet size; increasing the
probe packet size decreases the number of probe rate levels.
    The two middle curves show measurement estimates when there is a 250
Kbps constant-bit-rate (CBR) cross-traffic stream competing with the probe
packets on the wireless link. The two bottom curves correspond to the case
when a 500 Kbps CBR stream is present. Both the measured link capacity and
the measured available bandwidth increase with increasing probe-packet size.
Another equally important observation to be made is that the measured link ca-
pacity decreases when increasing the cross-traffic rate. Yet another interesting
phenomenon is that the difference between the measured link capacity and the
measured available bandwidth tends to be smaller for small probe packet sizes.
    For comparison we have varied the probe packet size in an all wired net-
66     Paper B


             Cross traffic          Measurement (Mbps)
                   0                    2.32 - 2.39
               250k cbr                 1.67 - 1.67
               250k exp                 1.73 - 1.73
               250k par                 1.40 - 1.63
               500k cbr                 0.96 - 0.99
               500k exp                 0.87 - 0.95
               500k par                 1.27 - 1.29

Table 6.1: Measurement results obtained from running Pathload in different
scenarios with varying cross-traffic intensity and distribution.



work. That network is essentially the network shown in Figure 9.1 but the link
between S and R1 is now a 100 Mbps wired link. The bottleneck is a 10 Mbps
link between R1 and R2. The measurement results can be seen in Figure 6.4.
Both the measured link capacity and the available bandwidth reported by Diet-
Topp are relatively accurate and stable, that is independent of the probe packet
size.
    We have also performed measurements using Pathload, a tool that esti-
mates the available bandwidth using 300 byte packets. The results obtained
from executing Pathload in our testbed with different cross-traffic distributions
and intensities can be seen in Table 6.1. When comparing results obtained
by Pathload (in Figure 6.3) to those of DietTopp we can see that Pathload re-
ports available bandwidth measurement estimations that are in line with esti-
mations made by DietTopp (using interpolation between packet sizes 250 and
500 bytes).
    Figures 6.5 and 6.6 report results from the same type of measurements that
the ones Figure 6.3 is based on. The available bandwidth and the measured
link capacity increases with increasing probe-packet size. Further, both the
available bandwidth and the measured link capacity decrease with increasing
cross-traffic rate. However, in these two scenarios more complex cross-traffic
distributions are used. In Figure 6.5, exponentially distributed arrival times
for the cross-traffic packets is used while in Figure 6.6 Pareto distributed ar-
rival times is used. As can be seen in both figures, the confidence intervals
are larger when the cross traffic is burstier. It is also obvious that the curves
are less smooth compared to the CBR case in Figure 6.3. In the Pareto case
(Figure 6.6) it is hard to distinguish between estimates of the link capacity in
                                                                   6.3 Experimental results               67


                           Available bandwidth and link capacity estimates in a wireless network
             7
                   Link capacity par0x
                   Available bandwidth par0x
                   Link capacity par250x
             6     Available bandwidth par250x
                   Link capacity par500x
                   Available bandwidth par500x


             5




             4
    (Mbps)




             3




             2




             1




             0
             250             500                 750                1000                1250       1500
                                                 Probe−packet size (byte)



Figure 6.6: Available bandwidth and measured link capacity measured under
the impact of 0, 250 Kbps and 500 Kbps Pareto distributed cross-traffic.


the two different cross-traffic scenarios (250 Kbps and 500 Kbps cross traffic).
However, we can still see that the measured link capacity and available band-
width are dependent on both the probe packet size and the cross-traffic rate.
Again, comparing the measurement results (at the 300 byte probe packet size
level) with results obtained by Pathload (in Table 6.1) we can conclude that the
available bandwidth estimate characteristics are compatible.
    The somewhat blurred measurement results obtained when the cross traffic
is more bursty (i.e. exponentially or Pareto distributed) can possibly originate
from the fluid-like TOPP model. As mentioned above, a more complete, non-
fluid, model has been developed in [11]. However, in the wired case which
results are shown in Figure 6.4 no large fluctuations of the estimated band-
width are visible, even though the cross traffic is Pareto distributed. Also, the
estimates are rather accurate. The problem of the blurred measurement results
is part of future research.
    Continuing to Figure 6.7. In this scenario, two cross-traffic generators are
68                Paper B


                             Available bandwidth and link capacity estimates in a wireless network
              7
                     Link capacity cbr0x
                     Available bandwidth cbr0x
                     Link capacity 2cbr125x
              6      Available bandwidth 2cbr125x




              5




              4
     (Mbps)




              3




              2




              1




              0
              250              500                  750                1000               1250       1500
                                                    Probe−packet size (byte)



Figure 6.7: Available bandwidth and measured link capacity measured under
the impact of 0 and 250 Kbps CBR cross-traffic. The cross traffic is generated
by two different sources that is injecting 125 Kbps each.



generating 125 Kbps of CBR cross traffic each (that is summed up to 250
Kbps). As before, the probe-packet size affects both the measured link capac-
ity and available bandwidth. Comparing Figure 6.7 to the measurement results
shown in Figure 6.3 we see that the confidence intervals are larger when hav-
ing multiple cross-traffic generators. Otherwise the curves in Figure 6.3 and
in Figure 6.7 are similar. Hence, several cross-traffic generators on different
nodes in a wireless network seem to increase the uncertainty, but not the over-
all estimates produced by DietTopp.

    In the next section the results presented above will be discussed and ana-
lyzed in detail. This analysis result in an extended TOPP model for describing
bandwidth measurements results obtained from networks with a wired or a
wireless bottleneck.
                                   6.4 Analysis of experimental results           69


6.4 Analysis of experimental results
The reason for the varying measurement estimates of the link capacity and
the available bandwidth in the experiments (as seen in the previous section)
can be derived from the link-level acknowledgements and the contention phase
used in 802.11 networks [16]. That is, if a probe packet is small, the relative
overhead induced by the link-level acknowledgement and the contention phase
is larger than if the probe packet were large. This will affect the probe-packet
separation, which is the basis of the the rate response curve. Thus, the available
bandwidth estimates produced by DietTopp, Pathload and other methods will
vary with varying probe-packet size.

6.4.1 The extended TOPP model
The basic assumption in the TOPP model is, as described above, that the in-
jected probe packets gain a proportional share of the link capacity when travers-
ing a bottleneck link (if first-come-first-served is used as the queue policy).
That is, the relationship during congestion (or overload) is described as
                                          o
                                m=             ∗L                              (6.3)
                                       (x + o)
where m is the measured probe rate (at the receiver side of the network), o is
the offered probe rate, x is the cross-traffic rate on the link and L is the link
capacity on the specific link. Converting this equation into its linear form gives


                          o            (L − x)    o
                               = (1 −          )+                              (6.4)
                          m               L       L
                               = α ∗ o + β.

    Now, assume that the link capacity can be rewritten as L = s/T where s is
the size in bits and T is the time it takes for the link to transmit s bits (assuming
the queueing delay to be zero). Further, the offered rate o can be written as s/to
where s again is the bit size and to the separation required in order to obtain a
specific offered rate o. Then Equation 6.4 can be rewritten as

                         o        (s/T − x)    s/to
                           = (1 −           )+                                 (6.5)
                         m           s/T       s/T
    Now, in a 802.11 wireless networks the time it takes to transmit one packet
of size s over the link can be expressed as [16]
70     Paper B




                   T    =    Tsif s + TBO + Tdif s + Tack + Ts               (6.6)
                        =    Tk + Ts

    where Tsif s and Tdif s correspond to the time it takes to access the 802.11
wireless link, TBO is the backoff time and Tack is the time it takes for a link-
level acknowledgment to return after sending a packet over the wireless link.
TBO is uniformly distributed in the interval [0, CW − 1] where CW grows with
each unsuccessful transmission (due to competing cross traffic). Thus, Tk in
the above equation, is an increasing function of the cross-traffic intensity x (i.e.
Tk = Tk (x)).
    Further, Ts is the time in seconds to transmit a packet of size s over the
wireless channel. This time is determined by how the wireless signal is mod-
ulated, settings in the wireless network card and the size of the packet to be
sent.
    Substituting T in Equation 6.5 with the expression for T in Equation 6.6

                   o   Tk (x) + Ts                     1
                     =             ∗ x + (Tk (x) + Ts ) .                    (6.7)
                   m        s                          to
    Assume in the above equation that the probe-packet sender is injecting
probe packets with a size of s bits. If changing the probe-packet size to s/n
bits, where n is an arbitrary number (n ≥ 1), the separation between the probe
packets must also change in order to obtain the same rate o (i.e. to /n). Further,
Ts is decreased to Ts /n. Thus


          o            Tk (x) + Ts /n                          1
               =                      ∗ x + (Tk (x) + Ts /n)       =⇒
          m                 s/n                              to /n
          o            n ∗ Tk (x) + Ts                          1
               =                       ∗ x + (n ∗ Tk (x) + Ts )              (6.8)
          m                    s                                to
                                 1
               =       β+α∗ .                                                (6.9)
                                 to

    From Equation 6.8, which describes the rate response curve when the bot-
tleneck is a wireless 802.11 link, it can be seen that a decreasing probe-packet
size (i.e. n ¿ 1) increases both α and β in Equation 6.9. This, in turn decreases
both the measured link capacity and the measured available bandwidth accord-
ing to the TOPP model described in Section 6.2.1. This behavior is visible in
                                 6.4 Analysis of experimental results        71


the diagrams in Section 6.3 that shows measurements in wireless 802.11 net-
works. Further, when increasing the cross-traffic intensity, Tk (x) will increase
due to increased contention and backoff time. This is also visible in the di-
agrams shown in the previous section where both the measured link capacity
and the available bandwidth decrease with increasing cross-traffic intensity.
     When the bottleneck is not a multi-access wireless link (i.e. when Tk (x) =
0) but rather a single access wired link the probe-packet size does not have
impact on the estimates, which is seen in Equation 6.8. (Ts /s is constant.) The
measurement results in Figure 6.4 support the above statement.
     The important conclusion to be drawn from this section is that wireless
802.11 networks changes the parameters describing the rate response curve in
Figure 6.1 when the probe-packet size is allowed to vary. A small probe-packet
size increases the slope of b (i.e. decreases the estimate of the link capacity)
while at the same time the intersection of b and y = 1 moves towards zero
(i.e. decreases the estimate of the available bandwidth). This fact affects the
estimates produced by DietTopp, Pathload and other methods that directly on
indirectly rely on the same basic bandwidth estimation model.
     The original TOPP model also describes how the relation between o and m
changes when several congestable links (a link that will be congested due to
the probe packets) between the sender and the receiver are present. The above
supplement to the TOPP model can of course be incorporated into the case with
several congestable links.

6.4.2 Estimated vs. physical link transmission capacity of a
      802.11 wireless link
There is a need to differentiate between the estimates of the link capacity pro-
duced by DietTopp, that is not constant with respect to the cross-traffic rate
and the probe-packet size, and the fixed physical link transmission capacity
of the 802.11 wireless link. Using the equations in the previous section the
differentiation is discussed in this section.
    The TOPP model suggests that the bottleneck link capacity is computed as
1/α, where α is the slope of the straight line b shown in Figure 6.1. As derived
in the previous section α = n ∗ Tk (x) + Ts where Ts is the actual transmission
time (which is constant if no change in the topology is made). Tk (x) corre-
sponds to contention and backoff time in multiple access networks, such as
wireless 802.11. Thus, calculating 1/Ts gives the physical link transmission
capacity of the wireless bottleneck link. First, assume that Tk (x) is constant,
with respect to the probe-packet size, then Ts is easily calculated by solving
72       Paper B


the equation system below
                                 1
                                 α1    = n1 ∗ Tk (x) + Ts
                                 1                                                       (6.10)
                                 α2    = n2 ∗ Tk (x) + Ts

    where α1 and α2 correspond to link capacity estimates while n1 and n2 are
fixed during the measurement.
    In Figure 6.3 (CBR case) the estimated link capacity is 5.92 Mbps (= α1 )
when no cross traffic is present using 1500 byte probe packets (fix n1 = 1).
Further, the estimated link capacity is approximately 4.16 Mbps (= α2 ) in the
same scenario but when using 750 byte probe packets (then n2 = 2). That
is, n1 = 1 and n2 = 2 since we use half the probe-packet size is the second
scenario. Solving the above equation system with these values as input Ts =
8.34−8 which in turn gives a physical link transmission capacity of 10.5 Mbps.
This estimate is fairly close to 11 Mbps which is the link transmission capacity
in the experiments.
    However, solving similar equation systems using input from scenarios with
competing cross traffic, Ts will not correspond to a link transmission capac-
ity of 11 Mbps at all. 1 This is impossible, since Ts is a physical, constant
property of the wireless link (in our experiments). Thus, Tk must vary with the
probe-packet size for some reason. In Figure 6.8 the variation of Tk is shown,
assuming Ts to be constant (Ts = 9.09−8 which corresponds to a link trans-
mission capacity of exactly 11 Mbps). As can be seen, the variation of Tk can
be described by a straight line.
    Further, as seen in the figure, Tk increases with increasing cross-traffic in-
tensity independent of the probe-packet size. This is obvious since a high cross-
traffic intensity increases the backoff time TBO in Equation 6.6. It should also
be noted that the slope of Tk increases with increasing cross-traffic intensity.
    We believe that the observed dependency of Tk on the probe-packet size
can be traced to the link-layer retransmission mechanism in 802.11 wireless
networks [16]. A 802.11 wireless node cannot tell whether a collision on the
link has occurred or not since the radio used cannot send and listen at the
same time. That is, the only way to detect a collision (or other link-layer er-
rors that requires a retransmission) is the lack of a link-layer acknowledgment
that should be transmitted back by the receiver node. When no link-layer ac-
knowledgment is received, the link-layer on the sending side retransmits the
packet. The question is; why is the packet size crucial to Tk when a link-layer
   1 The exact numbers for the estimated link capacity is left out, but a rough approximation can

be obtained from the figure.
                                         6.4 Analysis of experimental results          73


                 −7                         The variation of Tk
              x 10
        2.6
                      No cross traffic
                      250x
        2.4           500x


        2.2


         2


        1.8


        1.6
    s




        1.4


        1.2


         1


        0.8


        0.6
          250                    500     750                1000    1250        1500
                                         Probe−packet size (byte)



        Figure 6.8: The variation of Tk with increasing probe-packet size.


retransmission is triggered? There are probably several reasons for this and we
elaborate on two below.
    Collisions: In the case corresponding to the measurement results presented
in this work, the number of transmitted bits (1.2 Mbit) is constant indepen-
dently of the probe-packet size, as described in Section 6.3. Hence, sending
one 1500 byte probe packet corresponds to sending six 250 byte probe packets.
If one collision occur, in either case, one probe packet has to be retransmitted
by the link layer. The procentual increase in Tk will be much higher using 1500
byte probe packets compared to using 250 byte probe packets. That is why Tk
increases with increasing packet size.
    Noisy wireless link: If the wireless link is noisy the probability for bit
errors in the transmitted packet increases. Assuming that a constant number of
probe packets are sent, the probability for encountering bit errors will increase
with the probe-packet size. That is, the probe packet must be retransmitted and
Tk increases.
    It seems that Tk is actually a function of both the cross traffic and the packet
74     Paper B


size. That is, Tk = Tk (x, s). Assuming the cross-traffic intensity is constant
then Tk (s) = φ + ϕ ∗ s where s is the probe-packet size while φ and ϕ are
constants describing the suggested straight lines in Figure 6.8.
    To be able to calculate Ts , that is the value corresponding to the physical
link transmission capacity of the wireless link the function Tk (x, s) must be
known. In the case where no cross traffic flows on the wireless link, Tk is
rather constant (see Figure 6.8) and thus, Ts could be calculated, as shown
above. However, whether it is possible to calculate Ts in the general cross-
traffic intensity case is left for future research.


6.5 Other observations
Due to the fact that the probe-packet size affects the rate response curve shown
in Figure 6.1 (and thus affects both the measured link capacity and the mea-
sured available bandwidth when using DietTopp), a possible method to identify
a wireless bottleneck in a network path could be: if the available bandwidth
(and the measured link capacity) changes when probing the path with different
probe-packet sizes, this can be taken as an indication that the path includes a
wireless bottleneck. This is important since, as we have discussed, wireless
bottlenecks have different characteristics than wired bottlenecks.
    An important consequence of the measurements we have presented in this
paper is that the available bandwidth will be application dependent if the bottle-
neck is a wireless link. For example, a voice over IP application or a distributed
game probably use small packets while a file transfer may use large packets.
The available bandwidth for the applications will not be the same due to their
packet-size distribution. This means that when probing a path containing a
wireless bottleneck link, the estimation tool must use a probe-packet size dis-
tribution that corresponds to the specific application that is to use the estimates.


6.6 Conclusion
In this paper we have shown measurement results that illustrate the difference
between available bandwidth measurement results obtained in wired and wire-
less networks. The impact of the probe-packet size, the cross-traffic rate and
the number of cross-traffic generators has been analyzed. In the paper, we have
discussed in detail the underlying reasons for these differences by extending the
TOPP model. The extended model shows that the rate response curve has dif-
                                                      6.6 Conclusion        75


ferent properties when performing active end-to-end bandwidth measurements
in 802.11 networks compared to the one obtained in ordinary wired networks.
    Using the extended model it has been shown why the probe-packet size,
for example, is crucial to the available bandwidth estimate. Further, we have
shown why the estimated link capacity reported by TOPP is erroneous when
performing measurements where the bottleneck is a wireless 802.11 link. A
very important aspect of the extended TOPP model is that it also have im-
plications on the estimates reported by other end-to-end available bandwidth
measurement methods (e.g. BART, DietTopp, Pathload and PTR).
    The estimates of the available bandwidth and link capacity have been pro-
duced using our own tool, DietTopp. For comparison and validity we have
used Pathload. The measurements have been performed in a testbed where we
have used different types of cross traffic, from simple CBR to bursty Pareto
distributed cross traffic.
    Future research is to study whether the physical link capacity of 802.11
wireless bottleneck links can be estimated in general scenarios using DietTopp.
Another open issue is to study how the 802.11 properties can fit into a non-
fluid model for describing available bandwidth measurement methods. It is
also important to investigate why the use of small probe packets results in a
lower variance when used for active probing in wireless 802.11 networks.
   Bibliography




Bibliography

                                                             a
[1] S. Ekelin, M. Nilsson, E. Hartikainen, A. Johnsson, J. M˚ ngs, B. Me-
                     o
    lander, and M. Bj¨ rkman. Real-time measurement of end-to-end avail-
    able bandwidth using kalman filtering. In Proceedings of the IEEE/IFIP
    Network Operations and Management Symposium, Vancouver, Canada,
    2006.

[2] V. Ribeiro, R. Riedi, R. Baraniuk, J. Navratil, and L. Cottrel. pathChirp:
    Efficient available bandwidth estimation for network paths. In Proceed-
    ings of the Passive and Active Measurement Workshop, San Diego, USA,
    2003.

[3] Manish Jain and Constantinos Dovrolis. End-to-end available bandwidth:
    Measurement methodology, dynamics, and relation with TCP throughput.
    In Proceedings of the ACM SIGCOMM conference, Pittsburg, USA, Au-
    gust 2002.

[4] Ningning Hu and Peter Steenkiste. Evaluation and characterization of
    available bandwidth probing techniques. IEEE Journal on Selected Areas
    in Communication, 2003.

[5] Strauss, Katabi, and Kaashoek. A measurement study of available band-
    width estimation tools. In Proceedings of the ACM SIGCOMM Internet
    Measurement Workshop, Miami, USA, 2003.

                          o
[6] Bob Melander, Mats Bj¨ rkman, and Per Gunningberg. Regression-based
    available bandwidth measurements. In Proceedings of the SCS Sympo-
    sium on Performance Evaluation of Computer and Telecommunications
    Systems, San Diego, USA, July 2002.

                                     76
                                               o
 [7] Andreas Johnsson and Bob Melander Mats Bj¨ rkman. Bandwidth mea-
     surement in wireless networks. In Proceedings of the Fourth Annual
     Mediterranean Ad Hoc Networking Workshop, Porquerolles, France,
     2005.
 [8] R.S. Prasad, M. Murray, C. Dovrolis, and K. Claffy. Bandwidth estima-
     tion: metrics, measurement techniques, and tools. IEEE Network Maga-
     zine, 2003.
                                o
 [9] Andreas Johnsson, Mats Bj¨ rkman, and Bob Melander. A study of
     dispersion-based measurement methods in ieee 802.11 ad-hoc networks.
     In Proceedings of the International Conference on Communication in
     Computing, Las Vegas, 2004.
[10] Karthik Lakshminarayanan, Venkata N. Padmanabhan, and Jitendra pad-
     hye. Bandwidth estimation in broadband access networks. In Proceedings
     of the Internet Measurement Conference, Taormina, Italy, 2004.

[11] Xiliang Liu, Kaliappa Ravindran, and Dmitri Loguinov. Multi-hop prob-
     ing asymptotics in available bandwidth estimation: Stochastic analysis. In
     Proceedings of the ACM SIGCOMM Internet Measurement Conference,
     Berkeley, USA, 2005.
[12] Andreas Johnsson.          Diettopp implementation,        beta   version.
                           a
     http://www.idt.mdh.se/˜ jn12/, 2005.
[13] Constantinos Dovrolis, Parameswaran Ramanathan, and David Moore.
     What do packet dispersion techniques measure? In Proceedings of the
     IEEE Infocom conference, Anchorage, USA, April 2001.
[14] Paul E. McKenney, Dan Y. Lee, and Barbara A. Denny. Traffic generator
     software release notes. SRI International and USC/ISI Postel Center for
     Experimental Networking, January 2002.
[15] Yongmin Choi, Heung-No Lee, and Anurag Garg. Measurement and
     analysis of wide area network (wan) traffic. In Proceedings of the SCS
     Symposium on Performance Evaluation of Computer and Telecommuni-
     cation Systems, Vancouver, Canada, July 2000.
[16] Ieee 802.11. http://grouper.ieee.org/groups/802/11/.
Chapter 7

Paper C:
Real-time Measurement of
End-to-End Available
Bandwidth Using Kalman
Filtering

Svante Ekelin, Martin Nilsson, Erik Hartikainen, Andreas Johnsson, Jan-Erik
  a                               o
M˚ ngs, Bob Melander and Mats Bj¨ rkman
In proceedings of the 10th IEEE/IFIP Network Operations and Management
Symposium, Vancouver, Canada, April 2006.




                                    79
                                  Abstract

This paper presents a new method, BART (Bandwidth Available in Real-Time),
for estimating the end-to-end available bandwidth over a network path. It es-
timates bandwidth quasi-continuously, in real-time. The method has also been
implemented as a tool. It relies on self-induced congestion, and repeatedly
samples the available bandwidth of the network path with sequences of probe
packet pairs, sent at randomized rates. BART requires little computation in
each iteration, is light-weight with respect to memory requirements, and adds
only a small amount of probe traffic. The BART method uses Kalman filter-
ing, which enables real-time estimation (a.k.a. tracking). It maintains a cur-
rent estimate, which is incrementally improved with each new measurement
of the inter-packet time separations in a sequence of probe packet pairs. The
measurement model has a strong non-linearity, and would not at first sight be
considered suitable for Kalman filtering, but we show how this non-linearity
can be handled. BART may be tuned according to the specific needs of the
measurement application, such as agility vs. stability of the estimate. We have
tested an implementation of BART in a physical test network with carefully
controlled cross traffic, with good accuracy and agreement. Test measurements
have also been performed over the Internet. We compare the performance of
BART with that of pathChirp, a state-of-the-art tool for measuring end-to-end
available bandwidth in real-time.
                                                      7.1 Introduction        81


7.1 Introduction
7.1.1 Overview
The capability of measuring available bandwidth is useful in several contexts,
including service level agreement verification, network monitoring and server
selection. Measurement of available bandwidth in real-time, with per-sample
update of the bandwidth estimate (as in BART), opens up for adaptation based
on available bandwidth directly (rather than ”first-order” measures such as loss
or delay) in congestion control and streaming of audio and video.
    Passive monitoring of available bandwidth of an end-to-end network path
would in principle be possible, if we could access all the network nodes in
the path. However, in practice this is typically not possible, and estimation
of available end-to-end bandwidth is only feasible by active probing of the
network path. By injecting probe traffic into the network, and then analyzing
the observed effects of cross traffic on the probes, we can estimate the available
bandwidth. This kind of active measurement only requires access to the sender
and receiver hosts.
    The main contribution of this work is the technique for applying a Kalman
filter for real-time estimation of available bandwidth. The measurement model
used in the filtering algorithm has a strong non-linearity, and would not at first
sight be considered suitable for Kalman filtering, but we show how this non-
linearity can be handled.
    We introduce the convenient measure inter-packet separation strain of con-
secutive probe packets. When there is no congestion, this strain is zero on
average. When the total load starts to become larger than the path’s bottleneck
capacity, the strain becomes proportional to the overload.
    The Kalman filter technique, along with a suitable probing scheme, com-
prise a measurement method, BART (Bandwidth Available in Real-Time). We
present results obtained by a BART implementation from measurements in a
laboratory network as well as over the Internet, demonstrating the viability of
the method.
    Some of the features of BART are: it produces an estimate quickly; stabil-
ity can be traded for agility; no communication is required from the receiver of
the probe packets to the sender; tuning is largely automatic, i.e. there are few
parameters that need manual adjustment. Nevertheless BART may be tuned ac-
cording to the specific needs of the measurement application, such as agility vs.
stability of the estimate. In our implementation, only about a dozen floating-
point multiplications and divisions are needed for the filtering computations.
82     Paper C


The memory requirements are minimal, as only the previous estimate and the
new measurement are needed to calculate the new estimate.


7.1.2 Related work
A number of papers have appeared in recent years in the field of available
bandwidth measurements.
    Jain and Dovrolis have developed Pathload [1][2], and Melander et al.
TOPP [3]. Both use probe packet trains sent at various rates, and attempt to
estimate the point of congestion, i.e. the probe rate where the delay starts in-
creasing. TOPP fits data to a straight line in order to arrive at an estimate,
whereas Pathload looks for an increasing trend in the one-way delay, and per-
forms a binary search to successively find shorter and shorter intervals contain-
ing the available bandwidth estimate. Both these methods require a substantial
time for measurement and analysis before producing an estimate, and are not
suitable for real-time tracking of available bandwidth.
    Ribeiro et al. have developed pathChirp [4] for measuring available band-
width. pathChirp uses probe trains with internally varying inter-packet separa-
tions, in order to scan a range of probe rates with each train. By analyzing the
internal delay signature of each such ”chirp”, an estimate of available band-
width is produced. Estimates are smoothed by averaging over a sliding time
window.
    A Kalman filter has previously been applied in the context of some other
types of traffic measurements, but as far as we know, not to the estimation of
available bandwidth.
    In an early paper [5], Keshav discussed using a Kalman filter for the es-
timation of ”bottleneck service rate” for endpoint flow control purposes, and
concluded that this would not be practical. However, his analysis rested on the
assumption that queuing service in network nodes is based on stateful flow-
based round-robin instead of stateless first-come first-served (FCFS), whereas
typically the opposite holds in today’s Internet.
    Jacobsson et al. used a Kalman filter for RTT estimation from the per-
spective of TCP [6]. Kim and Noble did this for estimating another type of
bandwidth over a path [7][8]. However, their definition of bandwidth differs
from ours and from that of [1][2][3][9][10].
    Similarly to the TOPP method, we use the fact that the strain is approxi-
mately proportional to the overload. This means that there is more information
to be gained from each data point than merely whether or not the value indi-
cates overload.
                                 7.2 Measuring available bandwidth            83


     In TOPP, a measurement phase where trains of various rates are sent is
followed by an analysis phase, where linear regression is performed in order
to estimate the parameters of the straight line. A shortcoming of this method
is that it requires a long time before arriving at a measurement value, a value
which then is already ”old” when it is obtained.
     Our original contribution is a technique for applying Kalman filtering, main-
taining and continuously updating the available bandwidth estimate for each
new sampling. This leads to a simple, fast, and easy-to-implement method.
This paper reports on development and validation of the method first described
in [11].


7.1.3 Paper organization
In the next section of this paper, we shall describe the problem of measuring
available bandwidth. We introduce terminology and define the basic concepts.
In the third section, we first explain the network model, and then how BART
can be applied in order to measure available bandwidth in real-time. The fourth
section describes validation of the method, including measurement results from
the Internet as well as a laboratory network, both for synthetic traffic and for a
recorded traffic trace. The fifth section discusses the results, and concludes the
paper.


7.2 Measuring available bandwidth
Whereas several other traffic-dependent network path properties can be mea-
sured directly, such as packet delay and packet loss probability, the measure-
ment of the time-varying available bandwidth over a network path poses more
of a challenge.
    If we are not fortunate enough to have ready access to relevant statistics
from all the network nodes along the path (we almost never are, as the path
typically crosses administrative network domain borders, and transit network
providers are not compelled to share this information), there is no other way to
measure available bandwidth than by active probing.


7.2.1 Definition of available bandwidth
In the literature, the term ”available bandwidth” has been used in different
ways. To avoid any misinterpretations, we want to make clear what we denote
84     Paper C


by available bandwidth. For more details, see the review article [10].
    Each link j in a network path has a certain capacity, or nominal bandwidth,
Cj , determined by the network interfaces in the nodes on each end of the link.
This is simply the highest possible bit rate over the link. The nominal band-
width is quasi-constant, i.e. it typically does not vary.
    What is varying on short time-scale is the link load, or cross traffic, Xj =
Xj (t, ∆). Here, ∆ is the time resolution at which we are interested in describ-
ing traffic fluctuations. So, the cross traffic rate is defined by

                                    1
                             Xj =     Aj [t − ∆, t]                         (7.1)
                                    ∆

    where Aj [t − ∆, t] is the number of cross traffic bits transmitted over link
j during a time interval of length ∆.
    The time-varying available bandwidth Bj = Bj (t, ∆) of link j is defined
as:


                                Bj = Cj − Xj                                (7.2)

    One of the links along the path has the smallest value of available band-
width. This link is called the bottleneck link (or tight link using terminology
from [2]), and it determines the available bandwidth of the path. In other words,
for a network path from sender to receiver, the available bandwidth is defined
as the minimum of the available link bandwidths along the path:


                              B = min(Cj − Xj )                             (7.3)
                                     j


    This is in line with what is denoted by available bandwidth in [1][2][3][9][10].
    An interpretation of the available bandwidth B is: the smallest increase in
traffic load from sender to receiver at time t which causes congestion at some
hop on the network path.
    This interpretation is closely related to the method of measuring the avail-
able bandwidth by sending probe traffic at various rates, and determining the
threshold rate when the probe traffic in conjunction with the cross traffic tran-
siently experiences congestion, i.e., measurement by self-induced congestion.
In fact, one might argue that the rate thus measured can be seen as the definition
of available bandwidth.
7.3 Applying kalman filtering to a piecewise linear network model              85


7.3 Applying kalman filtering to a piecewise lin-
    ear network model
7.3.1 The network model
We model a network path from sender to receiver as a succession of concate-
nated hops. A hop consists of an input queue and a transmission link. We
consider FCFS hops, i.e. first-in first-out (FIFO) queues. Each hop j has a
constant capacity Cj and carries a time-varying cross traffic Xj .
    We will be concerned with the description of interactions between the reg-
ular cross traffic Xj and the injected probe traffic u, in order to arrive at a
formalism which allows drawing conclusions about the available bandwidth
by analyzing the effect that cross traffic has on the probe traffic.
    For a first rough idea, let us consider a fluid model of traffic flows, and tem-
porarily disregard the discrete nature of packet traffic. First, consider a single
hop with link capacity C and cross traffic X. The available link bandwidth is
then C − X. Now, assume that the link is also subject to probe traffic demand
at a rate u. If u ≤ C − X there is no congestion, and the received probe traffic
rate r exiting the link is the same as the demanded probe rate u. However, in
the case u > C − X we have an overload situation. In line with our model
of the hop as having a FIFO queue, we assume that there is no prioritization
in the router, so that all traffic is served on a FCFS basis. This means that the
cross traffic and the probe traffic each receive their proportionate share of the
link capacity C.
    So, we have for the received probe traffic rate:
                                       u
                                 r=        C.                         (7.4)
                                      r+X
    Rearranging, we see that the ratio of demanded to received probe traffic
rate is a first-order polynomial in u:
                                u  1   X
                                  = u+                                     (7.5)
                                r  C   C
   so

                        u          1     (u ≤ C − X)
                           =     1                                       (7.6)
                        r        C + X (u > C − X)
                                      C
    Thus, by varying u and trying to identify the point where u/r starts to
deviate from unity, we have a way of estimating the available bandwidth of the
link. This is essentially the network model proposed for the TOPP method [3].
86     Paper C


    By repeating this argument for several concatenated links, it was shown
in [3] that the available bandwidth of the entire path from sender to receiver
can be estimated by studying the ratio of offered probe rate at the sender to
received probe rate at the receiver, and determining the point where this ratio
starts deviating from unity. This value of u is the estimate of the available
bandwidth B of the entire path.
    Note that when one estimates the parameters of the straight line, one simul-
taneously also gets an estimate of the bottleneck link capacity C, as this is the
inverse of the slope.




Figure 7.1: The asymptotic relation between available bandwidth, probe inten-
sity and expectation value of inter-packet strain.

    Now, considering the discrete (packet-oriented) nature of network traffic, it
is necessary to adapt the above reasoning to take into account the fact that the
traffic is not a continuous flow, but is composed of discrete packets. We use the
generic multiple-hop model presented in [9], and the notation therein.
    Consider a sequence of packet pairs, where the two packets in pair i arrive
at a hop at times τi1 and τi2 and depart (i.e. arrive at the next hop) at times
τi1 and τi2 . We are not interested in the absolute arrival times, but rather the
  ∗        ∗

inter-arrival times of the packets in pair i:

                                 ti = τi2 − τi1                            (7.7)
7.3 Applying kalman filtering to a piecewise linear network model              87


                                 t∗ = τi2 − τi1
                                  i
                                       ∗     ∗
                                                                           (7.8)

    In [9], the focus is on the difference between these quantities, the delay
variation δi = t∗ − ti . Here, we instead introduce the dimensionless quantity
                 i
inter-packet strain ǫi , given by:

                                              t∗
                                               i
                                  1 + ǫi =                                 (7.9)
                                              ti
   The strain can be expressed in terms of the delay variation as:

                                           δi
                                    ǫi =      .                           (7.10)
                                           ti
    This strain provides a direct analogy in the discrete description to the fluid
model ratio of demanded to received traffic rate discussed above. With packet
size s, at the time resolution of inter-packet time separation level we have for
the demanded traffic rate u = s/ti and for the received traffic rate r = s/t∗ .  i
We immediately get:

                           u   s/ti   t∗
                                       i
                             =    ∗ = t = 1 + ǫi                          (7.11)
                           r   s/ti    i

    The deviation of u/r from unity is equivalent to the deviation of the strain
ǫ from zero, see Fig. 7.1. No strain means no congestion.
    It should be noted that when going beyond the fluid model and taking
packet-level interactions in router queues into account, the system measure-
ment model curve is expected to deviate somewhat from the form in Fig.
7.1 [12]. However, we still expect the asymptotic behavior to be the slop-
ing straight line, which is the main point of interest here. The deviation from
this straight line is taken care of by a noise term in our approach.
    In our model, observing the inter-packet strain over the path gives a handle
on estimating the available bandwidth for the tight link, and consequently the
available bandwidth of the entire path. We assume that for the average inter-
packet strain of consecutive probe packet pairs in a sequence of a given rate u,
any systematic deviation from zero is dominated by the effect of cross traffic
interaction in the FIFO queue at the bottleneck link.
    Input data for our BART estimation algorithm is generated by sending se-
quences of N probe packet pairs with the same internal traffic rate u. This way,
we reduce variance and improve the statistical precision in our measurements.
For each new sampling, i.e. new sequence, the rate u is randomized.
88      Paper C


    Each probe packet is time-stamped on sending and on reception. The re-
ceiver calculates the inter-packet strain ǫi for each probe pair i = 1, ..., N .
When the whole sequence has been received, the average inter-packet strain ǫ
and its variance R are computed from ǫi (i = 1, ..., N ).
    Now, how do we compute an estimate of the available bandwidth B using
successive measurements of ǫ and R? While TOPP uses linear regression and
fits a curve to a large number of measurement points, BART keeps a current
estimate, and updates it every time a new probe pair sequence is received, ap-
plying a Kalman filter with a special ”twist”. The BART filter is described in
the following subsection.

7.3.2 Estimation by filtering
Suppose that we have a system, a ”black box”, which is described by a state
vector x. Suppose also that we can affect the system with an input u. We
would like to know x, but the only way we can observe the system is through
measurements z of some aspect of the system.
    In our case, the black box is the network path, u is the probe traffic rate,
and z is the inter-packet strain. The system is also affected by noise w, and the
measurements by noise v. A key idea of BART is the choice of the state vector
x, enabling the Kalman filter estimation of x, and consequently of the available
bandwidth of the network path.
    All these entities, including the state vector, evolve over time. Describing
the evolution of the system by

                              xk+1 = g(xk , uk , wk )                        (7.12)
     and the measurement by

                               zk = f (xk , uk , vk )                        (7.13)
    where f and g are known functions, a filter is a procedure which takes uk ,
                      ˆ                                              ˆ
zk and an estimate xk as inputs, and computes a new estimate xk+1 . Provided
that f (x, u, v) and g(x, u, w) are linear in x, v and w, and that the noise v and w
are Gaussian with zero mean, there is an optimal filter known as a Kalman filter.
For a very brief introduction, see Appendix. A more thorough introduction to
this filter method is given in [13], which also contains derivations of the general
Kalman filter equations we use to calculate our estimates.
    Experience has shown that Kalman filters often work very well, even for
non-Gaussian noise and slightly non-linear models. There are also versions
7.3 Applying kalman filtering to a piecewise linear network model              89


of the Kalman filter that handle non-linear systems by local linearization, al-
though the filter is not optimal for such situations. Unfortunately, they will not
work when the derivative is discontinuous, which is the case we have. We will
return to this problem shortly.
    With the state vector

                                          α
                                 xk =                                     (7.14)
                                          β
    where α and β are the parameters of the sloping straight line in the mea-
surement model, i.e. ǫ = αu + β, our system is described by

                              g(x, u, w) = x + w                          (7.15)
   and our measurements can be described by (see Fig. 7.1):

                                            0        (u ≤ B)
                ǫ = f (x, u, v) = v +                                     (7.16)
                                          αu + β     (u > B)
    In this way we have provided a relation between the available bandwidth B
and the state variables α and β. Estimating the state will allow an estimation
                                                           ˆ      ˆ
of B, which is our objective. When we have estimates α and β we get the
estimate for the available bandwidth:

                                       ˆ
                                       β
                                   ˆ
                                   B=−                                    (7.17)
                                       ˆ
                                       α
    However, the function f (x, u, v) above is only piecewise linear. The sharp
corner at u = B prevents a direct application of a Kalman filter. In order to
circumvent this, we apply the filter only to measurements for which u > B      ˆ
where B ˆ is the current estimate of the available bandwidth, which is known.
We argue this method will converge if f is a convex function of u, as is the
case in our method.
    The convergence can intuitively be understood as follows: Given only
points for which u > B the Kalman filter attempts to find a straight-line ap-
proximation L1 to the convex curve ǫ(u), consider Fig. 7.3.2. The intersection
with the u-axis approximates the available bandwidth B. We do not know B,
                                                       ˆ
but suppose for now that we have an underestimate Bk < B. The next time
                                                                    ˆ
the Kalman filter is applied, we use only points for which u > Bk , and the
Kalman filter will attempt to find a new line L2 . Due to ǫ(u) being convex,
                                               ˆ           ˆ       ˆ
this line will intersect the u-axis at a point Bk+1 where Bk < Bk+1 < B,
90       Paper C




                 Figure 7.2: Convergence of the BART method.


showing that the new value in the mean will be an improved approximation.
                                                 ˆ
An analogous argument holds for an over-estimate Bk > B.
   The BART algorithm proceeds as follows:

                                                           ˆ
     1. The receiver initializes the state vector estimate x, the available band-
                        ˆ
        width estimate B and the error covariance matrix P for x.ˆ

     2. The sender generates a sequence of probe packet pairs with probe traffic
        intensity u, drawn from a probability distribution. The sender passes the
        current value of the intensity in the probe packets.

                                                                           ˆ
     3. For each received probe sequence, the receiver recovers u. If u < B, no
                                                                          ˆ
        updating is performed and the cycle repeats from step 2. If u > B, the
        receiver computes the average strain ǫ and its variance R.

     4. The receiver inputs these values to the Kalman filter, and also provides
        the filter with an estimate Q of the process noise covariance matrix (cf.
                                                                         ˆ
        below). The filter then updates the estimates of the state vector x and the
        matrix P .
                                          7.4 Validation of the method          91


                                    ˆ
   5. The receiver uses the updated u to compute a new u-axis crossing, pro-
                                                 ˆ
      ducing a new available bandwidth estimate B. The cycle repeats from
      step 2.


    One consideration when applying Kalman filters is finding a suitable Q.
This matrix describes the intrinsic fluctuations in the system, and is in our
application related to the volatility of the cross traffic. This can not be assumed
to be known a priori, but can be treated as an adjustable parameter. This enables
tuning of the BART estimation characteristics. Larger values result in low
stability but high agility of the filter, since the Kalman filter equations [13]
give greater weight to the recent measurement relative to the previous estimate.
Conversely, smaller values result in higher stability in presence of spurious
measurement errors, but slower step response [14].
    By definition, Q is a symmetric matrix. In our case, it is a 2x2-matrix, so
it contains 3 independent scalar parameters. In this implementation we have
taken the simple form:

                                            λ   0
                             Q = λI =                                       (7.18)
                                            0   λ
    As is seen in the next section, reasonable performance can be produced
without exploiting the full potential of the formalism. For all traffic cases in
the present paper, the BART estimation results have been produced using this
simple form for Q, with λ = 10−4 . There is plenty of freedom to relax this
constraint, allowing tuning for desired estimation performance.


7.4 Validation of the method
In order to evaluate the method, the performance of a BART implementation
has been measured in a controlled testbed environment as well as over the In-
ternet. Numerous simulations have also been performed, which corroborate the
evidence from the measurements, but these simulations are not described in the
present paper. We compare the performance of BART with that of pathChirp, a
state-of-the-art tool for measuring end-to-end available bandwidth in real-time
[4].
     In this implementation of BART, the sequence of probe packet pairs was
structured as a train, i.e. the second packet in one pair also constitutes the first
packet in the next pair. This reduces the traffic overhead, at the expense of
92         Paper C


statistical accuracy. We consider it conceivable that better performance can be
obtained if relaxing this constraint, although at a higher overhead. This will be
reported on in a forthcoming paper.
    The discussion of the measurement results presented in this section leads to
the conclusion that BART yields estimates with equal or better accuracy than
pathChirp, with lower overhead of probe traffic injected into the network.
    Part 9.1 of this section describes the experiment setup, including the testbed
and the different traffic cases. Thereafter, we study the impact that probe packet
size and probe train length have on BART performance. The remaining parts
of the section present results for BART and pathChirp in several experiments,
both in the testbed and measurements over the Internet. Observe that through-
out this section, measured data rates, available bandwidth estimates and packet
sizes refer to the network layer.


7.4.1 Measurement testbed setup
A schematic picture of the testbed is shown in Fig. 7.3. It contains two routers,
the interconnection of which is the bottleneck link in the system. The testbed
also includes senders and receivers for each probing tool and for the cross
traffic. In order to monitor the true network traffic, a machine running tcpdump
(a tool for logging network traffic) has been connected to hubs. The hosts run
Linux, and are equipped with processors equal to or better than Pentium III/500
MHz.
    All links in the testbed have a capacity of 100 Mbit/s, except the bottleneck
link between router 1 and router 2, which is nominally configured to 10 Mbit/s.
However, the actually observed capacity was 9.55 Mbit/s at the IP layer. Hence,
this value has been used as the true bottleneck capacity of the testbed.
    In the testbed measurements, both synthetic cross traffic, generated by IP-
TrafGen (an Ericsson proprietary traffic generator), and a traffic trace from
MAWI1 have been used. In the Internet measurements, additional cross traffic
that originates from the same traffic trace has been applied in order to generate
extra load on the Internet path.
    Unless otherwise specified, BART has used probe packets of 1500 bytes
and probe trains of 17 probe packets in all the experiments. As we see in sec-
tion 7.4.2, the accuracy of BART increases with increasing probe packet size
and with increasing probe train length. The probe packet size is limited by
the Ethernet maximum transmission unit. We chose to limit the train length
     1 http://tracer.csl.sony.co.jp/mawi/   (August 2005)
                                         7.4 Validation of the method          93




          Figure 7.3: A schematic view of the measurement testbed.


to 17 probe packets, as this gives reasonable accuracy while not causing ex-
cessive probe traffic overhead. The traffic intensity for each probe train was
randomly chosen from a uniform distribution, over the interval from 1 Mbit/s
to 20 Mbit/s. BART was configured to produce an estimate every second, i.e.
the inter-departure time between two consecutive probe trains is one second.
This implies that BART injects probe traffic with an average intensity of 0.204
Mbit/s at the IP layer, which is approximately 2.1% of the bottleneck link ca-
pacity in the testbed.


          Table 7.1: Distribution of synthetic cros traffic packet sizes
                    Packet Size: Percentage:
                          40 B               30.23%
                          52 B               18.75%
                         576 B               10.10%
                        1420 B               15.15%
                        1500 B               25.77%


    The default parameters have been used for pathChirp. Our experiments
indicate that pathChirp is producing 1.55 estimates per second with a probe
traffic intensity around 0.294 Mbit/s, i.e. in the region of 3.1% of the bottleneck
link capacity in the testbed.
    For the testbed measurements using synthetic cross traffic, the IPTrafGen
generator has been configured for measurements with high and low aggregation
94         Paper C


of cross traffic. The two scenarios can approximately be seen as having 10 or
100 users simultaneously utilizing a network link. New users arrive according
to a Poisson process and remain active during a period obtained from a Pareto
distribution (shape parameter α = 1.5, mean = 1.0 second). When the users are
active, they transmit packets with sizes according to Table 7.1. These packet
sizes refer to IP datagrams in bytes, and the distribution roughly corresponds
to observations from Sprint2 .
    Measurements with synthetic cross traffic have been made using three dif-
ferent distributions for the inter-arrival time of the cross traffic packets. These
are exponential inter-arrivals and Pareto inter-arrivals with shape parameter α
= 1.9 and α = 1.6, respectively. For all inter-arrival scenarios, the distribution
parameters have been chosen such that active users approximately get the same
transmission rate, and the expectation value of the overall cross traffic intensity
is 5 Mbit/s.
     To generate cross traffic from a traffic trace, we have used tcpreplay3. tcpre-
play sends packets from a recorded tcpdump log file. In our measurements, the
cross traffic generated by tcpreplay originates from real Internet links.
     The traffic trace from MAWI, which we use in our experiments, was cap-
tured during 15 minutes starting at 14.00 hours on April 21, 2005. The average
bit rate was 26.07 Mbit/s. In our testbed, the bottleneck link has a capacity of
9.55 Mbit/s, as described above. Thus, in order to use the trace obtained from
MAWI, the traffic intensity has been scaled down to suit the testbed (one of the
features of tcpreplay). Most cross traffic packets in the trace have a size less
than 100 bytes while the larger packets, between 1400 and 1500 bytes, contain
the majority of the transferred bits.

7.4.2 Impact of probe packet size and probe train length
This subsection discusses the impact of the probe packet size and the probe
train length on the available bandwidth estimates produced by BART. The mea-
surements have been performed in the testbed, using tcpreplay to generate cross
traffic from a traffic trace.
    We have used probe packet sizes of 300, 900 and 1500 bytes at the IP layer,
and probe train lengths of 9, 17, 26 and 33 packets (for each probe packet size).
    From these experiments, we have observed that the accuracy of BART in-
creases both with the probe packet size and the probe train length, see Fig. 7.4
- 7.7.
     2 http://ipmon.sprintlabs.com/packstat/packetoverview.php   (August 2005)
     3 http://tcpreplay.sourceforge.net/   (August 2005)
                                                             7.4 Validation of the method          95



                                      10


                                       9


                                       8


                                       7
       Available bandwidth [Mbit/s]




                                       6


                                       5


                                       4


                                       3


                                       2


                                       1
                                                                           Available bandwidth
                                                                           BART estimate
                                       0
                                           0   50   100     150      200       250           300
                                                          Time [s]




Figure 7.4: 300 bytes probe packet size, 9 probe packets per train.


                                      10


                                       9


                                       8


                                       7
       Available bandwidth [Mbit/s]




                                       6


                                       5


                                       4


                                       3


                                       2


                                       1
                                                                           Available bandwidth
                                                                           BART estimate
                                       0
                                           0   50   100     150      200       250           300
                                                          Time [s]




Figure 7.5: 300 bytes probe packet size, 33 probe packets per train.
96     Paper C



                                            10


                                             9


                                             8


                                             7
             Available bandwidth [Mbit/s]




                                             6


                                             5


                                             4


                                             3


                                             2


                                             1
                                                                                 Available bandwidth
                                                                                 BART estimate
                                             0
                                                 0   50   100     150      200       250           300
                                                                Time [s]




     Figure 7.6: 1500 bytes probe packet size, 9 probe packets per train.


                                            10


                                             9


                                             8


                                             7
             Available bandwidth [Mbit/s]




                                             6


                                             5


                                             4


                                             3


                                             2


                                             1
                                                                                 Available bandwidth
                                                                                 BART estimate
                                             0
                                                 0   50   100     150      200       250           300
                                                                Time [s]




     Figure 7.7: 1500 bytes probe packet size, 33 probe packets per train.
                                          7.4 Validation of the method           97


    The figures below show the essence of our findings by illustrating four dif-
ferent cases. In Fig. 7.4, the probe packet size is 300 bytes and the number of
probe packets per train is 9. This is a weak constellation, and the estimates of
BART almost directly increase toward 9.0 Mbit/s, corresponding to an almost
empty link. This is due to statistical effects, which we will report on in a forth-
coming paper. Fig. 7.5 shows that the performance improves if more probe
packets are included in each train, but still the quality of the estimates are un-
acceptable. The same conclusion holds for Fig. 7.6, which illustrates the result
when larger probe packets have been used. Fig. 7.7 depicts a measurement
result when larger probe packets in combination with an increased probe train
length have been used. The accuracy improves, and BART provides estimates
in good agreement with the true available bandwidth.
    Hence, this BART implementation will most likely overestimate the avail-
able bandwidth when using small probe packets and short probe packet trains.
In these experiments, the cross traffic, generated from the traffic trace, is bursty.
Consequently, the length of the probe train and the probe packet size are cru-
cial for the quality of the available bandwidth estimates produced by BART. An
optimization of the tool could be to tune these parameters on the basis of ob-
served variations in the timestamps of received probe packets (i.e., essentially
tune them indirectly based on the cross traffic characteristics).

7.4.3 Comparison of BART and pathChirp in the Testbed
      Environment
In this subsection, results will be presented showing the behavior of BART and
pathChirp when synthetic cross traffic has been used in the testbed.
    In total, six different cross traffic scenarios have been studied in the testbed.
For all these experiments, the measurements have been running for slightly
more than 2000 seconds; making it possible to verify the behavior in the long
term, e.g. consider Fig. 7.8. However, in order to provide clear illustrations, the
majority of the following figures only show 300 seconds of the measurements,
see Fig. 7.9 - 7.14. The used interval (1000 - 1300 seconds) was arbitrarily
chosen and does not significantly deviate from other 300 second intervals. Each
figure depicts the estimates of BART and pathChirp. In addition, a thin solid
curve indicates the actual available bandwidth. The available bandwidth is
obtained by subtracting the average cross traffic from the measured capacity
of the bottleneck link in the testbed. The average cross traffic is calculated by
using a sliding window, which covers three seconds of cross traffic caught by
the tcpdump machine.
98     Paper C



                                            10


                                             9


                                             8


                                             7
             Available bandwidth [Mbit/s]




                                             6


                                             5


                                             4


                                             3


                                             2


                                             1                                                              Available bandwidth
                                                                                                            pathChirp estimate
                                                                                                            BART estimate
                                             0
                                                 0   200     400   600     800    1000      1200     1400   1600      1800    2000
                                                                                 Time [s]




Figure 7.8: Long-term behavior of approximately 100 active users with expo-
nential inter-arrivals between cross traffic packets.

                                            10


                                             9


                                             8


                                             7
             Available bandwidth [Mbit/s]




                                             6


                                             5


                                             4


                                             3


                                             2


                                             1                                                              Available bandwidth
                                                                                                            pathChirp estimate
                                                                                                            BART estimate
                                            0
                                            1000           1050     1100          1150             1200        1250           1300
                                                                                 Time [s]




Figure 7.9: High cross traffic aggregation (100 users) with exponential inter-
arrivals between cross traffic packets.
                                                                    7.4 Validation of the method            99



                                            10


                                             9


                                             8


                                             7
             Available bandwidth [Mbit/s]




                                             6


                                             5


                                             4


                                             3


                                             2


                                             1                                     Available bandwidth
                                                                                   pathChirp estimate
                                                                                   BART estimate
                                            0
                                            1000   1050   1100    1150      1200      1250           1300
                                                                 Time [s]




Figure 7.10: Low cross traffic aggregation (10 users) with exponential inter-
arrivals between cross traffic packets.

                                            10


                                             9


                                             8


                                             7
             Available bandwidth [Mbit/s]




                                             6


                                             5


                                             4


                                             3


                                             2


                                             1                                     Available bandwidth
                                                                                   pathChirp estimate
                                                                                   BART estimate
                                            0
                                            1000   1050   1100    1150      1200      1250           1300
                                                                 Time [s]




Figure 7.11: High cross traffic aggregation (100 users) with Pareto (α = 1.9)
inter-arrivals between cross traffic packets.
100     Paper C


    Fig. 7.8 shows the long-term behavior of BART and pathChirp when
the cross-traffic generator is modeling aggregated traffic of around 100 active
users, where each user is transmitting packets with exponential inter-arrivals.
    In Fig. 7.9, only 300 seconds of Fig. 7.8 is depicted. It is quite clear
that pathChirp over-estimates somewhat, whereas BART estimates the avail-
able bandwidth more accurately.

                                            10


                                             9


                                             8


                                             7
             Available bandwidth [Mbit/s]




                                             6


                                             5


                                             4


                                             3


                                             2


                                             1                                     Available bandwidth
                                                                                   pathChirp estimate
                                                                                   BART estimate
                                            0
                                            1000   1050   1100    1150      1200      1250           1300
                                                                 Time [s]




Figure 7.12: Low cross traffic aggregation (10 users) with Pareto (α = 1.9)
inter-arrivals between cross traffic packets.


   Inter-arrival distribution:                                               BART:            pathChirp:
           Exponential                                                      0.0044              0.0496
         Pareto (α = 1.9)                                                   0.0044              0.0467
         Pareto (α = 1.6)                                                   0.0062              0.0487

Table 7.2: Normalized MSE of BART and pathChirp estimates when using
high cross traffic aggregation and three different packet inter-arrival distribu-
tions.


    Fig. 7.10 illustrates the performance of BART and pathChirp when the
cross traffic is less aggregated. 10 active users make the available bandwidth
more fluctuating, and it becomes harder to produce accurate estimates. BART
                                                                  7.4 Validation of the method              101



                                            10


                                             9


                                             8


                                             7
             Available bandwidth [Mbit/s]




                                             6


                                             5


                                             4


                                             3


                                             2


                                             1                                     Available bandwidth
                                                                                   pathChirp estimate
                                                                                   BART estimate
                                            0
                                            1000   1050   1100    1150      1200      1250           1300
                                                                 Time [s]




Figure 7.13: High cross traffic aggregation (100 users) with Pareto (α = 1.6)
inter-arrivals between cross traffic packets.


gives decent estimates, although it does not follow the rapid changes in the
available bandwidth. This corresponds to the choice of Q. Cf. the discussion
in section 7.3.2. As previously, pathChirp is overestimating.
    Results from measurements using Pareto inter-arrivals with infinite vari-
ance (α = 1.9) can be seen in Fig. 7.11. High aggregation makes it easy for
BART to produce accurate estimates, while pathChirp shows a fluctuating be-
havior with overestimated values.
    In Fig. 7.12, we can again see that less aggregated cross traffic makes
it harder to estimate the available bandwidth. However, BART still appears


   Inter-arrival distribution:                                               BART:            pathChirp:
           Exponential                                                      0.0207              0.0461
         Pareto (α = 1.9)                                                   0.0215              0.0499
         Pareto (α = 1.6)                                                   0.0146              0.0511

Table 7.3: Normalized MSE of BART and pathChirp estimates when using low
cross traffic aggregation and three different packet inter-arrival distributions.
102      Paper C



                                             10


                                              9


                                              8


                                              7
              Available bandwidth [Mbit/s]




                                              6


                                              5


                                              4


                                              3


                                              2


                                              1                                     Available bandwidth
                                                                                    pathChirp estimate
                                                                                    BART estimate
                                             0
                                             1000   1050   1100    1150      1200      1250           1300
                                                                  Time [s]




Figure 7.14: Low cross traffic aggregation (10 users) with Pareto (α = 1.6)
inter-arrivals between cross traffic packets.


to provide more stable estimates, which are on average closer to the actual
available bandwidth.
    Fig. 7.13 illustrates the estimates in the case of approximately 100 active
users and inter-arrivals between cross traffic packets following a Pareto distri-
bution with infinite variance (α = 1.6). The estimates are similar to the previous
cases with high cross traffic aggregation, i.e. BART is smooth and fairly accu-
rate whereas pathChirp is overestimating and displays a high variance.
    In Fig. 7.14, we can again make the same observations as for the other
low-aggregation traffic cases. However, it could be worth noticing the sharp
increases of the BART estimate after approximately 1200 seconds. In the cur-
rent implementation of BART, this is sometimes seen to occur, more markedly
the more bursty the cross traffic is. An interpretation is that when a high-rate
probe train happens to largely miss the cross traffic, this results in an underes-
timation of the strain ǫ, while at the same time producing a low value for the
variance R. The result is a sharp increase in the estimate of available band-
width. This is why we consider it interesting to decouple the probe pairs from
a train to a sequence of independent pairs in a future implementation.
    The mean squared error (MSE) has been calculated for the six cases of
synthetic cross traffic. See Table 7.2 and Table 7.3. The MSE is normalized
                                        7.4 Validation of the method         103


with respect to the bottleneck link capacity, and for each MSE computation,
estimates during 2000 seconds have been considered. Furthermore, the initial
few seconds of the measurements have been excluded, as we wished to describe
the ”steady-state”-properties, and not let variations caused by arbitrary initial
values influence the results. As a benchmark for the estimates, the previously
computed available bandwidth was used with a minor correction. In the figures
illustrating the testbed measurements, the available bandwidth curve is only
based on the generated cross traffic, although the tools producing estimates
also observe a minor amount of interfering probe traffic. In order to improve
the quality of the MSE calculations, the available bandwidth benchmark has
been revised, such that BART MSE computations take the pathChirp probe
traffic into account, and vice versa.
    Based on the values in Table 7.2 and 7.3, it can be concluded that BART
shows better performance in case of high aggregated cross traffic, compared to
low aggregation. The reason for a higher MSE in case of low aggregation is that
BART does not follow the rapid changes in the available bandwidth. However,
since BART is tunable, it is possible to improve the agility, simply by adjusting
the estimate of the process noise covariance matrix Q. The pathChirp MSE is
higher in all traffic cases, mainly due to overestimation. The reason for this
overestimation is yet unknown, although a similar observation has been made
in [15].

7.4.4 Comparison of BART and pathChirp using an Internet
      Path
In this subsection, the behavior of BART and pathChirp is illustrated in two
measurements performed on the Internet. The experiments were performed
over an Internet path in Sweden, consisting of 16 layer-3 hops, according to
traceroute. Probe traffic was transmitted from a host at Linkping University,
which connects to the Swedish University Network (SUNET), and received by
a host at Ericsson Research in Stockholm, connected to the Internet service
provider Telia. All the hosts in the experiments were equipped with processors
equal to or better than Pentium III/650 MHz, running Linux (Fedora Core 3).
They were provided with 100 Mbit/s Ethernet connections to the Internet.
    The probe traffic rate was randomly chosen from a uniform distribution,
over the interval from 10 Mbit/s to 100 Mbit/s. As in the previous experiments,
the estimated process noise covariance matrix Q was assigned the value of
10−4 on the diagonal, whereas the other elements were set to zero. BART
transmitted one probe train every second (17 probe packets per train), i.e. the
104       Paper C


probe traffic intensity was 0.204 Mbit/s.

                                                80



                                                70



                                                60
                 Available bandwidth [Mbit/s]




                                                50



                                                40



                                                30



                                                20



                                                10                                             Available bandwidth
                                                                                               pathChirp estimate
                                                                                               BART estimate
                                                 0
                                                     0   50   100     150     200      250   300      350        400
                                                                            Time [s]




Figure 7.15: Internet experiments with a smoothly changing cross traffic inten-
sity.

     The default parameters were used for pathChirp, except for the -J parame-
ter, which sets the number of packets per Jumbo packet (default = 1)4 . In the
Internet measurements, pathChirp was using two packets per Jumbo packet,
thus approximately one estimate per second was produced, which is in line
with the behavior of BART. pathChirp loads the network with a probe traffic
intensity around 0.294 Mbit/s.
     Immediately before the Internet experiments were activated, the through-
put between the probe sender at Linkping University and the probe receiver
at Ericsson Research was measured using Iperf 5 . Several measurements were
conducted, and the available bandwidth turned out to be fairly stable around
60 Mbit/s (at the IP layer). In the evaluation of the estimation performance of
BART and pathChirp, we made use of this value as a ”steady state” available
bandwidth reference.
     In order to affect the cross traffic on the path between Linkping University
and Ericsson Research, tcpreplay was used to generate additional traffic during
the experiments. The extra generated traffic was based on the previously used
  4 http://www.spin.rice.edu/Software/pathChirp/                              (August 2005)
  5 http://dast.nlanr.net/Projects/Iperf/                           (August 2005)
                                                                                       7.4 Validation of the method                     105



                                            80



                                            70



                                            60
             Available bandwidth [Mbit/s]




                                            50



                                            40



                                            30



                                            20



                                            10                                                                  Available bandwidth
                                                                                                                pathChirp estimate
                                                                                                                BART estimate
                                             0
                                             400       450         500         550      600       650         700         750     800
                                                                                      Time [s]




                                                     Figure 7.16: Continuation of Fig. 7.15.


                                            80



                                            70



                                            60
             Available bandwidth [Mbit/s]




                                            50



                                            40



                                            30



                                            20



                                            10                                                                  Available bandwidth
                                                                                                                pathChirp estimate
                                                                                                                BART estimate
                                             0
                                                 0    50     100         150    200     250      300    350         400    450    500
                                                                                      Time [s]




Figure 7.17: Internet experiments with a rapidly changing cross traffic inten-
sity.
106      Paper C



                                             80



                                             70



                                             60
              Available bandwidth [Mbit/s]




                                             50



                                             40



                                             30



                                             20



                                             10                                                    Available bandwidth
                                                                                                   pathChirp estimate
                                                                                                   BART estimate
                                              0
                                              500   550   600   650   700     750      800   850    900     950      1000
                                                                            Time [s]




                                                    Figure 7.18: Continuation of Fig. 7.17.


traffic trace from MAWI, using a time-dependent re-scaling factor to produce
the available-bandwidth characteristics of Fig. 7.15 - 7.18.
    By subtracting the known added traffic intensity from the throughput es-
timate of Iperf, we constructed a reference available bandwidth along the In-
ternet path, to use as a benchmark for BART and pathChirp. Of course, the
true available bandwidth is perturbed by unknown time-dependent traffic vari-
ations, and it should be kept in mind that the reference curve in Fig. 7.15 - 7.18
cannot reflect this perturbation.
    The results of the first experiment are depicted in Fig. 7.15 and 7.16. The
traffic load generated by tcpreplay increases from 0 Mbit/s up to approximately
33 Mbit/s during a period of 275 seconds. After that, the added traffic is rather
constant for 225 seconds, until it slowly decreases back to zero.
    Both BART and pathChirp seem to follow the general trend of the avail-
able bandwidth during this experiment, even if it might be argued that BART
has a lower variability and is somewhat better at following the mean available
bandwidth.
    The second experiment studied rapid changes of the additional cross traffic
load generated by tcpreplay. As can be seen in Fig. 7.17, the cross traffic
generator was configured to suddenly increase the traffic intensity after 110
seconds, approximately 10 Mbit/s. Another change occurs after 825 seconds,
                                    7.5 Discussion and conclusions         107


see Fig. 7.18. The cross traffic generator goes off and the available bandwidth
instantly increases with roughly 33 Mbit/s.
    BART is seen to clearly respond to these sudden changes in available band-
width, while the same is not evident for pathChirp.


7.5 Discussion and conclusions
We have presented BART, a novel method for real-time estimation of available
bandwidth over a network path using Kalman filtering. We have demonstrated
that Kalman filtering is applicable to the problem of measuring available band-
width, despite a strong nonlinearity in the system measurement model. We
have also shown, by experiments using an implementation of BART both in
a laboratory network and over the Internet, that reasonable accuracy can be
obtained with little computational efforts and with low cost in terms of extra
traffic load on the network from the probe packets.
    The method has considerable potential with regard to tuning for better per-
formance. Notably, the process covariance matrix Q, which is an input to
the BART estimation algorithm, contains three degrees of freedom, whereas
we have only started exploiting one in the measurements reported in this pa-
per. Also, it is conceivable that the estimation performance could be increased
by allowing the probe packet pairs in a sequence to be transmitted indepen-
dently of each other. Nevertheless, we have already with this implementation
of BART observed reasonable accuracy of the available bandwidth estimation
in a wide variety of traffic cases.
    We conclude that Kalman filtering provides a promising engine for fast and
efficient estimation of available bandwidth.


7.6 Appendix
The Kalman filter is a numerical method for sequentially updating and improv-
ing an estimation of the state of a system, based on a sequence of measure-
ments. For linear systems where the process noise and measurement noise are
Gaussian, the Kalman filter has been shown to be the optimal estimator, in the
sense that it minimizes the expectation value of the norm of the error. For de-
tails regarding this, as well as regarding the remainder of this Appendix, see
[13] and references therein.
     In our application of estimating network properties like available band-
width, we do not have the detailed knowledge of the system that would allow
108      Paper C


the method to guarantee optimality. However, by making educated guesses or
by treating some quantities as tunable parameters, the Kalman filter can be ap-
plied with good results even when some of the system properties are unknown.
    Our network model system is not linear. However, it is asymptotically lin-
ear in the region of interest (the region of congestion), so we solve the problem
by applying a Kalman filter in our estimate of this region.
    In general, a system suitable for Kalman filter estimation is described by a
state vector x ∈ Rn , the evolution of which is governed by a linear stochastic
difference equation of the type:

                        xk = Axk−1 + Buk−1 + wk−1                            (7.19)
    Here, u is the control input, and w is the process noise. Ideally, w has a
Gaussian probability distribution N (0, Q), where Q is the process noise co-
variance matrix.
    Information on the state of the system is gained by performing a series of
measurements z ∈ Rm , which depend linearly on the system state vector:

                                 zk = Hxk + vk                               (7.20)
    Here, the measurement noise v ideally has a Gaussian probability distribu-
tion N (0, R), where R is the measurement noise covariance matrix.
    The objective is to use the measurements to track the state of the system, i.e.
estimate the state vector xk as it evolves in discrete time (k = 0, 1, 2, ...). This
is achieved by doing two operations in each time step: first using the process
equation to project the latest estimate ahead in time (”prediction”), then using
the latest measurement to adjust this predicted state (”correction”).
    To do this we define the a priori state estimate x− , as well as the a posteriori
                                                     ˆk
               ˆ
state estimate xk , the latter of which also taking the measurement zk at step k
into account.
    We also need to compute at each step the a priori and a posteriori state
estimate error covariance matrices
                                 −
                                Pk = E(e− e−T )
                                        k k                                  (7.21)
                                 Pk =   E(ek eT )
                                              k                              (7.22)
    where e− is the a priori and ek is the a posteriori state error:
           k

                                           ˆk
                                 e− = xk − x−
                                  k                                          (7.23)
                                            ˆ
                                  ek = xk − xk                               (7.24)
                                                     7.6 Appendix       109


    The crucial point of the Kalman formalism is how to make the best of the
measurements. This is the ”correction” step mentioned above. The optimal
correction is

                        xk = x− + Kk (zk − H x− )
                        ˆ    ˆk              ˆk                       (7.25)
   where the Kalman gain Kk is given by:

                     Kk = Pk H T (HPk H T + R)−1
                           −        −
                                                                      (7.26)
   To sum up, the Kalman filter equations are:

                          xk = Axk−1 + Buk−1                          (7.27)

                          Pk = APk−1 AT + Q
                           −
                                                                      (7.28)

                     Kk = Pk H T (HPk H T + R)−1
                           −        −
                                                                      (7.29)

                        ˆ    ˆk              ˆk
                        xk = x− + Kk (zk − H x− )                     (7.30)

                           Pk = (I − Kk H)Pk
                                           −
                                                                      (7.31)
   Bibliography




Bibliography

[1] Manish Jain and Constantinos Dovrolis. Pathload: a measurement tool
    for end-to-end available bandwidth. In Proceedings of the Passive and
    Active Measurement Workshop, Ft Collins, USA, March 2002.

[2] Manish Jain and Constantinos Dovrolis. End-to-end available bandwidth:
    Measurement methodology, dynamics, and relation with TCP throughput.
    In Proceedings of the ACM SIGCOMM conference, Pittsburg, USA, Au-
    gust 2002.

                          o
[3] Bob Melander, Mats Bj¨ rkman, and Per Gunningberg. A new end-to-end
    probing and analysis method for estimating bandwidth bottlenecks. In
    Proceedings of IEEE Globcom, San Francisco, November 2000.

[4] V. Ribeiro, R. Riedi, R. Baraniuk, J. Navratil, and L. Cottrel. pathChirp:
    Efficient available bandwidth estimation for network paths. In Proceed-
    ings of the Passive and Active Measurement Workshop, San Diego, USA,
    2003.

[5] Srinivasan Keshav. A control-theoretic approach to flow control. In Pro-
                                                u
    ceedings of ACM SIGCOMM, pages 3–15, Z¨ rich, Switzerland, Septem-
    ber 1991.

                                        o
[6] K. Jacobsson, H. Hjalmarsson, N. M¨ ller, and K. H. Johansson. Estima-
    tion of rtt and bandwidth for congestion control applications in commu-
    nication networks. In IEEE Conference on Decision and Control (CDC),
    Paradise Island, Bahamas, 2004.

[7] M. Kim and B. Noble. Sane: stable agile network estimation. In Uni-
    versity of Michigan department of Electrical Engineering and Computer
    Science, CSE-TR-432-00, 2000.

                                    110
 [8] M. Kim and B. Noble. Mobile network estimation. In Proc. Mobile
     Computing and Networking (ACM MOBICOM), Rome, Italy, 2001.
              a
 [9] Attila P´ sztor and Darryl Veitch. The packet size dependence of packet
     pair like methods. In Tenth International Workshop on Quality of Service,
     Miami Beach, USA, May 2002.
[10] R.S. Prasad, M. Murray, C. Dovrolis, and K. Claffy. Bandwidth estima-
     tion: metrics, measurement techniques, and tools. IEEE Network Maga-
     zine, 2003.
[11] Svante Ekelin and Martin Nilsson. Continuous monitoring of available
     bandwidth over a network path. In Proceedings of the 2nd Swedish
     National Computer Networking Workshop (SNCNW), Karlstad, Canada,
     2004.
[12] Xiliang Liu, Kaliappa Ravindran, and Dmitri Loguinov. Multi-hop prob-
     ing asymptotics in available bandwidth estimation: Stochastic analysis. In
     Proceedings of the ACM SIGCOMM Internet Measurement Conference,
     Berkeley, USA, 2005.
[13] G. Bishop and G. Welch. An introduction to the kalman filter. In SIG-
     GRAPH 2001, Course 8, 2001.
[14] Erik Hartikainen, Svante Ekelin, and J. M. Karlsson. Adjustment of the
     bart kalman filter to improve real-time estimation of end-to-end available
     bandwidth. In Proceedings of the 3rd Swedish National Computer Net-
     working Workshop (SNCNW), Halmstad, Sweden, 2005.
[15] A. Shriram, M. Murray, Y. Hyun, N. Brownlee, A. Broido, M. Fomenkov,
     and K. Claffy. Comparison of public end-to-end bandwidth estimation
     tools on high-speed links. In Proceedings of the Passive and Active Mea-
     surement workshop, Boston, USA, 2005.
Chapter 8

Paper D:
Measuring the Impact of
Active Probing on TCP

                             o
Andreas Johnsson and Mats Bj¨ rkman
In proceedings of the SCS International Symposium on Performance Evalu-
ation of Computer and Telecommunication Systems (SPECTS’06), Calgary,
Canada, July 2006.




                                 113
                                   Abstract

Available bandwidth measurement methods have become more and more ac-
cepted to be used when seeking the status of a network path. To measure the
end-to-end available bandwidth without access to the path routers, these meth-
ods inject UDP based probe packets into the network path. The probe-packet
load can transiently be high and thus it is important to study the impact on the
existing network flows.
    In this paper, we show and discuss our simulation results on how the TCP
flows are affected when injecting probe packets with different flight patterns
into the network path. We investigate the relation between the amount of in-
jected probe packets and the reduction in TCP performance. Further, we sug-
gest a quantitative definition of the term “network friendly probing”.
                                                                 8.1 Introduction             115


8.1 Introduction

Active measurement methods inject so called probe packets into the network
path in order to measure network properties. The range of active measure-
ment methods spans from simple ping applications, that inject a small amount
of probe packets, to advanced end-to-end available bandwidth1 measurement
methods that require more measurement samples (referred to as samples in the
rest of the paper) and thus inject more probe packets. There are many ex-
amples of the latter, such as ABGET [1], BART [2], Pathchirp [3], Pathload
[4], Spruce [5] and TOPP [6][7]. Available bandwidth measurement methods
typically rely on self-induced congestion. That is, the methods inject UDP
based probe packets into the network path with a predefined packet separation
to cause network congestion for a short time interval. During the probe-packet
traversal of the network, the predefined separation changes due to competing
cross traffic [8][9] or due to the bottleneck spacing effect [6]. The probe-packet
separation is measured at a receiver. The separation between two probe packets
constitutes a sample. These samples are then used in an estimation algorithm
to calculate the end-to-end available bandwidth. The estimates of the available
bandwidth can later be used in applications that rely on properties of the path.
A good overview of methods is presented in [10].
    Efforts have been made previously to develop end-to-end available band-
width measurement methods and to compare their properties, such as accuracy
and speed. However, in this paper we focus on how the cross traffic (i.e. all
packets in the network but the probe packets) is affected by the probe packets.
When the cross-traffic flow is adaptive, that is the send rate depends on network
latency and packet loss, the injected probe packets can adversely decrease the
performance of such flows.
    In this paper we show and discuss our results on how one important dy-
namic protocol, TCP, is affected when injecting probe-packet trains with dif-
ferent lengths and intensities into the network path. That is, injecting the probe
packets with different flight patterns. Within this study the relation between
the number of injected probe packets and the reduction in TCP performance is
also examined. These findings then evolve into a discussion and definition of
the term “network friendly probing”.


   1 The available bandwidth is defined as the unused portion of the bottleneck link capacity during

some time interval. The estimate is independent on the network protocols used.
116     Paper D


8.2 Research questions and experimental setup
The aim of this paper is to show and discuss how UDP based probe packets
sent in varying flight patterns affect TCP flows, the major transport protocol
used on the Internet. The research questions are: “How does network probing
affect TCP performance?” and “what is network friendly probing?”.
    NS-2 has been used to study the above research questions. A network
topology was constructed, shown in Figure 8.1, which consists of the following
nodes: a probe-packet sender and receiver, one bottleneck link (between router
2 and router 3 having a link capacity of 10 Mbps and 10 ms latency) and 1 -
19 TCP SACK (with delayed acknowledgments) senders and receivers (in the
figure only three senders and receivers are shown) connected to the bottleneck
link via a router (router 1 and router 4). The total latency between a TCP sender
and receiver can vary between 42, 62 and 82 ms. X in the figure is 15, 25 or
35 ms depending on the desired latency. The TCP packet size is fixed to 550
bytes.
    The router queue size is limited only at the bottleneck link and is fixed to
100 packets (MTU = 1500 bytes). In [11] it is written that a common backbone
queue size should be able to store 250 ms of network traffic. This corresponds
to approximately 200 maximum size packets on a 10 Mbps link. However, 1)
in [11] it is pointed out that queue sizes have shown a decreasing trend in the
past and that this trend probably will continue and 2) 250 ms is a rather high
delay for a non-backbone 10 Mbps bottleneck link (consider IP-telephony for
example). Due to these reasons we have used the above stated queue. Droptail
has been chosen to be the router drop policy.
    The injection of probe packets is done at the probe-packet sender node.
The probe-packet train length and the number of probe-packet trains per sec-
ond vary. The separation between probe packets inside a probe-packet train is
proportional to a bit rate that is uniformly distributed between 5 and 15 Mbps.
The probe-packet size is fixed to 1200 bytes.
    A synthetic probe-packet generator has been used in order to circumvent
uncertainty due to the use of a specific available bandwidth measurement method.
The implementation of BART used in [2] does inject the probe-packets in this
random fashion to produce an estimate of the available bandwidth once a sec-
ond over time. It should be noted that most end-to-end available bandwidth
measurement methods use trains or pairs of different length when injecting
probe packets. Pathchirp is one exception; it uses trains with exponentially
decreasing separation between the probe packets instead of equally sized sep-
arations. Pathchirp is discussed in Section 8.5.
                   8.2 Research questions and experimental setup              117




Figure 8.1: The simulation topology used to study the impact of probe packets
on TCP. X in the figure can be 15, 25 or 35 ms.



    To avoid synchronization in the simulations, random timers are introduced.
The timers function as follows: Each TCP flow starts at time 0 + δ, where
δ is uniformly distributed between 0 and 5 seconds. Further, the calculated
separation s between two probe-packet trains is uniformly distributed between
[0.95∗s, 1.05∗s]. The simulation time is fixed to 100 seconds. Each simulation
has run 10 times.
    How similar is the simulation topology described above and a real network?
The topology can be though of as a rather simple network, especially since all
TCP flows experience the same minimum latency. Further, in a real network
the traffic on the bottleneck link will also consist of UDP packets, MAC-layer
packets and so on. The TCP flow aggregation on the bottleneck is also an
important aspect of this simulation setup. In this paper the aggregation is varied
between 1 and 19 flows as described above. Intuitively, the impacts from the
probe packets are higher on links with low aggregation. The impact of the
probe packets decreases with increasing aggregation towards an asymptotic
value.
118      Paper D


     It is most likely that the results in the simulations point out the direction of
the results that can be obtained in a real network. This assumption also holds
if the network topology changes, as long as there only is one bottleneck link
between the probe-packet sender and receiver. Then the probe packets only
cause packet loss and congestion at one point between senders and receivers.
Such measurements, in real networks, are left to future work since it requires a
large-scale testbed.


8.3 Results
8.3.1 Initial results
In this section some initial simulation results are shown to illustrate the impact
of probe packets on TCP in order to state that this is an important problem to
study.
    It is rather obvious that when injecting probe packets, the cross-traffic flows
on the bottleneck will have less bandwidth to consume. However, this is not
necessarily true. If, for example, a probe-packet sender is injecting one probe-
packet train consisting of 17 probe packets once a second (used by e.g. BART
and TOPP) the available bandwidth transiently increases. That is, the band-
width to consume has increased.
    The results in Figure 8.2 originate from a simulation where 5 TCP flows
are routed through the bottleneck link. The minimum latency between a TCP
sender and a receiver is fixed to 62 ms. The graph shows how the available
bandwidth (i.e. the true available bandwidth on the bottleneck link) changes
over time in two scenarios: In the first scenario the TCP packets do not have
to interact with probe packets while in the second scenario, the TCP packets
and probe packets do interact on the bottleneck link. When the TCP backoff
mechanism is triggered for a single or multiple flows, an increase in available
bandwidth is visible. The important observation from this plot is the following;
under the impact of probe packets, the available bandwidth on the link has
increased compared to the first scenario, even though the probe-packet sender
adds traffic to the bottleneck link.
    In Figure 8.3 the number of transferred bits by the aggregated TCP flows is
shown on the y-axis over time. This graph is created using the same simulation
results as the one in Figure 8.2. It is clear that the number of transferred bits
decreases when probe packets and TCP packets interact on the bottleneck link,
compared to when no probe packets are injected. The number of transferred
                                                                                                        8.3 Results                  119


                                                    Available bandwidth on bottleneck link, latency = 62ms
                                 10
                                                                                                                  17pp*1/s
                                                                                                                  No probing


                                  8




                                  6
    Available bandwidth (Mbps)




                                  4




                                  2




                                  0




                                 −2
                                      0   10   20        30        40       50         60        70          80    90          100
                                                                          Seconds



Figure 8.2: The available bandwidth changes when the TCP flows are affected
by the probe packets. The number of aggregated TCP flows is 5.



bits by TCP is decreased from approximately 950 Mbit to 900 Mbit during the
100 second simulation. It should be noted that the amount of injected probe
traffic during the simulation is 17 ∗ 1200 ∗ 8 ∗ 100 = 16.32 Mbit. This is less
than the decrease of 50 Mbit experienced by the aggregated TCP flows shown
in the figure. The important conclusion is that probe-packet trains of length 17
are bursty and have negative impact on TCP performance.
    Instead of sending one probe-packet train of length 17 once a second, one
could send 16 pairs of probe packets to obtain the same number of samples.
Then the plots in Figure 8.2 and 8.3 will differ. The question is how much,
why and what flight pattern of the probe packets is best from the TCP perspec-
tive. In the next section a deeper study of this phenomenon is presented. The
probe-packet train length varies along with the latency of the path used by the
TCP flows. Such simulations will show the impact of probe packets on the
performance of TCP.
120                        Paper D


                        1000
                                   17pp*1/s
                                   No probing
                        900


                        800


                        700


                        600
      Transfered Mbps




                        500


                        400


                        300


                        200


                        100


                          0
                               0    10          20   30   40     50      60   70   80   90   100
                                                               Seconds



Figure 8.3: The number of transferred bits decreases when injecting probe
packets in the path. The number of aggregated TCP flows is 5.


8.3.2 Measuring the impact of active end-to-end probing on
      TCP
In this section a number of simulation results are presented to illustrate the
impact of probe packets on TCP performance. The interpretation of the TCP
performance in this paper is the number of transferred bits during a simulation
(i.e. 100 seconds).
     Each plot in Figure 8.4 to 8.6 corresponds to results obtained from using
a fixed number of samples. An example: the total number of transferred TCP
bits, when injecting one probe-packet train of length 17 once a second is com-
pared to results obtained when injecting two probe-packet trains of length 9.
In both cases the probing application obtains 16 samples per second. It has
been shown that 16 samples per second give accurate estimates, in reasonable
time, of the available bandwidth when using available bandwidth tools such
as BART and TOPP. One sample is defined as the separation in time between
probe packet n and n − 1 in a probe-packet pair or train.
                                                                                                     8.3 Results                   121


                                             Transfered Mbits by TCP in different scenarios, latency = 42ms.
                               980


                               970


                               960


                               950
    Transfered Mbits in 100s




                               940


                               930


                               920


                               910


                               900
                                                                                                    No probing
                                                                                                    Probing 2pp*16/s
                               890
                                                                                                    Calc reduction 2pp*16/s
                                                                                                    Probing 9pp*2/s
                               880                                                                  Calc reduction 9pp*2/s
                                                                                                    Probing 17pp*1/s
                                                                                                    Calc reduction 17pp*1/s
                               870
                                     0   2   4         6         8       10       12          14        16        18          20
                                                                 Number of TCP flows



    Figure 8.4: The number of transferred TCP bits in different scenarios.


    If the number of samples per second is fixed, then shorter probe-packet
trains will inject a larger number of probe packets compared to a longer probe-
packet train. On the other hand, shorter probe-packet trains are less bursty.
    The plots in the Figures 8.4 to 8.6 show the number of aggregated TCP
flows on the bottleneck link on the x-axis. The y-axis describes the TCP per-
formance (i.e. the total number of transferred TCP bits by all aggregated TCP
flows) over 100 seconds. In each graph a curve corresponding to the number of
transferred bits without impact from probe packets is also plotted. Confidence
intervals have been left out in order to improve readability.
    The results shown in Figure 8.4 correspond to a simulation where the min-
imum latency between TCP senders and receivers is set to 42 ms. The solid
line is the TCP performance without impact from probe packets, and hence
can be seen as the optimal TCP performance line for the network path used
in the simulations. Results from several scenarios are shown along with the
optimum line, even though the lines can be hard to distinguish from each other.
The scenarios are the following: Injection of 16 probe-packet pairs, 2 probe-
122                                    Paper D


                                                 Transfered Mbits by TCP in different scenarios, latency = 42ms.
                                 980




                                 970




                                 960
      Transfered Mbits in 100s




                                 950




                                 940


                                                                                                        No probing
                                                                                                        Probing 2pp*16/s
                                 930                                                                    Calc reduction 2pp*16/s
                                                                                                        Probing 9pp*2/s
                                                                                                        Calc reduction 9pp*2/s
                                                                                                        Probing 17pp*1/s
                                                                                                        Calc reduction 17pp*1/s
                                 920
                                       4     6        8           10           12          14          16          18             20
                                                                       Number of TCP flows



      Figure 8.5: The number of transferred TCP bits in different scenarios.


packet trains of length 9 and 1 probe-packet train of length 17, once a second
respectively. Each configuration of the probe-packet flight pattern result in
16 samples per second. With each scenario a reference line corresponding to
the calculated reduction in TCP performance with respect to the probe-packet
flight pattern is also shown. In this paper a very simple calculation of the re-
duction in performance has been made. That is, the number of transferred TCP
bits without probing minus the number of injected probe-packet bits. This line
should be viewed as a reference line and not a correct value of the performance
reduction.
    To increase readability the case with 1 and 3 aggregated TCP flows are
omitted in Figures 8.5 and 8.6.
    The plot in Figure 8.5 is the reduced version of the plot shown in Figure
8.4. From the reduced plot there are several important observations that could
be made. Compare the line corresponding to the calculated performance re-
duction when injecting a probe-packet train of length 17 into the path with the
actual performance reduction. The actual reduction in performance is much
                                                                                                     8.3 Results                   123


                                             Transfered Mbits by TCP in different scenarios, latency = 82ms.
                               960




                               950




                               940
    Transfered Mbits in 100s




                               930




                               920




                               910
                                                                                                    No probing
                                                                                                    Probing 2pp*16/s
                                                                                                    Calc reduction 2pp*16/s
                               900                                                                  Probing 9pp*2/s
                                                                                                    Calc reduction 9pp*2/s
                                                                                                    Probing 17pp*1/s
                                                                                                    Calc reduction 17pp*1/s
                               890
                                     4   6        8           10           12          14          16          18             20
                                                                   Number of TCP flows



    Figure 8.6: The number of transferred TCP bits in different scenarios.


greater, in a large part of the plot, compared to the calculated reference line. In
the two other cases, that is when injecting 16 probe-packet pairs and 2 probe-
packet trains of length 9 the difference between calculated reduction and actual
reduction in TCP performance is smaller. This is better in some sense, because
then the probe-packet sender knows how large the impact from the probe pack-
ets is on TCP performance. Thus, it can take that into consideration when
estimating the available bandwidth. One more thing should be noted about
the plot, and that is that the reduction in TCP performance is less when prob-
ing with probe-packet trains of length 9 compared to the case when injecting
probe-packet pairs or probe-packet trains of length 17. Now, to minimize the
impact of the probe packets on TCP one might choose to use medium-length
probe-packet trains.
    Figure 8.6 corresponds to simulation results where the latency between
TCP senders and receivers has increased to 82 ms. The same pattern is vis-
ible here, but perhaps even more clear. The difference between the calculated
reduction in TCP performance when probing with one probe-packet train of
124               Paper D


                                                 TCP packet loss, latency = 42ms.
               2000
                          No probing
                          Probing 2pp*16/s
               1800       Probing 9pp*2/s
                          Probing 17pp*1/s

               1600


               1400


               1200
      Loss #




               1000


               800


               600


               400


               200


                 0
                      0    2         4       6        8       10       12           14   16   18   20
                                                      Number of TCP flows



                               Figure 8.7: The number of lost TCP packets.


length 17 and the actual reduction is significant. The differences in the two
other scenarios are, as in the previous case, much smaller. If the probe-packet
sender intends to minimize the reduction in performance experienced by the
TCP flows, it seems to be a good choice to inject probe-packet trains of length
9 two times per second.
    Simulations have also been performed using other probe-packet train lengths
(but still keeping the number of samples fixed at 16). However, the reduction in
TCP performance does not decrease much further and thus those simulations
are left out of this paper.
    It has been discussed in the past whether probe-packet pairs or probe-packet
trains is the most network friendly flight pattern. As we have shown above, it
is not trivial to decide. The reason for this is that the injection of probe packets
affects both the TCP loss rate and round-trip times. This is further discussed
below.
    In Figure 8.7 the TCP packet loss is shown on the y-axis. The x-axis is
the number of aggregated TCP flows over the bottleneck link. The loss is
                                                                                   8.3 Results                125


                                       TCP mean round−trip time, latency = 42ms.
                    100




                     95




                     90
    Mean RTT (ms)




                     85




                     80

                                                                                      No probing
                                                                                      Probing 2pp*16/s
                                                                                      Probing 9pp*2/s
                                                                                      Probing 17pp*1/s
                     75
                          0   2   4     6        8       10       12         14      16      18          20
                                                 Number of TCP flows



                                      Figure 8.8: The mean RTT.


minimized when no probing is performed (as expected). Further, it is seen
that the packet loss when injecting probe-packet pairs, 16 times per second, is
nearly equally low as in the case with no probing. Increasing the burstiness
of the probe packets (i.e. increasing the probe-packet train length) increases
the number of lost TCP packets. Based on this one would assume that using
the flight pattern of 9 probe-packets twice a second should decrease the TCP
performance more compared to the case when injecting probe-packet pairs.
However, as seen above this is not the case. Why is that?
    In Figure 8.8 the mean TCP round-trip time is plotted for the different flight
patterns. From the plot it is clear that injecting probe-packet pairs increases the
TCP round-trip time, which in turn decreases the TCP performance. Injecting
probe-packet trains of length 9 on the other hand decreases the round-trip time.
This is due to a higher loss rate, which decreases the queuing time in the routers
for the TCP packets. Hence the TCP performance is not reduced as much as in
the probe-packet pair case. That is, both the TCP loss rate and mean RTT are
affected, in different ways, by the probe packets sent in different flight patterns.
126     Paper D


   Important conclusions from this section are the following:
   • It seems that medium length probe-packet trains (9 packets) are prefer-
     able in order to minimize the reduction in TCP performance.
   • TCP performance decreases due to packet loss and increased round-trip
     time caused by the probe packets.
   • The available bandwidth can actually increase when sending probe pack-
     ets with a bursty flight pattern through a network path.


8.4 Network friendly probing
Many applications, such as games and IP-telephony, use UDP instead of TCP.
Some of these applications have to transfer large amounts of data over the net-
work. This can cause congestion in the network, since UDP does not have
built-in congestion control mechanisms. Congestion caused by UDP is harm-
ful to other UDP flows as well as to TCP flows sharing the bottleneck. The
Data Congestion Control Protocol, DCCP [12], is one attempt to standardize
a congestion control protocol for flows that does not need (or performs worse
using) TCP.
    However, in the case with available bandwidth measurement methods the
probe-packet rate (UDP packets) can not be controlled in the same way, since
these methods rely on causing congestion. Still, there is a need to make avail-
able bandwidth measurement methods as network friendly as possible.
    The findings in the previous section provide valuable insights to a quanti-
tative definition of network friendly probing for available bandwidth measure-
ment methods. In this paper, the following definition is proposed:

    Definition: If 1) the reduction in TCP performance is minimized and 2)
during a time interval, the injection of x Mbit of probe packets does not de-
crease the total amount of transferred TCP bits more than x Mbit, the injection
of probe packets is considered network friendly.

    Using this definition, different probe-packet flight patterns can be com-
pared to each other when considering network friendliness. Of course, to opti-
mize available bandwidth measurement methods, estimation accuracy and time
aspects must also be considered.
    Take the plot in Figure 8.6 as an example. Comparing the case when in-
jecting 16 probe-packet pairs to the case when injecting 2 probe-packet train
                              8.5 Tools with special flight patterns        127


of length 9 per second, at TCP aggregation 19 on the x-axis; which is the most
network friendly alternative to use with respect to the above definition? The
relation between calculated and actual reduction in TCP performance is ap-
proximately the same. However, it is clear that injecting probe-packet trains
of length 9 results in a smaller reduction in TCP performance compared to
the probe-packet pair case. Thus, using 2 probe-packet trains of length 9 per
second is the best alternative.
    From the view of available bandwidth measurement methods it is impor-
tant to keep a one-to-one relation between the amount of injected probe packets
and the reduction in TCP performance, not only due to the definition of net-
work friendly probing above. Consider the following; if a probe-packet sender
injects x Mbps of probe traffic while the reduction is higher than x, then the
packets injected by the probe-packet sender may actually have increased the
available bandwidth. This is a problem that must be addressed (e.g. by be-
ing network friendly). Another question that is important to ask is: what was
the available bandwidth before the probe-packets actually were injected? Is it
possible to calculate it if the one-to-one relation is not fulfilled?
    A study of how network friendly state-of-the-art available bandwidth mea-
surement methods are, compared to each other is left to future research.


8.5 Tools with special flight patterns
One end-to-end available bandwidth tool that does not inject probe packets
using probe-packet pairs or trains is Pathchirp. Instead, Pathchirp injects the
probe packets in so called chirps where the separation between two successive
probe packets is exponentially decreasing. Using this method several probe-
packet rates can be scanned in one chirp. Since the scope of this paper is
to investigate the impact of active probing on TCP such a flight pattern is of
interest.
    In Figure 8.9 it is shown how much the probe packets injected by Pathchirp
affect the TCP performance. Lines corresponding to the impact of probe-
packet trains of length 9 twice a second are shown for comparison. The number
of injected UDP based probe packets by Pathchirp during 100 seconds in dif-
ferent scenarios is shown in Table 8.1. The probe-packet size is 1200 bytes.
The results are means of 10 simulation runs.
    From the figure it is clear that Pathchirp also affects the TCP performance
in a negative way. At a higher TCP aggregation level the Pathchirp way of
injecting probe packets is more network friendly than using probe-packet trains
128                                    Paper D


                                             No. TCPs               1            3           5           7           9
                                           No. packets           2114          921         767         740          806
                                             No. TCPs              11           13          15          17          19
                                           No. packets            722          631         616         604          562

Table 8.1: Number of injected probe packets by Pathchirp in different scenarios
during 100 second simulations.



                                                   Transfered bits by TCP in different scenarios, latency = 82ms.
                                 970




                                 960




                                 950
      Transfered Mbits in 100s




                                 940




                                 930




                                 920



                                                                                                   No probing
                                 910                                                               Pathchirp probing
                                                                                                   Calc reduction using pathChirp
                                                                                                   Probing 9pp*2/s
                                                                                                   Calc reduction 9pp*2/s
                                 900
                                       4       6       8           10           12          14           16          18             20
                                                                        Number of TCP flows



      Figure 8.9: The number of transferred TCP bits in different scenarios.



of length 9. These results give a hint of how network friendly Pathchirp is.
    However, it is not possible to actually compare the two cases straight off
since the Pathchirp curve originates from a real tool that varies the number
of injected probe packets depending on the current path conditions while the
probe-packet train injection is done by a synthetic probe-packet generator.
                                                                 8.6 Discussion           129


8.6 Discussion
As shown in this paper the impact of the probe packets on TCP performance is
larger than the actual number of bits injected. But is it really a big problem?
    In Figure 8.62 the number of transferred bits by 19 TCP flows in one sim-
ulation run is decreased from 960 Mbits to 918 Mbits when injecting probe-
packet trains of length 17. That corresponds to a decrease of the TCP per-
formance close to 4.5%. The calculated reduction for this flight pattern is
1.7%. Further, the procentual difference between the impacts caused by in-
jecting probe packets with different flight patterns is quite small.
    When injecting probe-packet trains of length 9, which is the network friendly
flight pattern as described above, into the network path the reduction in TCP
performance is only 2.2% (in Figure 8.6 at TCP aggregation level 19). The
calculated reduction is 1.9%. The conclusion is that the impact of the probe
packets on TCP performance is quite small as well as the difference between
using different flight patterns. Even tough the difference is small it is desirable
to use the most network friendly flight pattern in order to reduce the impact on
TCP as well as other network flows.
    It is still a problem that the TCP performance is decreased by the probe
packets. This problem gets more apparent when available bandwidth measure-
ment methods are used in larger scales. That is, when several measurement
sessions may take place concurrently over the same network path. Then the
probe packets will push back the TCP flows.
    How to solve this problem is left to future research. However, viable ap-
proaches is to be network friendly. It is also important to investigate whether
application traffic can be used as probe packets in order to minimize the over-
head on the network.


8.7 Conclusions
Since TCP is a dynamic protocol that varies the send rate depending on network
latency and loss rate, the injection of probe packets will affect its performance.
    The impact of probe packets, injected by available bandwidth measurement
methods, on TCP has been investigated in this paper. The investigation has
been performed in an NS-2 environment.
   2 The same reasoning can be made for other TCP aggregation levels as well as for the results

shown in Figure 8.5.
130     Paper D


    The results indicate that medium length probe-packet trains (or perhaps
chirps) are the flight pattern to be used in order to minimize the reduction in
TCP performance.
    The reduction in TCP performance is due to the fact that the probe packets
cause TCP packet loss, by long probe-packet trains, and increased round-trip
time, mainly caused by many probe-packet pairs.
    The term “network friendly probing” has been defined and discussed in the
paper.
    Future work is to compare how network friendly end-to-end available band-
width measurement methods are compared to each other. It will also be inves-
tigated what estimates will be produced by available bandwidth measurement
methods when the cross traffic mainly consists of TCP flows. The relation
between network friendliness and measurement accuracy is also an open issue.
Bibliography

[1] Demetres Antoniades, Manos Athanatos, Antonis Padadogiannakis,
    Evangelos P. Markatos, and Constantine Dovrolis. Available bandwidth
    measurement as simple as running wget. In Proceedings of the Passive
    and active measurement workshop, Adelaide, Australia, 2006.

                                                             a
[2] S. Ekelin, M. Nilsson, E. Hartikainen, A. Johnsson, J. M˚ ngs, B. Me-
                     o
    lander, and M. Bj¨ rkman. Real-time measurement of end-to-end avail-
    able bandwidth using kalman filtering. In Proceedings of the IEEE/IFIP
    Network Operations and Management Symposium, Vancouver, Canada,
    2006.

[3] V. Ribeiro, R. Riedi, R. Baraniuk, J. Navratil, and L. Cottrel. pathChirp:
    Efficient available bandwidth estimation for network paths. In Proceed-
    ings of the Passive and Active Measurement Workshop, San Diego, USA,
    2003.

[4] Manish Jain and Constantinos Dovrolis. End-to-end available bandwidth:
    Measurement methodology, dynamics, and relation with TCP throughput.
    In Proceedings of the ACM SIGCOMM conference, Pittsburg, USA, Au-
    gust 2002.

[5] Strauss, Katabi, and Kaashoek. A measurement study of available band-
    width estimation tools. In Proceedings of the ACM SIGCOMM Internet
    Measurement Workshop, Miami, USA, 2003.

                          o
[6] Bob Melander, Mats Bj¨ rkman, and Per Gunningberg. Regression-based
    available bandwidth measurements. In Proceedings of the SCS Sympo-
    sium on Performance Evaluation of Computer and Telecommunications
    Systems, San Diego, USA, July 2002.

                                    131
                                o
 [7] Andreas Johnsson, Mats Bj¨ rkman, and Bob Melander. An analysis of
     active end-to-end bandwidth measurements in wireless networks. In Pro-
     ceedings of the IEEE/IFIP workshop on End-to-end Monitoring Tech-
     niques and Services, Vancouver, Canada, 2006.
              a
 [8] Attila P´ sztor and Darryl Veitch. The packet size dependence of packet
     pair like methods. In Tenth International Workshop on Quality of Service,
     Miami Beach, USA, May 2002.
                                                    o
 [9] Andreas Johnsson, Bob Melander, and Mats Bj¨ rkman. On the analy-
     sis of packet-train probing schemes. In Proceedings of the International
     Conference on Communication in Computing, Las Vegas, 2004.
[10] R.S. Prasad, M. Murray, C. Dovrolis, and K. Claffy. Bandwidth estima-
     tion: metrics, measurement techniques, and tools. IEEE Network Maga-
     zine, 2003.
[11] Damon Wischik and Nick McKeown. Part 1: Buffer sizes for core routers.
     In ACM/SIGCOMM CCR, 2005.
[12] Eddie Kohler, Mark Handley, and Sally Floyd.         Datagram con-
     gestion control protocol (dccp).        IETF draft, December 2005.
     http://www.ietf.org/html.charters/dccp-charter.html.
Chapter 9

Paper E:
On measuring the available
bandwidth in wireless
802.11b networks

                            o
Andreas Johnsson and Mats Bj¨ rkman
Not yet published.




                                 133
                                   Abstract

BART is a state-of-the-art active end-to-end bandwidth measurement method
that estimates not only the available bandwidth but also the link capacity of the
bottleneck link. It uses a Kalman filter to produce estimates in real time during
a measurement session.
    In this paper, BART has been used to continue our study of the impact
of 802.11b networks on obtained estimates of available bandwidth and link
capacity. The Kalman filter used by BART is tuneable and in this paper it is
tuned to improve the estimation accuracy when the bottleneck is a 802.11b
link. Further, Linux driver problems associated with measurements in 802.11b
networks are discussed. For example, in some cases the send rate of a probe-
packet train is lower than anticipated when injected by a probe-packet sender.
This can bias the estimates for BART as well as for other active measurement
methods.
    In the paper it is also argued for, that when using BART, the estimated link
capacity is equal to the available bandwidth when the bottleneck is a 802.11b
link. This property has been used to tune the Kalman filter.
                                                    9.1 Introduction        135


9.1 Introduction
Since wireless networks are becoming more popular, such as when accessing
the Internet from a personal laptop or when surfing webpages using a cellular
phone, it is important to continue the study on methods for actively measuring
end-to-end available bandwidth function in these networks. If these methods
are to be widely accepted as performance measurement tools, it is important to
ensure that they report correct values in wireless networks as well.
     Available bandwidth methods that exist today are for example ABget [1],
Pathchirp [2], Pathload [3], Spruce [4] and TOPP [5]. The principle is to in-
ject so called probe packets that traverse the network path. Each probe packet
arrival is then time stamped at the receiver node. By deploying different anal-
ysis methods to these time stamps the link capacity (a constant property of the
bottleneck link) and/or the available bandwidth (the unused portion of the link
capacity) can be deducted. Many available bandwidth measurement methods
need no prior information about the underlying network topology. Thus, these
methods can easily be applied in situations where the end-to-end performance
is to be measured. The methods mentioned above differ in how the probe pack-
ets are sent and in the analysis and estimation algorithms used. An overview
of available bandwidth measurement methods, tools and theory can be found
in [6].
     In this paper, a newly developed available bandwidth measurement method
called BART [7] is used to continue the study on how the properties of wireless
802.11b bottleneck links affects the produced bandwidth estimates. BART is a
successor of the TOPP method that also estimates the available bandwidth and
link capacity. Assuming a fluid network model and FIFO queues, TOPP uses
linear regression while BART deploys a Kalman filter to estimate the available
bandwidth and link capacity. Using BART, estimates are obtained in real time.
     This paper also discusses the relation between the estimate of the link ca-
pacity and the available bandwidth under the assumption that the bottleneck is
a wireless 802.11b link. It is argued for that the estimates are the same when
using BART, due to the underlying measurement model. It is also investigated
whether the measurement model that BART relies on really is applicable in
wireless 802.11b networks. Further, some problems regarding measurements
in wireless networks are pointed out. These problems are independent of the
measurement method used.
     One major part of this paper concerns tuning of the Kalman filter used by
BART in order to obtain more accurate estimates when the cross traffic changes
over time on an 802.11b bottleneck.
136     Paper E


    The rest of this paper is organized as follows; in Section 9.2 the measure-
ment model is described along with how BART actually estimates the available
bandwidth and link capacity. The relation between estimated link capacity and
available bandwidth using the measurement model is also discussed. Section
9.3 describes the research questions and the testbed. In Section 9.4 a deeper
investigation indicates that the measurement model is applicable in 802.11b
networks. Further, some problems concerning drivers are also pointed out. In
Section 9.5 the BART Kalman filter is tuned in order to optimize the accuracy
of the estimations by varying the so called Q matrix. Conclusions are located
in Section 9.6.


9.2 Measurement model
In this paper, BART [7] is used to estimate the available bandwidth and link
capacity of an end-to-end path when the bottleneck is a 802.11b link. The
BART tool is divided into two parts, one probe packet sender and one probe
packet receiver. The BART sender injects UDP probe packets into the network
at different input rates u, or with the mean packet packet separation ∆in . The
receiver, on the other end of the network, time stamps each incoming packet
and calculates the received rate r, or the mean packet separation ∆out . Then
the inter-packet strain ǫ = ∆out −∆in is calculated along with the variance R.
                                 ∆in
Assuming a fluid network model and FIFO queues, the strain is zero during
network underload and increases linearly during overload (often related to as
the TOPP model [5]). That is,

                                 0       underload
                        ǫ=                                                (9.1)
                               αu + β     overload

    The available bandwidth B is then defined as the intersection of the sloping
                                   β
line and the x-axis. That is B = − α . The available bandwidth is the maximum
rate u that can be used without saturating the network. Further, according to
                                                                          1
the TOPP model [5] the link capacity C is the inverse of α. That is C = α .


9.2.1 The BART Kalman filter
BART [7] deploys a Kalman filter [8] in order to measure and track changes in
α and β and thus the available bandwidth and the link capacity. BART injects
probe packets into the network path and measures the strain ǫ at the receiver.
                                              9.2 Measurement model          137


For each measurement sample, where the probe rate u is larger than the current
available bandwidth estimate, a new estimate is obtained, as described in [7].
    The Kalman filter can be described as an iterative procedure where a system
state xk = g(xk−1 , uk−1 , wk−1 ), that is not directly observable, is predicted
for a time step k. The function g describes the evolution of the system state
over time and must be linear in order to apply a Kalman filter, u is some input
to the measured system and w is the process noise. For the prediction at time
k a correction is made using a measurement sample zk = h(xk , uk , vk ) where
vk is the measurement noise. The weight of the new sample in the correction is
given by the Kalman gain K, that depends on the accuracy of the measurement
sample as well as on Q, the process noise covariance matrix. The estimate
                     ˆ
of the system state, xk , obtained by predictions and corrections, will track
changes in xk .
    The system state xk in the BART Kalman filter is chosen as

                                          α
                                 xk =                                       (9.2)
                                          β

from which the available bandwidth and the link capacity is obtained.
    Below, a short review of how the Kalman equations are applied in BART is
given. The estimated prediction of the system state x− and the predicted error
                                                    ˆk
covariance matrix Pk is updated according to
                     −


                                 ˆk    x
                                 x− = Aˆk−1                                 (9.3)

                            Pk = APk−1 AT + Q
                             −
                                                                            (9.4)
where A is a state transition matrix. A is chosen to predict the system state at
time k given the estimated system state at time k − 1. In BART it is assumed
that A = I which means no evolution of the system state.
    When the predictions for time k are calculated a measurement sample zk is
                                                                             ˆk
used as input in order to correct the predicted estimate of the system state x− .
This is done in three steps according to the following Kalman equations
                             − T     − T
                       Kk = Pk Hk (HPk Hk + R)−1                            (9.5)

                         ˆ    ˆk               ˆk
                         xk = x− + Kk (zk − Hk x− )                         (9.6)
                                              −
                            Pk = (I − Kk Hk )Pk                             (9.7)
                                                                    ˆ
where K is the Kalman gain. The new estimate of the system state xk is calcu-
lated using the Kalman gain multiplied with the so called resudial (zk − H x− ).
                                                                           ˆk
138     Paper E


This is the difference between the new sample and the prediction of that sam-
                                                                     ˆ
ple. Depending on the resudial and the Kalman gain the adaptation of x towards
x will vary.
     It is important to note that K increases with Q and decreases with the
measurement sample variance R. Q is unknown, but it is possible to choose
it to reflect the predicted stability of the system. By varying Q the Kalman
filter is tuned to weight the impact of the current sample on the prediction of
the system state x− differently. The last step is that a new estimated error
                   ˆk
covariance matrix Pk is calculated.
     In the Kalman filter, the measurement sample zk relates to the system state
as
                                zk = Hk xk + vk                           (9.8)

where vk is the measurement noise. In BART, the measurement sample zk
equals the strain ǫ, which is linear during network overload. To create a linear
relation between the strain and the system state, H is chosen as

                              Hk =     uk   1   .                          (9.9)

    Now Hk xk = αk uk + βk = ǫk = zk . That is, a relation between the
                                                                ˆ
measurement sample zk and the system state xk is created. Thus, x can be
calculated from the measurement samples z.


9.2.2 Tuning of the Kalman filter
It is possible to trade estimation stability for agility in the Kalman filter by
tuning the Q matrix. In the BART Kalman filter the predicted process error
covariance matrix Q has the following structure [9]

                q11   q12          V (∆α)           C(∆α, ∆β)
        Q=                   =                                           (9.10)
                q21   q22         C(∆α, ∆β)          V (∆β)

where V () is the variance and C() the covariance. A high value of the com-
ponents in Q means that the measurement sample zk is weighted more heavily
compared to the case with a low value. That is, the estimations may fluctuate
more using a high Q. But on the other hand, the adaptation towards a new
system state is faster using a high Q compared to a low Q. Currently in BART,
the covariance C() in Q is set to zero because no relation between α and β is
assumed.
                                             9.2 Measurement model          139


    Later in this paper the Q matrix is tuned in order to optimize the accuracy
of the estimation of the available bandwidth and link capacity in 802.11b net-
works using BART, both when the system state is stable but also when it varies
over time.

9.2.3 The relation between estimates of the available band-
      width and link capacity using BART in 802.11b net-
      works
Using the measurement model described above we state that the estimated link
capacity should be equal to the estimated available bandwidth, when the bottle-
neck is a 802.11b link. The original TOPP model [5] states that in FIFO queue
systems, the relation between the input rate u and the received rate r is

                        u           1       underload
                          =     X       u                                (9.11)
                        r       C   +   C    overload
     where X is the cross-traffic rate transmitted through the FIFO queue (com-
pare Equation 9.1). In [7] the conversion between u/r and ǫ is described and
it is shown that ǫ + 1 = u/r. Compare equations 9.1 and 9.11. We continue
the discussion on why the available bandwidth and the link capacity estimates
should be equal when the bottleneck is a 802.11b link using Equation 9.11 (i.e.
the TOPP model), since that equation was used in previous work to describe
the impact of link-layer cross traffic and the choice of the probe-packet size on
the relation between u and r [10].
     The quotient u/r during overload in 802.11 wireless networks was found
to be


         u   nTk (X802.11 ) + Ts                             1
           =                     X + (nTk (X802.11 ) + Ts )              (9.12)
         r           s                                      tu

    where Tk () is a function of the cross traffic on the wireless link X802.11 .
Tk () increases due to link-layer contention and retransmission caused by the
cross traffic on the wireless link. Ts is the physical transmission time of s
bits over the 802.11b link, n relates to the probe packet size s and tu is the
inter-packet separation in time (i.e. related to u). In [10] there was no dis-
tinction made between the FIFO based cross traffic X, that flows through the
same FIFO queue as the probe packets, and the cross traffic that flows from
other nodes on the wireless network X802.11 (i.e. that affects the link layer).
140      Paper E


This distinction is however crucial in order to understand that the link capacity
and available bandwidth estimates are equal in these types of networks when
deploying the TOPP model which BART relies on.
    Since the cross traffic sharing the FIFO queue used by the probe-packet
sender is zero (the probe-packet sender does not inject cross traffic during a
measurement session), X = 0. Thus, u/r = (nTk (X802.11 ) + Ts ) t1 during
                                                                        u
overload. This line passes through origo. For any such line the slope of the line
equals the intersection of the line u/r = 1. Further, the available bandwidth
is defined as the intersection of u/r during overload and the line u/r = 1.
That is, when using the TOPP model to obtain estimates in networks where the
bottleneck is a 802.11b access link, the available bandwidth and link capacity
will be the same. Since u/r = 1 + ǫ [7] the same results hold for BART.
    The important conclusion from this discussion is that the estimates of avail-
able bandwidth and link capacity are the same using this measurement model
in 802.11 wireless networks, and that both will vary with the cross-traffic rate
on the 802.11b link layer.


9.3 Research question and experimental setup
This section describes in more detail what is studied in this paper and why.
Further, the experimental setup used in order to perform the study of how the
available bandwidth and link capacity estimates are affected by 802.11b bot-
tleneck links is described.


9.3.1 Research questions
The research questions studied in this paper concern bandwidth measurements
in wireless 802.11b networks. The TOPP model described above, that BART
makes use of, relies on the fact that the forwarding mechanism in the routers is
FIFO based. Thereby, the strain induced between probe packets is proportional
to the amount of cross traffic sharing the same path. Using this fact the rate-
response curve can be obtained by injecting probe packets at increasing rates.
This will be a segmented curve described by Equation 9.1; first it is zero, then
when the network is saturated it will show a linear increase. The available
bandwidth and the link capacity of the bottleneck link can then be deducted
either by using linear regression, used by TOPP, or a Kalman filter, applied by
BART.
    One obvious research question is to investigate whether the rate-response
                    9.3 Research question and experimental setup             141




                            Figure 9.1: The testbed.


curve obtained from 802.11b networks can be used to estimate the available
bandwidth. The queues in each wireless node is FIFO based, but the layer 2
mechanism in 802.11b networks is approximately fair queuing based. That is,
is the TOPP model used by BART really applicable?
     The major part of this paper however address the following research ques-
tion; how to set the Q matrix in the Kalman filter used by BART when the
bottleneck is a wireless link. As previously described, a large Q instructs the
Kalman filter to anticipate large variance in the system state while a low Q
does not.

9.3.2 Experimental setup
The testbed, see Figure 9.1, consists of 6 ordinary PC machines running Linux
with a stable 2.4 kernel. Three of them are configured as routers. These ma-
chines are connected using wired single-access links that operate at 10 or 100
Mbps. In the wireless part of the testbed there are 4 laptops, 3 installed with
Linux. The cross-traffic senders in the wireless part of the network are installed
with Linux kernel 2.6 while the probe-packet sender can switch between 2.4
and 2.6 kernels. The administrator laptop is running Windows XP. The laptops
connect to the wired part of the network using a standard 802.11b/g access
point. The wireless cards in the laptops are Prism2 compatible and deploy
the standard 802.11b drivers in Linux. The bottleneck in this network is the
142      Paper E


802.11b access link.
     Cross traffic can be generated from two of the laptops as well as from the
cross-traffic machine in the wired part of the network. The cross-traffic receiver
is the same independent of source. The cross traffic used for the experiments
that this paper reports on has been generated using tg (http://www.postel.org/tg/),
from two cross-traffic nodes in the wireless part of the network. The cross traf-
fic rate X802.11b is uniformly distributed in an interval (0.9X802.11b , 1.1X802.11b )
and consists of 1400 byte UDP packets. This cross-traffic scenario may not be
very realistic, but it is sufficient for the study in this paper. No cross traffic is
injected by the probe-packet sender.


9.4 Measurements of the rate response curve in
    802.11b networks and its relation to available
    bandwidth
This section investigate the rate-response curve obtained by BART when mea-
suring in a network where the bottleneck is a 802.11b wireless link.


9.4.1 Send rate problems
During the work of measuring the available bandwidth in 802.11b networks,
we have encountered a problem concerning the sending mechanism of packets.
The problem has been identified when Linux kernels 2.4 and 2.6 has been used
at the probe-packet sender node. Also, tests have shown that the impact of the
choise of wireless cards can be omitted.
    The problem is discussed below; available bandwidth measurement meth-
ods needs to know the send rate u of the probe packets as well as the measured
rate r at the receiver side (compare Equation 9.1). When injecting probe pack-
ets into a wired network, the packets will leave the network interface at the
selected rate u, decided at the application layer (this is at least our experience
from experimenting with several testbeds and measurements over the Internet).
Thus, each probe packet does not have to be time stamped at the sender side in
order to be certain of the send rate u. Selecting a rate and then send the probe
packets with the corresponding time separation is enough. If the rate is higher
than the capacity of the network card, the probe packets are buffered, either
by the driver or by the network card itself, and then sent at the capacity of the
network card.
9.4 Measurements of the rate response curve in 802.11b networks and its
                                relation to available bandwidth    143



                 3



                2.5



                 2



                1.5



                 1
           z




                0.5



                 0



               −0.5



                −1
                      0   2      4           6             8   10      12
                                      Probe rate [bit/s]               6
                                                                    x 10




Figure 9.2: Rate-response curve when the selected rate is shown on the x axis.


     However, when sending probe packets (using the same available bandwidth
measurement application, BART) through the standard drivers for 802.11b net-
works in Linux, the probe packets do not necessarily leave the application at
the specified rate. Instead, the actual send rate can not increase further when
the selected send rate exceeds a send-rate threshold (that varies with e.g. the
probe-packet train length). It seems that the 802.11b driver suspends the ap-
plication instead of buffering the probe packets in the driver or on the network
card. That is, the probe packets does not leave the application at the selected
rate.
     This problem is illustrated using two rate-response curves shown in Figures
9.2 and 9.3. The cross-traffic rate on the wireless link is zero in the experiment
corresponding to these graphs. In Figure 9.2, the strain ǫ is shown as a function
of the selected rate, that is the send rate the application believe it can inject
probe packets at. As seen, the linear increase of ǫ stops at approximately 10
Mbps. As discussed above, the application is suspended and thus delays the
send mechanism when sending at rates above the 10 Mbps threshold. That
is, the application believe it send probe packets at a selected rate while the
actual rate can not exceed 10 Mbps. Then the horizontal line in Figure 9.2 is
obtained. No increase of the actual send rate means no increase of the strain
ǫ. The rate-response curve obtained using the actual rate, that is measured by
144      Paper E



                 3



                2.5



                 2



                1.5



                 1
           z




                0.5



                 0



               −0.5



                −1
                      0   2      4           6             8   10      12
                                      Probe rate [bit/s]               6
                                                                    x 10




 Figure 9.3: Rate-response curve when the actual rate is shown on the x axis.




time stamping each probe packet when sent from the application, is shown in
Figure 9.3. As seen, the actual rate reach a maximum at approximately 10
Mbps while the selected rate is in the interval (1, 12) Mbps.

    Assuming a linear system model during overload, as BART does, the es-
timates of α and β will be underestimated and overestimated, respectively, if
using the selected send rate as an input parameter to the Kalman filter.

    The cluster of samples seen at 10 Mbps in Figure 9.3 originates from the
fact that the measured system varies over time, and also because all samples
corresponding to a selected rate in the interval (10, 12) Mbps will be located at
the 10 Mbps threshold.

    It is out of scope of this paper to study the drivers in the Linux operating
system in order to find out exactly why this phenomenon appears. Rather, this
problem is just illustrated. It can be handled by time stamping each probe
packet at the sender in order to know the actual send rate of the probe packets,
and perhaps by not using the measurement samples at the highest rates. Then
the system model is linear during overload, which is required by BART.
9.4 Measurements of the rate response curve in 802.11b networks and its
                                relation to available bandwidth    145



                 3



                2.5



                 2



                1.5



                 1
           z




                0.5



                 0



               −0.5



                −1
                      0   2     4           6             8   10      12
                                     Probe rate [bit/s]               6
                                                                   x 10




Figure 9.4: Two rate-response curves in one graph. During the first part of the
measurement session the cross traffic is zero, then it increases to 6 Mbps.


9.4.2 Does BART estimate correctly?
This section tries to answer whether BART estimates the available bandwidth
and link capacity correctly when the bottleneck is a 802.11b wireless link. To
answer that question the true available bandwidth must be known (the true
value of the available bandwidth is what BART and other methods tries to esti-
mate). In a wired testbed setting the true available bandwidth can be measured
using tools such as tcpdump. Then the available bandwidth is the fixed link
capacity minus the cross-traffic load, obtained by tcpdump, during some time
interval. However, in wireless 802.11b networks the capacity of the link is very
hard to measure since it varies with the radio quality. Further, even if the ra-
dio quality was known there is no simple formula to calculate the cross-traffic
impact on the capacity since it depends on the packet size, link-layer retrans-
mission, backoff mechanisms etcetera. Instead, the UDP throughput has been
used as a rough value of the available bandwidth.
    The graph in Figure 9.4 originates from an experiment where the probe
packets are sent over time from a node running Linux version 2.4 and cross
traffic from a node running 2.6. The graph in Figure 9.5 originates from an
experiment where all packets are sent from nodes running Linux version 2.6.
146      Paper E



                 3



                2.5



                 2



                1.5



                 1
           z




                0.5



                 0



               −0.5



                −1
                      0   2       4          6             8   10      12
                                      Probe rate [bit/s]               6
                                                                    x 10




Figure 9.5: Two rate-response curves in one graph. During the first part of the
measurement session the cross traffic is zero, then it increases to 6 Mbps.


In both scenarios the cross traffic is zero during the first 250 time ticks. Then it
increases to 6 Mbps in order to saturate the 802.11b link during 250 additional
time ticks. Due to the change in cross-traffic rate during the experiments there
will be two distinct rate-response curves in each graph. Observe that the right-
most rate response curve in Figure 9.5 isn’t really linear. Further, there is a gap
between samples corresponding to a send rate between 5 and 6.25 Mbps. This
phenomenon only appears when the wireless link is empty (or almost empty)
and using a 2.6 Linux kernel node as the probe-packet sender. More research
is needed to explain why. BART requires a linear model, and in this work it is
assumed that the rate-response curve is approximately linear in this case.
    In Figure 9.6 the BART estimates of the available bandwidth (solid lines)
and link capacity (dashed lines) are shown for both scenarios. As can be seen,
the available bandwidth is approximately 6 Mbps when sending probe packets
from a Linux 2.4 node when no cross traffic is present. During overload the
available bandwidth goes down to 4 Mbps. When all traffic is sent from Linux
2.6 nodes the available bandwidth and link capacity goes from 4 Mbps to 3
Mbps.
    Since the estimates vary depending on the Linux kernel version used at the
probe-packet sender node, and thus the driver version for the 802.11b network
9.4 Measurements of the rate response curve in 802.11b networks and its
                                relation to available bandwidth    147


                                    6
                                x 10
                            8

                            7

                            6
           Estimate (bps)




                            5

                            4

                            3

                            2

                            1
                                0       50   100   150   200   250   300   350   400   450       500
                                                                                       Time (ticks)
                                    6
                                x 10
                            8

                            7

                            6
           Estimate (bps)




                            5

                            4

                            3

                            2

                            1
                                0       50   100   150   200   250   300   350   400   450       500
                                                                                        Time (ticks)




Figure 9.6: The BART estimates of the available bandwidth (solid) and link
capacity (dashed). The upper graph corresponds to the rate-response curves in
Figure 9.4 and the lower to Figure 9.5.


card, it is just to ask whether the obtained estimates are correct or not. As
described above, the UDP throughput is used as a rough value of the available
bandwidth. The size of the UDP packets is 1400 bytes (i.e. equal to the size of
the probe packets). Assuming the same scenarios as before, a UDP throughput
test from a Linux 2.4 node gives approximately 6 Mbps and then 3 Mbps.
When all nodes run Linux 2.6 the throughput is 4 Mbps during the first 250
time ticks and then 3 Mbps. Thus, comparing the throughput test results with
the results obtained by BART shown in Figure 9.6, the available bandwidth
and link capacity estimates are more correctly estimated when all nodes run
the same kernel version (i.e. 2.6). When using kernel version 2.4 at the probe-
packet sender, the available bandwidth is overestimated when cross traffic is
present on the bottleneck.
    It is also important to notice that the available bandwidth is only 4 Mbps
when the 802.11b link is empty when sending probe packets from a Linux 2.6
node, compared to 6 Mbps when measuring from a platform running a 2.4
Linux kernel. However, in both cases when no cross traffic is present on the
802.11b link, the BART estimates are in line with the UDP throughput tests.
148      Paper E


    Why does BART overestimate the available bandwidth when sending probe
packets from an older Linux version? And why is the throughput and available
bandwidth as low as 4 Mbps when the cross-traffic rate is zero when measuring
from a 2.6 kernel? This paper does not give answers to that, it rather states that
there is a problem with the Linux drivers and that care must be taken when
measuring the available bandwidth and link capacity in these scenarios. More
research is needed here.
    It should be noted that when the drivers on the cross-traffic nodes and the
probe-packet sender nodes are the same the estimates produced by BART are
in line with the UDP throughput tests. The measurements discussed later in
this paper are from experiments where all wireless nodes are running Linux
kernel 2.6.


9.5 Tuning the BART Kalman filter for qualita-
    tive estimation in 802.11b networks
The Q matrix plays an important role when configuring the Kalman filter ap-
plied in BART. K tells the Kalman filter how much weight a new measurement
sample zk should have compared to the previous estimate of the system state
ˆ
xk−1 , and K depends on P which in turn is calculated using Q. A high value
of Q can be used when the system state to be measured is expected to change
rapidly and a low Q is used when the measured system is predicted to be stable.
That is, it is possible to trade estimation stability for agility when deploying the
BART Kalman filter for estimating available bandwidth and link capacity.
    In [9] the Q matrix has been tuned in order to increase the accuracy of the
estimates produced by BART. In that study a wired network was used in which
the available bandwidth was allowed to change over time. Thus, the Q matrix
was mainly configured in order for the Kalman filter to react to changes in the
available bandwidth but not to changes in the link capacity estimate. In [11] a
general change detection algorithm was added to the existing Kalman filter in
order to be able to determine whether a change in the available bandwidth or
the link capacity actually has happened. Using the change detection algorithm
the Q matrix does not have to be fixed, rather it changes depending on whether
BART guesses that the measured system is stable or not.
    In this paper the relation between the available bandwidth and the measured
link capacity in 802.11b networks has been discussed. As argued for in Section
9.2, these are the same when using BART, if the probe-packet sender does not
inject cross traffic into the network. The cross traffic injected by other nodes
9.5 Tuning the BART Kalman filter for qualitative estimation in 802.11b
                                                     networks      149


                                     6                    Q = (0.0001 0.0; 0.0 0.0001)
                                 x 10
                             6

                             5
         Estimation (Mbps)




                             4

                             3

                             2

                             1
                                 0       50   100   150     200       250       300      350   400    450       500
                                                                                                     Time (ticks)
                                     6                     Q = (0.0001 0.0; 0.0 0.01)
                                 x 10
                             6

                             5
         Estimation (Mbps)




                             4

                             3

                             2

                             1
                                 0       50   100   150     200       250       300      350   400    450       500
                                                                                                      Time (ticks)



Figure 9.7: By varying the choice of Q the link capacity (dashed line) and
available bandwidth (solid line) estimates vary. In the upper graph q11 = q22 =
0.0001 and in the lower q11 = 0.0001, q22 = 0.01


in the wireless network causes a change in both the available bandwidth and
the estimated link capacity. The question this section tries to answer is what
impact the properties of 802.11b networks have on the choice of Q.
    In Figure 9.7 and 9.8 the estimated available bandwidth (solid lines) and
link capacity (dashed lines) are shown over time. The graphs originate from
the same measurement scenario; the probe packets are sent from the probe
packet sender and cross traffic is generated from one laptop in the testbed. The
cross traffic is zero during the first 250 time steps and then jumps up to 6 Mbps.
Remember the throughput tests discussed in the previous section that suggested
that the available bandwidth first was 4 Mbps and then 3 Mbps.
    The Q matrix differ in the four graphs and as can be observed, the impact
of the choice of Q is crucial to the stability of the estimates of the available
bandwidth and link capacity. To obtain the results shown in the top graph
in Figure 9.7 both q11 and q22 are low which instructs the Kalman filter to
expect low variance in the system state and thus it should give low weight to
new samples. That is, the Kalman filter does not anticipate sudden changes in
150      Paper E


                                     6                     Q = (0.01 0.0; 0.0 0.01)
                                 x 10
                             6

                             5
           Estimate (Mbps)




                             4

                             3

                             2

                             1
                                 0       50   100   150   200        250       300     350   400   450       500
                                                                                                   Time (ticks)
                                     6                    Q = (0.01 0.0; 0.0 0.0001)
                                 x 10
                             6

                             5
           Estimate (Mbps)




                             4

                             3

                             2

                             1
                                 0       50   100   150   200        250       300     350   400   450       500
                                                                                                   Time (ticks)




Figure 9.8: By varying the choice of Q the link capacity (dashed lines) and
available bandwidth (solid lines) estimates vary. In the upper graph q11 =
q22 = 0.01 and in the lower q11 = 0.01, q22 = 0.0001


available bandwidth nor link capacity estimates. As seen, the estimate of the
link capacity is stable and slowly reacts to the sudden change in cross traffic at
time 250. The available bandwidth estimate is also stable but seem to overreact
to the change in cross traffic at first. Later it moves towards the correct value
of 3 Mbps.
    In the lower graph in Figure 9.7 q22 is large compared to q11 and thus
                                    ˆ
each sample has large impact on β, that is the estimate of β that is a part of
the system state x. This is clearly visible; the link capacity estimate is stable
and reacts slowly to the change in cross traffic while the available bandwidth
estimate fluctuates a lot and overreacts to the change in cross traffic on the
wireless link.
    In the top graph in Figure 9.8 both q11 and q22 are large and thus both
estimates fluctuate because each sample is weighted heavily. In the lower graph
in Figure 9.8 q11 is large and q22 small. Now both the estimates of available
bandwidth and link capacity seem to quickly react to the sudden change in
cross-traffic rate while at the same time the values are not fluctuating as in
previous examples.
9.5 Tuning the BART Kalman filter for qualitative estimation in 802.11b
                                                     networks      151


                                          5
                                       x 10

                                 10


                                  9


                                  8


                                  7
           Difference (bit/s)




                                  6


                                  5


                                  4


                                  3


                                  2


                                   1
                                0.01
                                       0.008
                                               0.006                                                      0.002   0
                                                       0.004                                      0.004
                                                               0.002              0.008   0.006
                                                                       0   0.01

                                                                                          q11
                                                          q22




Figure 9.9: The absolute difference of the estimated link capacity and available
bandwidth on the z axis. The Q matrix varies in two dimensions on the x and
y axes. The cross traffic is zero during half the experiment and then increases
to 6 Mbps, that saturates the bottleneck.


    As described previously in this paper the link capacity estimate and the
available bandwidth estimate should be the same. Thus, the latter Q matrix
configuration seems to be the best choice in this experiment scenario.

9.5.1 Tuning of Q
To decide which Q to use in order to get the most accurate estimates one could
vary Q and then calculate the mean square error using the estimate and the true
value of the available bandwidth as input. This is possible in a network where
the true values of the available bandwidth and the link capacity are known,
such as in a wired network. However, in a wireless network the correct values
are not known since they vary with the radio quality between the sender and
the receiver. Therefore, another method is applied in this paper in order to find
the best choice of Q in the wireless setting. This is described below.
    Since the available bandwidth and the estimated link capacity theoretically
should be equal, according to the previous discussion, a good choice of the
Q matrix has been found by minimizing the difference between the available
152      Paper E



                                         5
                                      x 10

                                7.5

                                 7

                                6.5
           Difference (Bit/s)




                                 6

                                5.5

                                 5

                                4.5

                                 4

                           3.5
                          0.01
                                      0.008
                                              0.006                                                              0
                                                                                                         0.002
                                                      0.004                                      0.004
                                                              0.002                      0.006
                                                                                 0.008
                                                                      0   0.01
                                                                                         q11
                                                         q22




Figure 9.10: The absolute difference of the estimated link capacity and avail-
able bandwidth on the z axis. The Q matrix varies in two dimensions on the x
and y axes. The cross traffic is zero during the whole experiment.


bandwidth and the link capacity estimates. In Figures 9.9-9.11 the x and y axes
represent q11 and q22 , respectively. Thus the two-dimensional value space of
the Q matrix can be searched. The z axis is the absolute difference between the
available bandwidth and the link capacity estimates during an entire measure-
ment session (i.e. 500 time ticks, compare Figure 9.7 or 9.8).
    The graph in Figure 9.9 originate from a scenario where the cross-traffic
rate was 0 during 250 time ticks and then 6 Mbps during the rest of the session
(corresponding to the results shown in Figures 9.7 and 9.8). Figure 9.10 origi-
nate from a session where there was no cross traffic at all during 500 time ticks
while the graph in Figure 9.11 is derived from when one cross-traffic sender is
injecting 3 Mbps during the first 250 time ticks. Then two cross-traffic senders
are injecting probe traffic at 3 Mbps each during the rest of the experiment (i.e.
250 additional time ticks).
    The graphs illustrate that the difference between available bandwidth and
link capacity estimates is minimized when using a low value of q22 , indepen-
dent of the cross-traffic scenario. The value of q11 seems to depend on the
cross-traffic scenario though. If q11 is small, the Kalman filter does not antic-
ipate sudden changes, thus a small value of q11 minimizes the difference be-
9.5 Tuning the BART Kalman filter for qualitative estimation in 802.11b
                                                     networks      153


                                       5
                                    x 10

                                9


                                8


                                7


                                6
           Difference (bit/s)




                                5


                                4


                                3


                                2


              1
           0.01
                                    0.008
                                            0.006                                                              0
                                                    0.004                                              0.002
                                                                                               0.004
                                                            0.002                      0.006
                                                                    0          0.008
                                                                        0.01

                                                                                       q11
                                                       q22




Figure 9.11: The absolute difference of the estimated link capacity and avail-
able bandwidth on the z axis. The Q matrix varies in two dimensions on the
x and y axes. The cross traffic is 3 Mbps during half the experiment and then
increases to 3 + 3 Mbps send from two nodes, that saturates the bottleneck.


tween available bandwidth and link capacity estimates if the cross-traffic rate
is constant. On the other hand, a high value is better if the cross-traffic rate
actually changes as in the scenarios corresponding to Figure 9.9 and 9.11.


9.5.2 Theoretical discussion of choice of Q
The following is a theoretical discussion on why the above values of Q should
be used when measuring in 802.11b networks.
    As described in Section 9.2 the available bandwidth B and the link capacity
                                                                       1
C is derived from the system state vector xk by B = −β and C = α . Also,
                                                           α
the available bandwidth and the link capacity are equal in 802.11b networks
when no cross traffic is sent from the probe-packet sender node. That is C =
B which means that β = −1. Hence, β is constant and thus q22 should be
                                       ˆ
small since the expected variation in β will be small. On the other hand, if the
estimated link capacity varies, that is the cross traffic from other nodes in the
wireless network varies, q11 must be larger in order for the Kalman filter to
154     Paper E


adapt to changes in α.
   This corroborates the results shown in the previous subsection.


9.6 Conclusions
In this paper a state-of-the-art available bandwidth measurement method called
BART has been used to study the impact of wireless 802.11b bottleneck links
on the produced estimates of both available bandwidth and link capacity.
    In the paper it has been discussed and argued that the estimated link ca-
pacity and available bandwidth are equal when the bottleneck is a wireless
802.11b link using BART. Further, Linux driver problems associated with mea-
surements in 802.11b networks are discussed. For example, in some cases the
send rate of a probe-packet train is lower than anticipated when injected by
the sender. This can bias the estimations. Further, the available bandwidth is
not estimated correctly if the probe-packet sender uses one Linux kernel ver-
sion while the cross-traffic generator is running on another version. This is
important to be aware of.
    The Kalman filter used by BART has been tuned in order to improve ac-
curacy when measuring available bandwidth and link capacity in 802.11b net-
works. It was shown that the configuration of Q must be different in these
networks compared to ordinary single-access wired networks.
Bibliography

[1] Demetres Antoniades, Manos Athanatos, Antonis Padadogiannakis,
    Evangelos P. Markatos, and Constantine Dovrolis. Available bandwidth
    measurement as simple as running wget. In Proceedings of the Passive
    and active measurement workshop, Adelaide, Australia, 2006.
[2] V. Ribeiro, R. Riedi, R. Baraniuk, J. Navratil, and L. Cottrel. pathChirp:
    Efficient available bandwidth estimation for network paths. In Proceed-
    ings of the Passive and Active Measurement Workshop, San Diego, USA,
    2003.
[3] Manish Jain and Constantinos Dovrolis. End-to-end available bandwidth:
    Measurement methodology, dynamics, and relation with TCP throughput.
    In Proceedings of the ACM SIGCOMM conference, Pittsburg, USA, Au-
    gust 2002.
[4] Strauss, Katabi, and Kaashoek. A measurement study of available band-
    width estimation tools. In Proceedings of the ACM SIGCOMM Internet
    Measurement Workshop, Miami, USA, 2003.
                          o
[5] Bob Melander, Mats Bj¨ rkman, and Per Gunningberg. Regression-based
    available bandwidth measurements. In Proceedings of the SCS Sympo-
    sium on Performance Evaluation of Computer and Telecommunications
    Systems, San Diego, USA, July 2002.
[6] R.S. Prasad, M. Murray, C. Dovrolis, and K. Claffy. Bandwidth estima-
    tion: metrics, measurement techniques, and tools. IEEE Network Maga-
    zine, 2003.
                                                             a
[7] S. Ekelin, M. Nilsson, E. Hartikainen, A. Johnsson, J. M˚ ngs, B. Me-
                     o
    lander, and M. Bj¨ rkman. Real-time measurement of end-to-end avail-
    able bandwidth using kalman filtering. In Proceedings of the IEEE/IFIP

                                    155
     Network Operations and Management Symposium, Vancouver, Canada,
     2006.
 [8] G. Bishop and G. Welch. An introduction to the kalman filter. In SIG-
     GRAPH 2001, Course 8, 2001.
 [9] Erik Hartikainen and Svante Ekelin. Tuning the temporal characteris-
     tics of a kalman-filter method for end-to-end bandwidth estimation. In
     Proceedings of the IEEE/IFIP workshop on End-to-end Monitoring Tech-
     niques and Services, Vancouver, Canada, 2006.
                                o
[10] Andreas Johnsson, Mats Bj¨ rkman, and Bob Melander. An analysis of
     active end-to-end bandwidth measurements in wireless networks. In Pro-
     ceedings of the IEEE/IFIP workshop on End-to-end Monitoring Tech-
     niques and Services, Vancouver, Canada, 2006.
[11] Erik Hartikainen and Svante Ekelin. Enhanced network-state estimation
     using change detection. In Proceedings of the first IEEE LCN Workshop
     on Network Measurements, Tampa, USA, 2006.

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:2
posted:1/14/2013
language:English
pages:166