simulation support for
scalable routing experiments
Yue Li*, Jason Liu, and Raju Rangaswami
School of Computing and Information Sciences,
Florida International University, Miami, Florida 33199, USA
Abstract: This article describes a new software infrastructure that combines the scal-
ability and ﬂexibility beneﬁts of real-time network simulation with the realism of open-
source routing protocol implementations. The infrastructure seamlessly integrates the
open-source XORP router implementation with a real-time large-scale network sim-
ulation engine. The design uses a novel forwarding plane oﬄoading approach that
decouples routing from forwarding and conﬁnes the more resource consuming forward-
ing operations inside the simulation engine to reduce I/O overhead. Experiments
demonstrate superior performance of the software routing infrastructure without im-
pairing accuracy. The infrastructure is shown to able to support large-scale routing
experiments on light-weight virtual machines.
Keywords: network simulation; real-time simulation; network emulation; routing pro-
tocols; large-scale routing experiments
Reference This article is an extended version of earlier work published in the Pro-
ceedings of the 22nd Workshop on Principles of Advanced and Distributed Simulation
Biographical notes: Yue Li is a postdoctoral research associate in the School of
Computing and Information Sciences, Florida International University. He received
Ph.D. degree in Computer Science from Xi’an Jiao Tong University, China in 2005.
He joined Tsinghua National Laboratory for Information Science and Technology as
a postdoc from 2005 to 2007. His research interests include network simulation and
emulation, high performance simulation, and multimedia networking.
Jason Liu is an Assistant Professor in the School of Computing and Informa-
tion Sciences, Florida International University. His research focuses on parallel
discrete-event simulation, performance modeling and simulation of computer systems
and communication networks. He received a B.A. in Computer Science from Beijing
University of Technology in China in 1993, an M.S. in Computer Science from College
of William and Mary in 2000, and a Ph.D. in Computer Science from Dartmouth
College in 2003.
Raju Rangaswami is an Assistant Professor of Computer Science at the Florida
International University in Miami. He received a B.Tech. degree in Computer
Science from the Indian Institute of Technology, Kharagpur, India. He obtained
M.S. and Ph.D. degrees in Computer Science from the University of California at
Santa Barbara. His research interests include operating systems, storage systems,
virtualization, security, and real-time systems.
1 INTRODUCTION niques can help evolve our understanding of complex net-
work problems, such as the stability issues with BGP and
Researchers are continuously searching for techniques that OSPF (Basu and Riecke, 2001; Shaikh et al., 2000; Shaikh
can better evaluate the dynamic behavior of routing proto-
cols in complex, large-scale network scenarios. Such tech- Copyright c 200x Inderscience Enterprises Ltd.
and Greenberg, 2001; Shaikh et al., 2002), BGP conver- scribe below:
gence (Obradovic, 2002), and BGP security and miscon-
ﬁguration (Goodell et al., 2003; Chim and Yeung, 2004;
Mahajan et al., 2002; Nordstrom and Dovrolis, 2004). Ex-
• Realism. Our infrastructure uses a previously de-
isting techniques for understanding and evaluating net-
veloped real-time network simulator, called PRIME
work routing protocols can be categorized as analytical,
(2008). PRIME provides an emulation infrastructure
simulation-based, and emulation-based. Here we exclude
which supports a seamless integration of a large num-
direct experimentation on physical networks, because it
ber of prototype network applications or protocols
is infeasible to conduct custom large-scale routing experi-
into the experimental environment. The infrastruc-
ments directly on routers deployed deep in the Internet.
ture we have developed integrates PRIME with the
Analytical techniques, which can often bring valuable open-source XORP (2008) router software, making it
insight to the design of large-scale complex systems, are possible to conduct large-scale routing experiments.
ill-equipped to address the complexities and dynamics in- This technique also achieves a high level of realism
herent with the actual protocol execution. Simulation is since actual network protocol implementations are di-
used commonly by researchers to perform large-scale rout- rectly included into the experiments.
ing experiments since this technique can capture the net-
work dynamics with a great amount of detail. (Bauer et al.,
2006; Liu and Chien, 2004; Dimitropoulos and Riley, 2004).
However, a simulation model of a router’s behavior can be • Scalability. Traﬃc on the virtual network must be
quite diﬀerent from a real routing protocol implementation forwarded based on routing decisions made by XORP
and thus may not provide the necessary realism of protocol instances. If each packet must be exported from the
execution behavior. Further, simulation implementation of simulator and sent to an XORP router instance, and
routing protocols typically comes with only a modest se- then immediately received and imported back into the
lection of protocols and protocol versions, thus limiting its simulator as soon as forwarding is achieved at the
utility. router, the I/O overhead would be substantial and
Emulation-based techniques allow the execution of net- might limit the scale of the routing experiments. We
work routing protocols “as is” within an emulated net- propose a forwarding plane oﬄoading approach, which
work environment, providing the necessary realism in the allows the XORP router to designate the packet for-
protocol execution. With the emergence of open-source warding function to the simulator and communicates
router software, such as XORP (2008), Zebra (2008), and with the simulator its routing decisions through an
Quagga (2008), researchers can now prototype and eval- out-of-band channel. In doing so we can conﬁne the
uate routing protocols on actual systems as part of the background traﬃc (unrelated to routing) within sim-
emulation testbed. Emulation-based techniques, however, ulation, thereby eliminating the I/O overhead associ-
must address the issues of scalability and ﬂexibility. The ated with the bulk of the network traﬃc. Further-
scalability of an emulation testbed is constrained by the more, since packet transmissions can be parallelized
available physical resources. For example, experimental somewhat easily, we are able conduct routing exper-
network scenarios must observe the limits on bandwidth, iments with a network size far beyond what can be
latency, and node counts of the underlying physical infras- supported by emulation testbeds.
tructure. Emulation-based solutions are thus incapable of
creating arbitrary network topologies and controlling back-
ground network traﬃc conditions. • Flexibility. It is straightforward to apply real-time
Our position is that the design of a successful framework simulation to explore a wide spectrum of network sce-
for routing experiments must comprehensively address the narios and address what-if questions (e.g., by inject-
three challenges of realism, scalability, and ﬂexibility. In ing network failures). Real-time simulation is based
this article, we describe an infrastructure that we have on simulation. Once a simulation model is developed,
built to support scalable routing experiments based on a reasonably veriﬁed and validated, it takes little eﬀort
real-time network simulation system. A key advantage of to conduct simulation experiments to cover a large
our approach is that, since it provides a real-time commu- parameter space. The proposed infrastructure also al-
nication interface to real client machines, commodity ap- lows incorporating analytical models in the real-time
plications can be executed unmodiﬁed within this frame- network simulation. For example, a user of the in-
work (Liu, 2008). Consequently, if engineered correctly, frastructure can employ low-resolution models to de-
such a simulation framework can provide a realistic net- scribe aggregate Internet traﬃc (e.g., Liu, 2006) and
work infrastructure indistinguishable from a real network thus considerably increase the size of the network un-
from the standpoint of the real applications. When this der test. Further, it is relatively easy to manipulate
framework is coupled with machines running open-source and control dynamic network behavior with simula-
router software, it can provide a realistic, scalable, and tion. For instance, the user can bring down a link in
ﬂexible infrastructure for conducting routing experiments. the virtual network and observe the reaction mecha-
We address the three challenges comprehensively as we de- nism of the routing protocols.
2 BACKGROUND PRIME Environment
Virtual Virtual network
Our infrastructure for scalable routing experiments builds
on top of a real-time parallel simulation engine called
PRIME, which stands for a Parallel Real-time Immer- Simulation gateway
sive Modeling Environment (PRIME, 2008). PRIME en- Router 10.10.0.2
ables real-time large-scale network simulations and sup-
10.10.1.20 10.10.1.9 10.10.1.20
ports seamless interactions with real distributed applica-
tions. PRIME is built on top of a parallel discrete-event Virtual
simulation kernel that implements the Scalable Simulation Router
Framework (SSF) (Cowie et al., 1999). The parallel sim- Virtual
ulation kernel deals with synchronization and communi-
VPN server ssfgwd
Simulation gateway Scalable Cluster
cation issues between logical processes that execute sim-
ulation events in parallel, while PRIME itself enables the Figure 1: The emulation infrastructure based on VPN.
notion of real time by incorporating packets from exter-
nal application as real-time events. In order to meet the
deadline requirement of the real-time events, PRIME em- IP address 10.10.1.9.
ploys a greedy priority-based scheduling algorithm—real- A distinct advantage of this approach is that the emula-
time events are assigned with a higher priority to ensure tion infrastructure does not require special hardware to set
their timely processing. up. It is also secure, a merit inherited directly from the un-
PRIME can interact with a large number of clients, derlying VPN implementation, and scalable, since clients
which may be traﬃc sources, sinks, or other network enti- need not be co-located within the simulation infrastruc-
ties like routers. It is important to be able to incorporate a ture. Further, multiple simulation gateways can be used
large number of such clients and yet remain transparent to simultaneously to accommodate larger traﬃc capacity be-
the applications running on the clients. There are several tween the clients and the real-time network simulator. In
ways to incorporate real applications as part of the sim- order to produce accurate results, however, the emulation
ulation environment, which include using packet captur- infrastructure needs a tight coupling between the emulated
ing techniques (such as libpcap, IP table, and IP tunnel), entities (i.e., the clients) and the real-time simulator. In
preloading dynamic libraries, and modifying the binary ex- particular, the segment between the clients and the real-
ecutables, and so forth. PRIME instead uses an open sys- time network simulator needs to include low-latency links.
tems approach based on a Virtual Private Network (VPN) To maintain potentially high traﬃc throughput demand,
solution (Liu et al., 2007). We customize VPN to function this segment must also have suﬃcient bandwidth for tun-
as a gateway that bridges traﬃc between the clients and neling traﬃc through the emulation infrastructure.
the simulated network. Clients establish connection to the In the next section, we develop techniques that allow the
simulation gateway. Traﬃc generated by the clients and above real-time simulation infrastructure to scale up when
destined for the virtual network is directed by the mod- used speciﬁcally for network routing experiments.
iﬁed VPN through the gateway to the real-time network
We illustrate this approach using an example. Suppose
we have two BGP routers which need to build up the peer- 3 SCALABLE ROUTING INFRASTRUCTURE
ing relationship with each other. VPN sets up two TUN
interfaces with IP addresses of 10.10.0.2 and 10.10.1.8 for The availability of open-source router platforms, such as
one router, similarly 10.10.1.9 and 10.10.1.20 for another XORP, Zebra, and Quagga, has simpliﬁed the task of re-
router. The corresponding simulated nodes have the same searchers, who can now prototype and evaluate routing
IP address conﬁguration, as shown in Figure 1. One BGP protocols with relative ease. To support experiments on a
speaker ﬁrst sends an OPEN message with destination IP large-scale network consisting of many routers with multi-
address of 10.10.1.9 via its interface 10.10.1.8, then the ple traﬃc sources and sinks, we propose to integrate the
packet is sent to the modiﬁed VPN server at the simu- open-source router platforms with a real-time network sim-
lation gateway through a VPN connection. The simula- ulation infrastructure.
tion gateway forwards the packet to PRIME via a dedi-
cated TCP connection. In PRIME, the packet is treated 3.1 Virtual Router
as a real-time event and injected into the simulation event
queue. PRIME simulates the packet being forwarded on In order to scale up the experiment in an easy way we adopt
the virtual network as if it was created by the simulated virtualization technology to host multiple router instances.
node with the same IP address 10.10.1.8. Upon reaching Virtualization helps maintain isolated namespaces, each of
the node 10.10.1.9, the packet will be exported from sim- which manages the virtual resources necessary for execut-
ulation engine, traveling in the reverse direction via the ing a router instance. We therefore call each instance a
simulation gateway back to the external router with the virtual router. Speciﬁcally, four types of network resources
XORP Virtual Router
Table 1: Network virtualization in Xen, OpenVZ, Linux-
VServer and Linux-VRF. Routing Process
Xen OpenVZ VServer VRF update
Network sockets Forwarding
Routing Information Base
Network interfaces packets (FIB) Manager
Forwarding table offloading
I/O Manager PRIME Agent
shall be provided within a virtual router: network sock-
ets, network interfaces, forwarding table, and loopback de- R1 Output
vice. Routing protocols, e.g., BGP, RIP, and OSPF, need Forwarding Table
to send and receive packets via standard network sockets
H1 Forwarding H3
(TCP, UDP, and raw sockets) in order to establish peering Engine
connectivity and exchanging routing information. Network
interfaces and the forwarding table together represent the
network forwarding engine of the virtual router on which Figure 2: Oﬄoading the forwarding plane to the PRIME
routing protocols are run. A network loopback device is simulation environment.
sometimes used by the routing protocols to communicate
among separate processes (e.g., see XORP, 2008).
In particular, we investigate four popular virtualization entirely with PRIME, we can eliminate the I/O overhead
technologies: Xen (Barham et al., 2003), OpenVZ (2008), associated with communicating bulk-traﬃc back and forth
Linux-VServer (Soltesz et al., 2007) and VRF (2008). As between the router software and the real-time simulator.
we can see in Table 1, all four types of network resources This approach, however conceptually straightforward,
are provided in Xen and OpenVZ, while Linux-VServer raises two key design questions. First, the forwarding table
and VRF have only partial network virtualization support. for a speciﬁc virtual node used by the simulator for inter-
For this reason, we focus only on Xen and OpenVZ in our nal forwarding decisions must be synchronized with the
experiment studies. actual forwarding table, which is updated by the routing
We also examined the scalability aspects of these net- protocols running within the corresponding external router
work virtualization choices. We found that the net- instance. The synchronization is critical to ensure correct
work stack in Xen (version 3.0.4) becomes unstable un- forwarding operations within the simulation engine. Sec-
der high traﬃc load with 32 or more domains. This phe- ond, the simulation engine must be instantly informed of
nomenon is consistent with similar observations in other any network interface reconﬁgurations performed at the
studies (Padala et al., 2007; Bhatia et al., 2008). As a re- external router instance. For instance, a network admin-
sult, we choose OpenVZ as the platform for our large-scale istrator may create virtual network interfaces or reset the
routing experiments. MTU of an existing network interface; in such cases, the
corresponding simulated router node must be kept consis-
tent with the changes.
3.2 Forwarding Plane Oﬄoading
PRIME provides real-time simulation capabilities and an 3.3 The XORP Architecture
emulation infrastructure that can seamlessly integrate mul-
tiple clients, may they be routers or otherwise. Since the In our current implementation, we combine the open-
routers must be emulated outside PRIME on client ma- source XORP router software with PRIME to provide
chines where they can run the real routing software di- a scalable platform for conducting routing experiments.
rectly, every packet traveling along its path from the source Two reasons prompted our choice of XORP. First, XORP
to the destination must be exported to each intermediate has the most up-to-date implementations of routing pro-
router for forwarding decisions, and subsequently imported tocols, including those commonly used protocols, such as
back into the PRIME simulation engine. Thus, the for- BGP, OSPF, and RIP; it’s operation is consistent with
warding operation for each packet at each hop incurs a real router software and it is a well-established experimen-
substantial I/O overhead. Consequently, the overall over- tal platform for routing protocol design and evaluation by
head will signiﬁcantly impact the performance of the in- the routing protocol research community. Second, XORP
frastructure, especially in large-scale routing experiments. implements a Forwarding Engine Abstract (FEA) layer,
To avoid this problem, we propose a forwarding plane which splits the routing and forwarding plane as two sep-
oﬄoading approach, which moves the packet forwarding arate layers within its implementation, thus considerably
functions from the router software to the simulation en- simplifying our task of oﬄoading the latter to PRIME.
gine. Since packet forwarding operations are carried out The primary role of FEA is to shield XORP processes
Xen Xen Xen
from concerns of forwarding plane operations, so that arbi- Domain U Domain U Domain U
trary forwarding mechanisms can be employed via plug-ins. XORP
FEA has three main components: Forwarding Information
Base (FIB) manager, interface manager, and I/O manager.
These components are depicted in Figure 2, along with the
corresponding XORP processes that interact with FEA.
The FIB manager receives forwarding table updates from S R1 R2 ··· Rn C
the routing information base (RIB) process, which arbi-
trates route updates among routing protocols (e.g., BGP PRIME
or OSPF). The FIB manager propagates the necessary Xen Domain0
changes (such as inserting or deleting routing entries) to
the underlying forwarding engine. Currently, XORP can Figure 3: Model topology used for microbenchmark exper-
choose either the traditional Unix kernel (via Netlink) or iments.
the Click modular software router (Kohler et al., 2000) for
forwarding. Access to the forwarding plane is performed corresponding router node. The emulation protocol ses-
by the corresponding plug-ins. sion interprets the control commands and executes them
The router manager process can also issue interface con- by modifying the forwarding table on the router node as di-
ﬁguration requests to the FEA interface manager, accord- rected by XORP. The procedure of interface management
ing to a pre-speciﬁed router conﬁguration ﬁle or real-time is essentially the same. The interface manager can modify
input from the system administrator via a command-line the network interface conﬁguration at the corresponding
interface. FEA interprets and executes these requests on router node through primitives provided by the Interface
behalf of the underlying forwarding engine. Individual Agent of the FEA plug-in.
routing processes can register with FEA to be notiﬁed of Since forwarding table updates and interface manage-
changes in the interface conﬁguration. The I/O manager ment requests are propagated to the simulation engine,
also provides an interface for all the routing protocols to the forwarding decisions inside simulation are synchronized
communicate routing information (such as OSPF’s Link with the XORP instances, except when the update com-
State Update messages) with their peers on other router mands are in transit. (We investigate the possible discrep-
instances. ancy in Section 4.3). The routing messages created by
the XORP routing protocol instances are treated as fore-
3.4 XORP Forwarding Plane Oﬄoading Imple- ground traﬃc from the perspective of PRIME and they
mentation are forwarded through the simulation engine with the aid
of the emulation infrastructure as described in Section 2.
Figure 2 describes our architecture for forwarding plane This mechanism is enabled by independent data channels
oﬄoading with XORP. We create an additional XORP on top of VPN automatically.
forwarding plane plug-in within FEA, which we call the
PRIME Agent. It consists of two components: the PRIME
Socket and the Interface Agent. Both components main- 4 EXPERIMENTS
tain a common command channel with the PRIME simu-
lator for transferring forwarding information updates (via We conducted three sets of experiments to evaluate our
the PRIME Socket) and interface conﬁguration requests infrastructure for performance, accuracy, and scalability.
(via the Interface Agent) to the corresponding router node The initial set of experiments (Section 4.1) are controlled
in PRIME. This eﬀectively results in the forwarding func- microbenchmarks that allow ﬁne-grained evaluation of the
tionality being oﬄoaded from XORP to PRIME; traﬃc on proposed infrastructure, both with and without forward-
the simulated network can thus be kept within simulation. ing plane oﬄoading to the simulator. Next, we carried
The interaction between the PRIME Agent and the sim- out an intra-domain routing experiment of a real network
ulation mechanism is managed as follows. The PRIME topology (Section 4.2) to demonstrate the capability of our
Socket provides primitives to manage the forwarding table infrastructure. Finally we conducted scalability studies to
in PRIME, including those that add or remove a single for- explore how large an experiment can be supported by our
warding entry as well as the entire forwarding table. The routing infrastructure.
PRIME Socket accepts a command from the FIB manager,
packages the command into a control packet, and sends it
through the command channel. We designate an unused
address as the destination address of the control packets, Since PRIME needs to interact with the XORP instances
so that they can be distinguished from regular data packets and simulate the virtual network in real time, the timing
when reaching the simulator through the emulation infras- accuracy is paramount in obtaining trustworthy experi-
tructure. At the PRIME-end of the command channel, mental results. This means that the infrastructure (both
the control packets are translated into simulation events PRIME and the external XORP router instances) should
and presented to the emulation protocol session at the keep up with the wall-clock time, receive real-time events
2 Mb/s offered load, offloading disabled
1 Mb/s offered load, offloading disabled
800 2 Mb/s offered load, offloading enabled
RTT (ms), 95% confidence interval
1 Mb/s offered load, offloading enabled
Observed throughput (Mb/s)
700 ideal round-trip time 1.5
2 Mb/s offered load, ideal throughput
0.5 2 Mb/s offered load, offloading enabled
2 Mb/s offered load, offloading disabled
100 1 Mb/s offered load, ideal throughput
1 Mb/s offered load, offloading enabled
1 Mb/s offered load, offloading disabled
12 4 8 16 32 12 4 8 16 32
Number of XORP instances Number of XORP instances
Figure 4: Impact on round-trip delays. Figure 5: Impact on throughput.
promptly and output them without missing their real-time We varied the UDP background traﬃc between the server
deadlines. The purpose of our microbenchmark-based eval- and the client to be either 1 Mb/s or 2 Mb/s. When we
uation was to illustrate that the proposed infrastructure disabled forwarding plane oﬄoading, the observed round-
can support scalable and real-time routing experiments trip times deviated from the ideal case, especially when the
with the forwarding plane oﬄoading technique. number of intermediate XORP instances was large. The
We used an extensible network model for microbench- two curves representing the cases with oﬄoading disabled
marking, as shown in Figure 3. Node S and C acted overlap. The observed values deviated by as much as 30%
as the server and client, respectively; the server initiated with 32 XORP instances. With 2 Mb/s oﬀered load, the
constant-bit-rate UDP traﬃc to the client. The interme- variation of the round-trip time was slightly larger. Note
diate nodes, R1 , R2 , ... , Rn , were routers. Each router that the link bandwidths were set to 100 Mb/s, which was
node had one corresponding XORP instance running out- substantially larger than the oﬀered load, and consequently
side PRIME. We conﬁgured each XORP instance to run a should not noticeably aﬀect the round-trip times in the
OSPFv2 protocol. ideal case. The observed values when oﬄoading was en-
For the microbenchmark experiment we adopted Xen abled matched almost exactly with the ideal values, with
virtual machines (Barham et al., 2003) to host PRIME less than 0.2% error, regardless of the oﬀered load.
and XORP instances. In Xen terminology, each domain is Figure 5 depicts the eﬀect of the simulation infrastruc-
a virtual machine running on the host machine. Domain ture overhead on traﬃc throughput between the client and
0 is the ﬁrst domain created when the host is powered on server nodes, again as we varied the number of intermedi-
and has privileges to access physical machine resources. It ate XORP router instances. The throughput is calculated
also provides an interface to manage other domains (called by the total size of data transferred from the server to
Domain U’s) run unprivileged. the client divided by the time since the download request
We set up the Xen environment on a DELL OPTIPLEX is made by the client. The throughput decreased slightly
745 workstation with Intel Core 2 Duo 2.4 GHz processors (even in the idea case) with more router instances and
and 4 GB memory. Both the host OS and guest OS ran De- therefore longer path between the server and the client.
bian Linux 4.0 distributions with kernel version 126.96.36.199. With oﬄoading disabled, the observed throughput was
PRIME and the simulation gateway ran in Xen Domain lower than that of the ideal case, becoming more apparent
0 and each Domain U contained one XORP instance. We with more intermediate XORP instances. With oﬄoading
wrote a set of scripts to support rapid deployment of an enabled, the observed throughput deviated only slightly
arbitrary network experiment scenario. (less than 3% with 32 XORP instances). As expected,
To evaluate performance of the simulation infrastructure since the data download traﬃc was routed within simula-
and its impact on timing ﬁdelity, we measured round-trip tion, rather than being forwarded through the emulation
delay and throughput as the primary metrics for evalu- infrastructure, we achieved much better accuracy.
ation. We evaluated two settings of the simulation in-
frastructure implementation: oﬄoading-disabled (in which 4.2 An Intra-Domain Routing Experiment
the forwarding plane oﬄoading optimization was not used)
versus oﬄoading-enabled (in which the optimization was In this section we describe an intra-domain routing ex-
employed). periment as a case study to demonstrate that a realistic
Figure 4 depicts the round-trip delay measured by ping routing experiment can be achieved with the integration
between the client and the server as we increased the num- of PRIME and XORP. The experiment was originally used
ber of intermediate router hosts running XORP. The prop- by VINI (Bavier et al., 2006) to demonstrate its capability
agation delay of individual links were set to be 10 ms to conduct elaborate routing tests on the PlanetLab. The
each. Consequently, the ideal round trip time would be experiment consisted of a realistic Abilene network model
(2×10×n), where n is the number of intermediate routers. with eleven routers, as shown in Figure 6. We conﬁgured
Denver Indianapolis Washington D.C.
Figure 6: The Abilene network.
Figure 8: TCP transfer during link failure and recovery.
100 1 te N
0 te N
Ping round-trip time (ms)
0R 1R 1R
0 10 20 30 40 50 60 70
Time (seconds) 0R 1R
Figure 7: OSPF route convergence indicated by ping RTT. 2R 3R
NAL N AL
NAL 4R 5R 6R
the model using the real topology, with link delays and N AL 3 te N
bandwidths obtained from the observatory data available NAL N AL NAL
at Abilene Observatory Web Site (Abilene Network, 2008).
2 te N
In particular, our model reﬂected a snapshot of the Abi-
lene backbone on Jan 14 2008 at 07:45:11 UTC. We also
Figure 9: A campus network represents one AS, where the
conﬁgured each XORP instance to run OSPFv2 protocol.
router R0 in N et0 acts as the border router of the AS.
We used the same Xen virtual machine environment as
described in Section 4.1 for the experiment setup.
The purpose of the experiment is to observe the conver- found (at 32 seconds). At 50 seconds, the link was re-
gence of OSPF and its eﬀect on data traﬃc. We injected paired, the shortest path was re-established, and a better
a link failure, followed by a recovery between the routers throughput was achieved (demonstrated by a steeper slope
at Chicago and Indianapolis (via the XORP command-line in the ﬁgure).
interface). We measured the eﬀect on the round-trip time
and data throughput between Denver and Chicago. Fig-
4.3 Scalability Studies
ure 7 shows the round-trip time measured by ping. When
the link went down at 14 seconds, OSPF instances recom- The objective of our scalability studies is to explore how
puted the routes resulting a diversion of the traﬃc from the large an experiment can be supported by our routing in-
upper links to the lower ones. The round-trip delays were frastructure with the virtual router technique. The net-
increased from around 36 ms to around 60 ms. The link work model for this set of experiments consisted of multi-
was later repaired at 43 seconds into the experiment. Af- ple Autonomous Systems (ASes), each being an identical
ter the routes converged, OSPF re-established the shortest medium-size network called the campus network (shown in
path between the two cities. Figure 9). The campuses were connected to form a ring-
We also used iperf to initiate a TCP transfer from Den- like topology. This campus network model was used by
ver to Chicago in a separate run and analyzed the tcpdump DARPA NMS program as the baseline network model for
results. Figure 8 shows the number of bytes received at studying simulation performance (Nicol, 2008). In particu-
the receiver end (i.e., the router at Chicago). The TCP lar, the campus network consisted of 508 end hosts and 30
transfer was interrupted when link failure occurred at 18 router nodes. We set the delay and bandwidth of all links
seconds and then resumed when an alternative route was between the routers to 10 ms and 100 Mbps, respectively.
BGP Table 2: Memory usage (in MB) by XORP instances.
3000 # VRs 30 90 150 210 240
Memory 385.6 1570.6 3640.6 6295.6 7879.9
Number of packets
study; the other is the cost due to routing information
maintained at each virtual router. The latter requires an
500 O(m × p × n2 ) memory consumption, where n is the num-
ber of virtual routers involved in the study, p is the average
0 1000 2000 3000 4000 5000
6000 7000 8000 9000 number of preﬁxes associated with each virtual router, and
m is the memory consumed per preﬁx in the XORP RIB.
Figure 10: Routing packets seen by the emulation gateway As a result of these two factors, we observe a super-linear
with 240 XORP instances. increase in the memory consumption.
Figure 10 shows the number of OSPF and BGP pack-
ets which passed through the routing infrastructure in an
experiment consisting of eight campuses, i.e., with 240 vir-
20 tual routers. The sampling interval was 10 seconds. We
can clearly observe the periodic OSPF link state refresh-
ing behavior that occurred at 30-minute intervals (with
signiﬁcantly higher traﬃc through the routing infrastruc-
ture). In the default setting, OSPF requires that each Link
State Advertisement (LSAs) be refreshed every 30 minutes.
That is, the OSPF router that originates an LSA needs
5 to increase the LSA’s link state sequence number and re-
ﬂood the network every 1800 seconds. Since we started the
routers sequentially during a period of approximately 25
0 1000 2000 3000 4000 5000
6000 7000 8000 9000
minutes, we observe that the impact from this refreshing
behavior was spread out accordingly. We used the default
Figure 11: CPU utilization with 240 XORP instances. conﬁguration for the BGP speakers that exported all local
IP preﬁxes. Consequently, the OSPF link state refreshing
triggered BGP updates to be sent to BGP peers. Thus, the
Each router node had one corresponding XORP virtual BGP traﬃc ﬂuctuated similarly with the OSPF traﬃc.
router running outside of PRIME. The virtual routers were
conﬁgured to adopt OSPFv2 and BGP4 as IGP and EGP Figure 11 shows the CPU utilization on the XORP ma-
respectively. Each AS installed one border router acting chine in the same run. The CPU utilization varied consis-
as a BGP speaker and peering with two BGP speakers in tently with the behavior of routing protocols. We notice
neighboring ASes. We scaled up the experiment by in- that the peak CPU utilization was less than 25%. We can
creasing the number of ASes. conclude that memory, instead of CPU, was the critical
We used two DELL PowerEdge 1900 servers for this ex- resource constraint for scaling up the experiment.
periment, which were connected with a 100 Mbps Ether- We also measured the latency of the emulation in-
net. Each machine had an Intel Quad-Core Xeon 2.0 GHz frastructure, which is the major contributor to the la-
CPU and 8 GB memory. We set up OpenVZ environment tency overhead for synchronizing forwarding information
on one machine, which hosted all the XORP virtual routers between PRIME and XORP instances. We enabled the
as well as the simulation gateway. PRIME ran on the other LOOPBACK option in PRIME, in which case the reader
machine. We wrote a set of scripts to support automatic thread immediately forwards the packets to the writer
deployment of arbitrary network experiment scenarios. In thread upon receiving them. Then we used ping to send
particular, we started the XORP instances sequentially one 52-byte packets (i.e., the same size as the the forward-
after another using a script. In the case of eight campuses, ing information update packets) from one virtual router to
the entire initialization could take about 25 minutes. another. Since the packets actually traverse the emulation
Table 2 shows the memory consumption on the XORP infrastructure four times during a round trip, we can easily
machine as we increased the number of virtual routers. calculate the latency imposed by the emulation infrastruc-
The memory usage depends on two main factors: one is ture. The result shows an average latency of 0.11 ms, which
the ﬁxed memory consumption associated with the initial- conﬁrms our belief that no signiﬁcant forwarding skew be-
ization of each virtual router, which leads to a cost pro- tween PRIME and XORP instances could occur with our
portional to the number of virtual routers included in the forwarding plane oﬄoading technique.
5 RELATED WORK with an emulated implementation of open-source XORP
router software to provide a realistic, scalable, and ﬂexible
Most current real-time network simulators are based on testbed for routing experiments. Oﬄoading the forward-
existing network simulators added with the capabili- ing plane from the XORP instances to the simulator allows
ties of interacting with real applications. Examples in- scalability, by conﬁning bulk traﬃc to be forwarded more
clude NSE (Fall, 1999), IP-TNE (Bradford et al., 2000), eﬃciently within the simulator without exposing it to the
MaSSF (Liu et al., 2003), and Maya (Zhou et al., 2004). external routers. An experimental evaluation of this in-
NSE is an emulation extension of the popular ns-2 simula- frastructure, with microbenchmark tests and a case study,
tor (Breslau et al., 2000) augmented with the support for demonstrates the eﬀectiveness of our approach in mini-
connecting with real applications and scheduling real-time mizing overhead and consequently improving accuracy and
events. IP-TNE is the ﬁrst simulator we know that adopts scalability. The scalability studies show that our routing
parallel simulation techniques for large-scale network em- infrastructure can support for a large scale routing exper-
ulations. MaSSF is based on the DaSSF simulator (Liu iment with as many as 240 virtual routers running on a
and Nicol, 2008) with support for the grid computing en- single physical machine. We also report the memory is
vironment. Maya is an emulation extension of the com- the bottleneck for multiplexing a large number of virtual
mercial QualNet simulator (Scalable Network Technolo- routers by using virtualization technology.
gies, 2008) for wireless mobile networks. ROSENET (Gu Our current infrastructure for scalable routing exper-
and Fujimoto, 2007) is an emulation framework that sep- iments has limitations. For example, the synchroniza-
arates network simulation that provides high-ﬁdelity net- tion updates between the XORP protocol instances and
work resource estimates from low-ﬁdelity emulation traﬃc their corresponding router nodes within PRIME are only
forwarding. To some extent, this is similar in concept to one-directional in our current prototype implementation;
our approach of separating routing and forwarding func- updates are only propagated from the emulated XORP
tions. routers to the simulation engine. As immediate future
Other alternatives for conducting experimental network work, we will allow information exchange in both direc-
research are emulation and physical testbeds. Emu- tions so that we will be able to deal with more complex
Lab (White et al., 2002) is a notable emulation tool that requirements in protocol operation.
consists of a large number of networked computers inte-
grated and coordinated to present a virtual network envi-
ronment. EmuLab currently supports hundreds of projects ACKNOWLEDGMENT
with thousands of users running over 18,000 experiments
per year. PlanetLab (2008) is a collection of machines This research is supported in part by National Science
distributed across the Internet and used by researchers to Foundation grants CNS-0546712, CNS-0836408, and HRD-
develop new network services. These machines run a cus- 0317692. We would like to thank Eric Johnson at FIU SCIS
tomized operating system that allows the computing and for the helpful discussions. We would also like to thank the
networking resources to be shared among multiple experi- anonymous reviewers for their constructive comments.
mentations that may take place simultaneously. A critical
component of this type of physical testbeds is the live traf-
Our work bears resemblance with VINI (Bavier et al., REFERENCES
2006). VINI is a virtual network infrastructure that en-
ables realistic and controlled network experiment over Abilene Network (2008). Abilene Observatory Sum-
PlanetLab. By using virtualization technology, researchers mary Data View. http://abilene.internet2.edu/
are able to create a ﬂexible virtual network topology. The observatory/data-views.html.
implementation of VINI runs XORP instances within UML
Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris,
virtual machines and Click for packet forwarding. We drew
T., Ho, A., Neugebauer, R., Pratt, I., and Warﬁeld, A.
inspiration from the VINI design. Our scheme is diﬀerent,
(2003). Xen and the art of virtualization. In Proceed-
however, in that we focus on enabling real network simu-
ings of the 19th ACM Symposium on Operating Systems
lation to achieve realism, scalability, and ﬂexibility. Our
approach allows a large number of XORP instances beyond
what can be supported by PlanetLab. Basu, A. and Riecke, J. G. (2001). Stability issues
in OSPF routing. In Proceedings of the ACM SIG-
COMM Conference on Applications, Technologies, Ar-
6 CONCLUSIONS AND FUTURE WORK chitectures, and Protocols for Computer Communica-
A sound infrastructure allowing scalable routing experi-
ments is a key enabler for the design and development of Bauer, D., Yuksel, M., Carothers, C., and Kalyanaraman,
network routing protocols. We propose a novel infrastruc- S. (2006). A case study in understanding OSPF and
ture that combines a real-time network simulation engine BGP interactions using eﬃcient experiment design. In
Proceedings of the 20th Workshop on Principles of Ad- Liu, J. (2006). Packet-level integration of ﬂuid TCP models
vanced and Distributed Simulation (PADS’06). in real-time network simulation. In Proceedings of the
2006 Winter Simulation Conference (WSC’06), pages
Bavier, A., Feamster, N., Huang, M., Peterson, L., and 2162–2169.
Rexford, J. (2006). In VINI veritas: Realistic and
controlled network experimentation. In Proceedings Liu, J. (2008). A primer for real-time simulation of large-
of the ACM SIGCOMM Conference on Applications, scale networks. In Proceedings of the 41st Annual Sim-
Technologies, Architectures, and Protocols for Computer ulation Symposium (ANSS’08).
Liu, J., Mann, S., Vorst, N. V., and Hellman, K. (2007).
Bhatia, S., Motiwala, M., Muhlbauer, W., Valancius, An open and scalable emulation infrastructure for large-
V., Bavier, A., Feamster, N., Peterson, L., and Rex- scale real-time network simulations. In Proceedings of
ford, J. (2008). Performance evaluation of virtualization the 26th Annual IEEE Conference on Computer Com-
technologies for server consolidation. Technical Report munications, Joint Conference of the IEEE Computer
GT-CS-07-10, Georgia Tech Computer Science, Atlanta, and Communications Societies (INFOCOM’07) Min-
USA. iSymposium, pages 2476–2480.
Bradford, R., Simmonds, R., and Unger, B. (2000). A par- Liu, J. and Nicol, D. M. (2008). Dartmouth Scalable Simu-
allel discrete event IP network emulator. In Proceedings lation Framework (DaSSF). http://www.cis.fiu.edu/
of the 8th International Symposium on Modeling, Analy- ~liux/research/projects/dassf/index.html.
sis and Simulation of Computer and Telecommunication
Systems (MASCOTS’00), pages 315–322. Liu, X. and Chien, A. A. (2004). Realistic large-scale online
network simulation. In Proceedings of the SuperComput-
Breslau, L., Estrin, D., Fall, K., Floyd, S., Heidemann, J.,
ing Conference (SC’04).
Helmy, A., Huang, P., McCanne, S., Varadhan, K., Xu,
Y., and Yu, H. (2000). Advances in network simulation. Liu, X., Xia, H., and Chien, A. A. (2003). Network emu-
IEEE Computer, 33(5):59–67. lation tools for modeling grid behavior. In Proceedings
Chim, T. W. and Yeung, K. L. (2004). Time-eﬃcient al- of 3rd IEEE/ACM International Symposium on Cluster
gorithms for BGP route conﬁguration. In Proceedings of Computing and the Grid (CCGrid’03).
the IEEE International Conference on Communications
Mahajan, R., Wetherall, D., and Anderson, T. (2002).
Understanding BGP misconﬁguration. In Proceedings
Cowie, J., Nicol, D., and Ogielski, A. (1999). Modeling the of the ACM SIGCOMM Conference on Applications,
global Internet. Computing in Science and Engineering, Technologies, Architectures, and Protocols for Computer
1(1):42–50. Communications (SIGCOMM’02).
Dimitropoulos, X. A. and Riley, G. F. (2004). Large-scale Nicol, D. (2008). DARPA NMS baseline network topol-
simulation models of BGP. In Proceedings of IEEE In- ogy. http://www.ssfnet.org/Exchange/gallery/
ternational Symposium on Modeling, Analysis, and Sim- baseline/index.html.
ulation of Computer and Telecommunication Systems
(MASCOTS’04). Nordstrom, O. and Dovrolis, C. (2004). Beware of
BGP attacks. ACM Computer Communications Review,
Fall, K. (1999). Network emulation in the vint/ns sim- 34(2):1–8.
ulator. In Proceedings of the 4th IEEE Symposium on
Computers and Communications (ISCC’99), pages 244– Obradovic, D. (2002). Real-time model and convergence
250. time of BGP. In Proceedings of the 21st Annual Joint
Conference of the IEEE Computer and Communications
Goodell, G., Aiello, W., Griﬃn, T. G., Ioannidis, J., Mc- Societies (INFOCOM’02).
Daniel, P., and Rubin, A. (2003). Working around BGP:
An incremental approach to improving security and ac- OpenVZ (2008). OpenVZ. http://openvz.org/.
curacy of interdomain routing. In Proceedings of the
10th Annual Network and Distributed System Security Padala, P., Zhu, X., Wang, Z., Singhal, S., and Shin, K. G.
Symposium (NDSS’03). (2007). Performance evaluation of virtualization tech-
nologies for server consolidation. Technical Report HPL-
Gu, Y. and Fujimoto, R. (2007). Applying parallel and 2007-59, HP Laboratories.
distributed simulation to remote network emulation. In
Proceedings fo the 2007 Winter Simulation Conference. PlanetLab (2008). Planetlab: An open platorm for devel-
oping, deploying, and accessing planetary-scale services.
Kohler, E., Morris, R., Chen, B., Jannotti, J., and http://www.planet-lab.org/.
Kaashoek, M. F. (2000). The click modular router. ACM
Transactions on Compauter Systems, 18(8):263–297. PRIME (2008). http://www.cis.fiu.edu/prime/.
Quagga (2008). Quagga Routing Suite. http://www.
Scalable Network Technologies (2008). http:
Shaikh, A., Dube, R., and Varma, A. (2002). Avoid-
ing instability during graceful shutdown of OSPF. In
Proceedings of the 21st Annual Joint Conference of the
IEEE Computer and Communications Societies (INFO-
Shaikh, A. and Greenberg, A. (2001). Experience in black-
box OSPF measurement. In Proceedings of the SIG-
COMM Internet Measurement Workshop.
Shaikh, A., Kalampoukas, L., Dube, R., and Varma, A.
(2000). Routing stability in congested networks: Ex-
perimentation and analysis. In Proceedings of the ACM
SIGCOMM Conference on Applications, Technologies,
Architectures, and Protocols for Computer Communica-
Soltesz, S., Potzl, H., Fiuczynski, M. E., Bavier, A., and
Peterson, L. (2007). Container-based operating sys-
tem virtualization: A scalable, high-performance alter-
native to hypervisors. In Proceedings of the 2nd ACM
SIGOPS/EuroSys European Conference on Computer
VRF (2008). Linux Virtual Routing and Forward-
ing (VRF). http://sourceforge.net/projects/
White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad,
S., Newbold, M., Hibler, M., Barb, C., and Joglekar,
A. (2002). An integrated experimental environment for
distributed systems and networks. In Proceedings of the
5th Symposium on Operating Systems Design and Im-
plementation (OSDI’02), pages 255–270.
XORP (2008). XORP. http://www.xorp.org/.
Zebra (2008). GNU Zebra. http://www.zebra.org/.
Zhou, J., Ji, Z., Takai, M., and Bagrodia, R. (2004).
Maya: integrating hybrid network modeling to the phys-
ical world. ACM Transactions on Modeling and Com-
puter Simulation, 14(2):149–169.