A Comparative Study of Extensible Routers by warrent


									                          A Comparative Study of Extensible Routers

                                    Yitzchak Gottlieb and Larry Peterson

                                              zuki,llp @cs.princeton.edu

                                       Department of Computer Science
                                            Princeton University

                      Abstract                              customization of router functionality. These architec-
                                                            tures accommodate changes to both the router’s con-
Motivated by the demand for routers with new ca-            trol plane, as required by programmable networks [3,
pabilities, researchers have been building extensible       10], and the router’s data plane, as allowed by active
routers that aid in the design and development of net-      networks [1, 19].
work protocols and services. This paper evaluates and          When considering these systems, it is evident that
compares three such systems: (1) Princeton’s Scout-         the designers addressed a common set of issues: what
based Extensible Router, (2) MIT’s Click router, and        functions should routers execute on packets, how to
(3) Washington University’s Router Plugins. To pro-         specify the function for a particular packet flow, how
vide a framework in which these three systems can           to identify a packet as part of a flow, and so on. Dif-
be studied, the paper also presents a simple model          ferent architectures address these issues in different
of an extensible router based on four primitive ob-         ways. A model that provides a framework for rea-
jects: queues, classifiers, forwarders, and schedulers.      soning about the functioning of routers would be use-
By composing these primitive objects it is possible         ful both in evaluating currently available extensible
to model everything from a standard, best-effort IP         routers and in designing future systems.
router to an application-level proxy. The paper also           The rest of the paper describes one such model
briefly discusses the role that extensible routers play      and uses it to evaluate three software-based extensible
in the construction of active, programmable, and over-      routers. The paper is organized as follows. Section 2
lay networks.                                               motivates our model from first principles, and Sec-
                                                            tion 3 uses the model to describe three software-based
1 Introduction                                              extensible router architectures: Scout [13], Click [9],
                                                            and Router Plugins [4]. Section 4 then compares these
Several novel network services have been proposed in        three systems in terms of the model. Finally, Sec-
the past few years. Examples include packet tagging,        tion 5 briefly discusses the relationship between ex-
application-level proxies, application-specific packet       tensible routers and research in active, programmable,
dropping, performance monitoring, intrusion detec-          and overlay networks.
tion, and assorted filters and firewalls. Software based
routers [12] provide an excellent platform for devel-       2 Model
oping and deploying these new services. Although
software is easily changed, reprogramming a router to       This section describes a simple model for extensible
support a new service is still a challenging endeavor.      routers based on intuitive assumptions of how routers
It is also important that the software have an architec-    in general, and extensible routers in particular, ought
ture that governs how the router may be extended.           to work. We do not claim that the model is the only
    Motivated by earlier work on extensible operating       one possible, only that it is useful for understanding
systems [2, 6, 13], researchers have recently defined        the differences and similarities of the three systems
extensible router architectures that support run-time       we elect to study. Also, there are many details that we
could add to the model to make it more precise. How-       teed to a flow. In a router that allows certain packets
ever, we include only those details needed to under-       to receive a different level of service, packets are not
stand the important similarities and differences among     necessarily sent in the same order in which they ar-
the three example systems.                                 rive. To allow modeling more complex types of queu-
                                                           ing behavior, we add a second component that allows
2.1 Components                                             us to abstract a router’s queuing discipline. We can
                                                           then reason about it as a separate component of the
In its simplest form, a router forwards packets from       system. In our model, a scheduler, depicted in Fig-
one of its input ports to one of its output ports. This    ure 2, choses a packet from one of its input queues
process involves reading the packet from one port, ex-     and places it on its single output queue.
amining it to decide whether and whither to forward
it, and writing the packet out to an output port. Be-
cause packets from multiple inputs may contend for a
common output port, packets are queued on the output
port. Because it is possible that packets arrive on the
input link asynchronously with respect to the router,                                   S
we also model the input port as a queue. For our pur-
poses, we assume both queues are serviced in first-in,
first-out (FIFO) order.
    In this simple scenario, the router’s primary task
is to decide the packet’s outgoing port based on its
                                                                       Figure 2: A packet scheduler
destination address—a process typically called packet
classification. To reflect this behavior, the first com-
                                                              Simply connecting a set of classifiers (one per in-
ponent of our model is a classifier, which dequeues a
                                                           put port) to a set of schedulers (one per output port)
packet from a single input queue and places it on zero,
                                                           is sufficient to model a bridge or an Ethernet switch—
one, or multiple of the available output queues. The
                                                           network elements that do no processing on the packet
classifier places the packet on no queues if it elects
                                                           beyond classification. A router is distinguished in that
not to forward the packet (e.g., to implement a filter),
                                                           it modifies the packet in some well-defined way; we
on one in the common case, and on more than one to
                                                           say it applies a forwarding function to the packet. In
implement multicast. Figure 1 shows a simple config-
                                                           general, a router may support more than one forward-
uration of a router with a classifier associated with a
                                                           ing function. For example, commercial routers tend to
single input port. In general, a classifier is bound to
                                                           have at least two functions: one that handles option-
each input port.
                                                           free packets and one that processes IP options. Exten-
                                                           sible routers extend this practice by allowing arbitrary
                                                           processing. Thus any router model, especially one for
                                                           extensible routers, should include an element encap-
                                               Output      sulating forwarding functions.
  Input                                         Port
                         C                                    The last component in our model, a forwarder, re-
   Port                                                    moves at most one packet from its input queue and
                                                           places one packet on its single output queue. Al-
                                               Output      though the expectation is that most of the data will
                                                Port       be similar, the packet placed on the output need not
                                                           be identical to that placed on the input. Between read-
       Figure 1: A simple, best effort IP router           ing and writing the packet, a forwarder may engage in
                                                           any amount of arbitrary computation.
   While it is possible to associate a single queue with      Note that some forwarders implement control plane
each output port, such a configuration is not general       functionality, while others are part of the data plane.
enough to model complicated queuing strategies de-         Control forwarders typically manage the other com-
signed for preserving the quality of service guaran-       ponents of the system; they set scheduling parameters,
program classifiers to recognize new packet flows,             constructed by linking the output one classifier to the
and get or set state variables associated with data for-     input of another.
warders. The interface through which control for-
warders interact with the rest of the router is beyond
the scope of this paper.
   Forwarders implement a wide range of functional-          2.2 Extensibility
ity. For example, the standard IP forwarder decre-
ments the TTL field, recomputes the checksum, and
                                                             This model helps us understand that to extend a router,
changes the link-layer header. However, routers are
                                                             one has to change one of the four primitive compo-
not limited to this base case. Applications such as
                                                             nents. However, it does not prescribe exactly how
various proxies, peer-to-peer and overlay networks,
                                                             this “change” is accomplished. As we will see in the
packet taggers, and denial-of-service detectors result
                                                             following sections, different systems provide different
in far more complex forwarders. For example, they
                                                             mechanisms to manage the base components.
may modify the IP and TCP headers to reroute the
packet to a different host, or they may rewrite the con-        For the purpose of this paper, a router may be ex-
tents of the packet to request nearby, cached versions       tended in three different ways: it is possible to insert
of data. In general, several processing steps may be         an additional component, modify an existing compo-
composed to produce different behaviors. For exam-           nent, or replace an existing component. While any
ple, a proxy forwarder that rewrites TCP headers may         component can be extended in any way, some ex-
then pass the new packet to an IP forwarder.                 tensions are more common for particular components
   A rich combination of forwarders implies that clas-       than others. For example, our model provides ex-
sification may also need to be complex. For example,          plicit support for inserting additional forwarders in the
a best-effort IP router connected via Ethernet uses the      router. In contrast, typically classifiers and schedulers
Ethernet and IP type fields to determine the correct          are either replaced with an alternative or modified to
processing for the packet and prefix matching of the          support some new behavior. Modifying a component,
IP destination address to decide on the proper routing.      in turn, involves either giving it a new set of parame-
For many applications, classification can be based on         ters or extending its functionality with new code frag-
both TCP ports and IP addresses. Other classifiers            ments. The latter method implies that the component
may require reading data deep in the packet. For in-         has some externally visible structure that can be ex-
stance, a router being used as a front end to a cluster of   ploited, while the former consists of programming the
load-balanced servers can read the HTTP GET com-             component (e.g., with a new routing table, level-four
mand from a packet and redirect it to an idle server.        demultiplexing key, or QoS reservation) and therefore
Thus, in general, the classifier must choose not just         requires a control interface.
an output port, but also the appropriate sequence of            As an example, application-specific droppers, like
forwarders.                                                  those proposed to thin video streams in the face of
   Figure 3 shows a complete model of an extensi-            congestion, extend the classifier by providing it with a
ble router. As mentioned above, this model is under-         new function which is applied to each packet. On the
specified; e.g., we do not specify how deep queues            other hand, application-level proxies and peer-to-peer
can be, whether other components can query its prop-         networks add a whole new forwarder into the router;
erties and contents, or how to model input and output        these applications also modify the classifier by pro-
devices. These and other similar details are not re-         gramming it to divert packets matching a particular
quired to compare the extensible router systems we           level-four key over to this forwarder. Finally, conges-
describe later, so we leave them unstated. Also, this        tion control mechanisms (e.g., RED [7]) and queuing
figure shows only one possible combination of classi-         disciplines used for QoS (e.g., Weighted Fair Queu-
fiers, forwarders, and schedulers; other combinations         ing [5] and Deficit Round Robin [16]) replace the sys-
are possible. For example, a forwarder may mod-              tem’s default scheduler. Each of these new compo-
ify packets then “loop back” and re-insert them for          nents may allow modification. For instance, a RED
classification. Also, sequences of forwarders can be          implementation could offer a control interface for set-
linked together and a hierarchy of classifiers can be         ting its thresholds.

     Input                                                                                                   Output
                      C                                     F                                  S
      Port                                                                                                    Port


                                      Figure 3: Modeling an extensible router

2.3 Engineering Trade-offs                                      dropped in the middle of a forwarder, but since pack-
                                                                ets that arrive with a TTL of 1 will be dropped by the
The distinction between classifiers and forwarders is            router, the classifier could decide to drop the packet
a matter of degree: classifiers may process packets af-          by selecting no output queue for it. Another, similar
ter they have determined in which queue to put them,            choice involves multicast. Since forwarders can cre-
and forwarders may make branching decisions. When               ate packets, it is possible to model multicast behavior
choosing how much processing to put into classifiers             as a set of two components: a forwarder which cre-
and how much decision making to put into forwarders,            ates copies of a multicast packet and a classifier would
there are two possible extremes. One extreme is to de-          then choose a single output for each copy. In order to
clare that all branches are classification and must be           show that unicast is merely a degenerate form of mul-
chosen before any processing is done on the packet;             ticast, we chose instead to have the classifier create
the other extreme is to decide that only the classifi-           the copies.
cation decision that chooses an output port is defined
as classification and forwarders may branch arbitrar-
                                                                3 Systems
ily. Choosing simpler forwarders leads to many clas-
sifying components in a router, choosing complicated
                                                                This section uses the model to describe three ex-
ones restricts a router to just one classifier. Similarly,
                                                                tensible routers. The three systems: Scout, Click,
if we permit classifiers to modify packets, we find that
                                                                and Router Plugins, were developed independently,
some computations are simple enough to be included
                                                                and each approaches extensibility in a different way.
in a classifier.
                                                                Scout, developed at University of Arizona and later
   The system designer must choose the component                Princeton University, was designed as a stand-alone
in which to place a specific block of code. On real              operating systems for network appliances but is also
systems, in which compromises must be made for ef-              available as a Linux kernel module. Click, developed
ficient operation, a middle ground is usually chosen.            at MIT, uses a kernel module to replace the Linux
One possible criterion is maximum computation time:             kernel’s networking subsystem. The Router Plugins
classifiers and schedulers must complete in a short,                                               u
                                                                architecture, developed by ETH Z¨ rich, Washington
bounded time in order to send or receive packets at             University and Ascom under the Crossbow project,
line speed. Another criterion might be extensibility:           extends the networking subsystem of a BSD kernel
if it is difficult to replace or modify the classifier at         to provide hooks for new forwarding functions.
run time, then the function is best encapsulated in a
                                                                3.1 Scout
   For the sake of this paper, we bar classifiers and
schedulers from modifying packets. We also re-                  Scout is a modular, communication-oriented operat-
quire any branches that change the type of process-             ing system. Its major abstraction is the path: a linear
ing that is done to a packet, as opposed to those that          flow of data that starts at a source device and ends at
change the parameters used in the processing, go in             a destination device. Paths are composed of stages,
the classifier—not the forwarder. One good illustra-             which are instances of modules. Each Scout module
tion of the choice is decrementing IP’s time-to-live            implements a well understood protocol, such as IP or
field. IP routers must discard packets whose TTL                 TCP. A module may contribute stages to many paths,
decrements to 0. The packet could be processed and              but a stage is unique to a single path. The system cre-
                                                       Stage        Stage          Stage

  Input                                                                                                             Output
                                                          Stage                Stage                 S
   Port                                                                                                              Port

                                                       Stage   Stage       Stage   Stage

                                            Figure 4: Modeling Scout

ates paths on demand at run time. Scout’s infrastruc-              ways.
ture provides tools to create, modify, schedule, and                  First, Scout originally had no scheduler; it’s output
control paths.                                                     queues were strictly FIFO. In order to schedule the
   In addition to providing stages to paths, each mod-             output one scheduled the paths. This method proved
ule also implements a module-specific, programmer-                  inefficient since we could not effectively schedule a
defined, classification function. As part of path cre-               given packet’s execution to match its scheduled trans-
ation, each module places a reference to the path in               mission slot. We therefore added an explicit packet
a private data structure that it can access during clas-           scheduler to the output module. This allowed us to
sification. If the module cannot uniquely identify the              transmit packets whose processing was already com-
path, the module stores a reference to another mod-                plete immediately when the bandwidth was available.
ule’s classification function that might be able to dis-               Second, to make packet classification faster, we en-
ambiguate an incoming packet. Scout’s hierarchical,                hanced Scout’s classifier with the ability to cache re-
extensible packet classifier queries each module for a              sults from previous lookups at lower levels in the clas-
partial decision based on the packet, then recurs to the           sification hierarchy. For instance, Scout now keeps a
next module. The recursion continues until a path is               cache of the most recently seen IP addresses to make
selected or it is determined that the packet belongs to            forwarding lookups on those addresses faster. To clas-
no path.                                                           sify a packet Scout first does simple lookups in its
   Once a path has been selected, the packet is placed             caches, and only if it fails to find a match does the
on that path’s input queue, and the path is scheduled to           full classification described earlier take place.
run. When Scout’s thread scheduler decides that the                   Third, when we needed to extend Scout to the In-
path’s thread is ready to execute, the thread dequeues             tel IXP1200 [8] network processor, we decided not
the packet and hands it to the first stage in the path,             to port Scout directly, but instead designed a minimal
which processes it and invokes the next stage. The                 version for this new environment. The architecture,
stages process the packet in sequence, with the last               called the Virtual Router Processor (VRP) [17], al-
stage placing the packet on an output queue. Eventu-               lows the user to specify a function to be applied to a
ally, the scheduler selects the packet from this queue             specific packet flow on the line card. In effect, this al-
and transmits it.                                                  lows a Scout path to run on one of a hierarchy of pro-
                                                                   cessors. The VRP also extends Scout’s classification
   As shown in Figure 4, Scout has one classifier with
                                                                   hierarchy to an even lower level—the network card.
an internal hierarchical structure per input port, one
                                                                   Any packets that should be processed at the line card
scheduler per output port, and paths implementing
                                                                   will match in the card’s classifier. Any packet that re-
forwarders. Alternatively, we could model each stage
                                                                   quires greater processing, or fails to match at the line
as a separate forwarder, but equating paths and for-
                                                                   card, is diverted to Scout’s full classifier running on a
warders captures the important properties of Scout.
                                                                   more powerful processor.
   Given our experience with the Scout operating sys-
tem, it is not surprising that it easily reflects our
                                                                   3.2 Click
model. However, this is true because we continued
to evolve Scout as our notions of a useful design for              Like Scout, Click is a modular architecture for build-
an extensible router matured. This happened in three               ing routers. Its design is based on composing many
simple elements to produce a system that implements         For example, elements with push-type interfaces have
the desired behavior. Each element may have mul-            the input chosen for them, so their scheduler is effec-
tiple ports to connect it to other elements. Both in-       tively null, while elements with pull-type interfaces
put and output ports may be either push-type (through       have their output selected for them, so their classifier
which a packet must be sent) or pull-type (from which       is effectively null.
a packet may be requested). Pull-type and push-type
input ports can connect only to pull-type and push-
                                                            3.3 Router Plugins
type output ports, respectively. Packets may be stored
only in queues, an element that has a push-type input       The Router Plugins architecture is designed to allow
port and a pull-type output port.                           limited extensions to an IP router. Implemented in
   When a packet arrives at the router, the device          the NetBSD operating system, Router Plugins allows
driver pushes it through a series of elements. Since        users to write extensions that can be placed at well
elements may have more than one output port, each           known points of the router’s IP execution. These
element chooses the proper outgoing port as part of         points, called gates, were chosen to suit a wide va-
the packet processing. The packet is pushed through         riety of applications, such as routing, packet schedul-
elements until it hits a queue. Once enqueued, the          ing, and security processing. The user invokes com-
packet waits until it is pulled from its queue. When        mand line tools to load plugins at run time.
an output device is ready for transmission, the driver         Router Plugins may be associated with distinct
pulls a packet from one of its upstream neighbors. The      flows. Since not every plugin is appropriate for each
neighbor will pull the data from one of its upstream        packet, Crossbow1 allows the user to specify a fixed
neighbors until some element queries a queue. Each          key with each plugin. The key specifies IP address
individual element decides which of its input ports to      pairs, protocol number, ports, and incoming interface;
query for a packet. The packet is then removed from         any part of the key can be a wildcard meaning that
the queue and will traverse the same sequence of mod-       any value matches. At each gate, Crossbow attempts
ules that selected it before reaching the device driver     to match the packet to the keys registered for that gate.
and being transmitted.                                      If a key matches, the packet is passed to the plugin in-
                                                            stance that registered that key. When the packet leaves
          Input         Element
                                         Output             the plugin, the next part of the IP processing path ex-
                                                            ecutes until the next gate. This sequence of events
                                                            repeats until there are no more gates and the IP pro-
                    S       F        C                      cessing terminates.
                                                               As shown in Figure 6, Crossbow can also be mod-
                                                            eled as a sequence of our basic components. A plugin
                                                            operates on a single packet then emits it into the pro-
         Figure 5: Modeling a Click element                 cessing path, and is, therefore, a forwarder. Crossbow
                                                            must choose a plugin at each gate, so a gate is a clas-
   In terms of our model, each Click element encapsu-       sifier. Upon leaving the gate, the packet must tran-
lates one scheduler, one forwarder, and one classifier,      sit the next part of the IP processing code, which is
as illustrated in Figure 5. The scheduler is necessary      just another forwarder. Since all packets must execute
because an element has to select a packet from poten-       the same IP code after choosing only one forwarder,
tially many input ports. Similarly, the classifier is nec-   our model dictates that a scheduler must be present.
essary because the element may have multiple output         Since Crossbow makes no scheduling decision at this
ports. The forwarder simply connects the scheduler to       time, we place a FIFO scheduler before the next frag-
the classifier.                                              ment of IP code. For each defined gate, Crossbow is
   By leaving any two of these three components             a classifier, a forwarder (plugin or null forwarder) and
empty, a Click element can implement each of the            a scheduler (FIFO). The scheduler’s output queue is
primitive components of our model, making it a very            1 For the rest of the paper,we will use the project name, Cross-
general architecture. It is more common, however, for       bow, to refer to the infrastructure provided by the Router Plugins
a given element to define two of the three components.       architecture.
                             F                                                     F

                           Plug−ins                                              Plug−ins
             Gate                                    IP Code        Gate                                   IP Code
                 C           F             S               F           C           F            S             F

                             F                                                     F

                                      Figure 6: Modeling Crossbow’s gates

connected to the input queue of the the next classifier         compare how much flexibility each system gives us
by the next piece of NetBSD’s IP infrastructure.               within its fixed architecture.
   While this decomposition of Crossbow is accurate,
in some instances there is a better one. Assuming              Classifiers
that the packet’s IP and TCP headers are not modi-
fied by any plugin, the packet need be classified only           Many useful classifiers are possible. IP routing is
once. The entire sequence of plugins executing on this         based on a longest prefix match of the destination ad-
packet is then known before the first forwarder exe-            dress. Transport-level packet flows requires match-
cutes. We can then define the forwarding function ex-           ing IP addresses and TCP/UDP port pairs. Other ap-
ecuting on this packet as the concatenation of all that        plications may inspect the message content to differ-
packet’s plugins and IP infrastructure code. We no             entiate among packets. The classifier fundamentally
longer need the FIFO schedulers between gates, and             limits the granularity at which flows can be distin-
the only remaining scheduler is the packet scheduler           guished, and consequently, determines the set of pack-
at the output port. Crossbow then looks like Scout, as         ets on which a given forwarder operates. Crossbow
depicted in Figure 4, except plugins replace stages.           and Scout both prescribe a fixed classifier structure
                                                               which may be modified, while Click allows new clas-
4 Comparison                                                   sifiers to be inserted in arbitrary places.
                                                                  Crossbow’s classifier can be modified only by giv-
The previous section described three extensible router         ing it new keys. It identifies flows by IP address, TCP
operating systems and showed how we can model                  port pair, and incoming network interface, and it uses
each of them as a collection of classifiers, forwarders,        only these fields to find the correct forwarder (plugin)
and schedulers. This section compares the systems in           at each gate. The classification algorithm allows fast
areas not restricted by the model. Primarily we will           lookups even in the presence of full or partial wild-
focus on three categories of differences:                      card values in any field. The function itself, though,
                                                               is fixed. While Crossbow does not permit replacing
Extensibility: Changing a system’s components.                 the core IP classification function, the system will, in
Scheduling: Scheduling packet processing.                      the future, provide a gate for replacing the IP prefix
Flow Isolation: Segregating flow processing.                    matching algorithm. It may also allow a plugin to
                                                               modify the routing table lookup, but these gates will
                                                               be restricted to operating on flows identified by IP ad-
4.1 Extensibility
                                                               dress and port numbers.
We first consider how each of the three systems can                Scout’s single hierarchical classifier can be mod-
be extended by either inserting, replacing, or modify-         ified by programming new modules and giving new
ing its components. It is important to note that nothing       keys to existing modules. During path creation,
prevents us from extending these systems by rewriting          Scout’s classification function is implicitly extended
any code fragment we choose. However, doing so ob-             by adding a call to the demultiplexing function of each
viates the reasons for choosing the structure imposed          module that contributed a stage to the path. Dele-
by that particular architecture. Our goal is to under-         gating classification to the modules allows flows to
stand the structure of the extension mechanism, so we          be readily identified on the basis of any criteria the
module programmer chooses. Specific TCP flows are            provide no such facility, and the programmer who im-
identified by their port numbers; IP flows are iden-         plements a connection-oriented protocol would have
tified by destination address; Ethernet frames by their     to create such a module from scratch. Scout eases the
type number, and so on. Changing a specific module’s        programmer’s burden by providing explicit interfaces
classification function is easier than altering Cross-      specific to a given task, thereby eliminating tedious
bow’s since the modifications are confined to a single       semantic checks needed if such calls must be encoded
module, as opposed to affecting the entire system.         in a simple data push/pull interface.
   Unlike the other two systems, Click has no global          To properly interact with the structure provided
classification structure. Each element acts as a clas-      by the underlying architecture, modules may also
sifier to one of its outputs during packet processing,      implement methods demanded by that architecture.
so there is no extra classification step. However, in-      Router Plugins and Scout modules are both compli-
serting a new element into the component graph cre-        cated because they exist within a complex infrastruc-
ates new possibilities for a packet’s route through the    ture. Router Plugins are BSD loadable kernel mod-
router and explicitly extends the classification avail-     ules; they must register themselves with the Plugin
able. Like Scout, reprogramming a specific element’s        Control Unit when they are loaded, and they must re-
“routing” decision is fairly easy, but one must be care-   spond to a small set of messages. Similarly, Scout
ful that the module still implements the function for      modules must implement an initialization function
which it was designed.                                     and a demultiplexing function, in addition to their data
                                                           forwarding function. In effect, both Router Plugins
                                                           and Scout modules are factories, creating dynamic,
                                                           run-time instances which are actually used in com-
Scout, Click, and Crossbow were all designed to allow      putation. The interface for creating and destroying
easy insertion of forwarders. In this case, the more in-   stages (or plugin instances) adds to the complexity of
teresting question is how difficult it is to program an     the modules (plugins).
entirely new service as one or more forwarders. Al-           Click elements are much simpler. Click imposes
though this discussion of the relative complexity of       very little structure on an element, so initialization is
programming a system is not very rigorous, it is still a   simple. In order to create a working system, how-
useful comparison point.                                   ever, the Click infrastructure provides a mechanism to
   Perhaps the most convenient measure of the com-         tie the elements together at configuration time. Since
plexity of creating a new module is the number of          this mechanism runs at configuration time, Click must
methods a module must support. Modules support two         reinitialize the entire configuration each time a new el-
classes of methods: those invoked by other modules         ement is added, even if the code for the element is
in the system, and those invoked by the architecture in    already in the kernel. Neither Scout nor Crossbow
which the module is embedded. To forward data be-          stop processing while installing forwarders; modules
tween peers, modules provide calls to transfer packet      include the code needed to incrementally insert them-
data. A system designer must balance the elegance          selves into a running system. In Scout, path creation
of a simpler interface with the convenience of a full-     affects only those modules that contribute a stage to
featured interface.                                        the path. Similarly, when a user registers a Crossbow
   Each Router Plugin supports a single method to be       plugin instance with a certain flow, no other plugin is
called when packets arrive for it; Click enhances this     affected.
simple push interface by adding a pull method. Scout          While measuring number of methods is convenient,
does not fix the module interface. Instead, it sup-         it is not wholly accurate. Simple functions, even if
ports multiple interface types; e.g., a file system in-     there are many of them, are easier to write than a
terface, a framebuffer interface, and a protocol inter-    smaller number of more complex ones. To quantify
face. Even its protocol interface is more general and      the amount of work needed to implement minimal
provides methods for many useful operations. For ex-       useful functionality, we also looked at how many lines
ample, Scout provides an method that a connection-         of code need to be written to create a module in each
oriented protocol like TCP can use to determine when       system. For the sake of uniformity, we will use the
a peer is attempting to connect. Click and Crossbow        “null” forwarder which does nothing but forward the
packet unchanged.                                                  packets efficiently.
   It makes sense that smaller, less restrictive archi-                Being designed around NetBSD’s IP core and net-
tectures imply less code. The limited set of required              work subsystem, Crossbow’s thread scheduling is lim-
functionality allows Click elements to be very small;              ited by BSD. On packet arrival, packets are classified
the null element is just 8 lines of C++ code. The fact             and processed through the gates and sent to the output
that elements can be this small encourages program-                ports. This process happens with no context switches,
mers to write small, easily comprehensible modules                 no data copying, and no additional interrupt overhead.
that are then composed into a complex system. By                   These are all laudable performance goals, but this de-
comparison, the null Router Plugin is 96 lines long,               sign may experience a type of head-of-line blocking.
and Scout’s null module is 200 lines long. 2 Again, the            Consider the situation where two packets arrive, the
main reason for having to write more code to imple-                first packet belongs to a flow that requires process-
ment a null module in Scout and Crossbow is that the               ing at each available gate, while the second matches
module has to manage its own insertion into the larger             no filters at any gate. The second packet can be for-
system.                                                            warded very quickly with minimal overhead, yet it
   While allowing smaller modules eases extension,                 is delayed behind the first packet’s processing. This
there are compromises made to achieve this benefit.                 is true even if the packet scheduler later decides that
One trade off is that there are many more modules                  the second packet is more important and should be
to understand in a full system. For example, imple-                transmitted before the first. As detailed in Section 3,
menting a minimal IP router requires 38 elements in                our model suggests that Crossbow can have a packet
Click. Scout implements the same router with eight                 scheduler between gates, not just at the very end of
modules; Crossbow uses just one. Thus, the effort                  processing. It is possible that such an addition would
in programming a router from scratch is probably not               enhance Crossbow’s ability to provide different qual-
less in Click than in other systems.                               ities of service to different flows.
                                                                       Click also has the notion that operations on a packet
Schedulers                                                         should not be interrupted until its processing is com-
                                                                   plete. In particular, a packet is pushed or pulled from
The final extensible component in our model is the                  one queue to another. This observation might lead to
packet scheduler. Scout has a single scheduler per                 the conclusion that Click, like Crossbow, must sched-
output port. This scheduler can be replaced, but it                ule execution based solely on packet arrival times, but
is not possible to insert multiple schedulers. In many             this is not the case. Click’s thread scheduler handles
configurations, Click has a single packet scheduler per             elements. In particular, the device drivers that poll the
port; this is a separate component that rests between              input and output ports place themselves on the sys-
the output port and a queue. Just as with classifiers,              tem’s scheduling queue. When a driver discovers a
however, Click can have multiple components acting                 packet, the input process pushes it through the ele-
as a scheduler, and therefore allows users to both in-             ments until the packet is placed on some queue. Sim-
sert and replace schedulers, as well as modify them                ilarly, the output process pulls a packet from the out-
with different parameters. Crossbow provides a spe-                put device’s upstream neighbor, and from it’s neigh-
cial gate for packet scheduling decisions, allowing the            bor until a queue is reached. Thus, the placement of
user to insert a new packet scheduler for a given flow              queues in the element graph determines when process-
and modify previously registered instances.                        ing is scheduled. By placing queues early in the graph,
                                                                   it is possible to force packet processing to match avail-
4.2 Process Scheduling                                             able output, thereby avoiding head-of-line blocking.
                                                                   Unfortunately, since a packet may come from any
One aspect of a router that our model does not restrict
                                                                   reachable queue in the graph, it becomes nearly im-
is the scheduling of packet processing. The choice of
                                                                   possible to predict how long the execution of a sin-
how to schedule these computations can have a no-
                                                                   gle packet might take. A scheduler may decide that a
ticeable impact on the ability of the router to forward
                                                                   packet ought to come from one of its neighbors only
    2 The null VRP forwarder is completely empty since it has no   to discover that the function operating on that packet
interface to implement.                                            takes too long to complete.
   Instead of scheduling individual modules, Scout         ble to store flow-specific state without needing a new
schedules whole paths for execution. Since a path          lookup at each element, since the packet is the only
for a packet is chosen at packet arrival, Scout can        item which is passed along.
predict the processing time that will be spent on a           For a given Click configuration, however, it is pos-
packet. This knowledge allows high priority packets        sible to create another configuration that can distin-
to be scheduled first for execution, not just transmis-     guish flows. To do this, the classification decision
sion. Also, packets may be dropped if the scheduler        from all of the elements must be separated out and
predicts that executing the path will degrade forward-     placed at the beginning of the element graph. Each
ing performance of other packets to unacceptable lev-      possible outcome would feed a single queue. Each
els.                                                       of these queues would feed an element graph with no
                                                           branches, all of which feed into a scheduler at the out-
                                                           put port. Thus, by manually enumerating each possi-
4.3 Flow Isolation
                                                           ble path through the system, flows could be isolated. It
The ability of a system to explicitly recognize and ma-    would probably be unwise to attempt to do this, how-
nipulate the processing of packet flows is important to     ever. It would make the element graph more com-
its ability to manage its resources. As noted in Sec-      plex, hence more difficult to read, thereby negating
tion 4.2, a router can provide service guarantees if it    Click’s primary advantage. Also, in the current design
can schedule the processing of flows individually [15].     of Click, this effort would most likely result in code
Processing sequences specialized for a known pur-          bloat since elements would have to be repeated sev-
pose can also be simpler and therefore execute more        eral times in the element graph. Scout solves the con-
quickly [11]. A system that isolates the processing of     figuration problem by making each module a factory
particular flows can also more effectively monitor re-      of stages for paths, thereby avoiding multiple declara-
sources used by the flow and therefore better detect        tions in many cases.
denial-of-service attacks [18].                               Packet flows map very naturally onto Scout’s path
   Neither Crossbow nor Click has explicit support for     abstraction. Paths are Scout’s unit of scheduling as
isolating flows. In both cases, the design of the archi-    well as its unit of resource management. Paths allow
tecture seems to preclude it. Crossbow’s monolithic        Scout to isolate the processing of one flow of packets
infrastructure indicates the existence of a single ex-     from the processing of any other flow. Memory and
ecution path for processing all flows. As described         CPU use are accounted to individual paths allowing
earlier, however, a single lookup can be sufficient to      Scout to monitor at a fine granularity the resources
determine which sequence of plugins will operate on        consumed by a packet flow. Given our long experi-
a packet. Knowing this sequence would allow Cross-         ence with Scout, it is once again not unreasonable that
bow to predict the execution time of the packet and        we would endorse this particular abstraction.
therefore schedule the packet’s processing more effec-        Scout’s paths also give individual modules access
tively. Unfortunately, this is not possible in Crossbow.   to the current state of the flow-specific module that
This is because even though a plugin may be able to        has acted, or will shortly act, on a given packet. The
recognize different classes of packets that should be      module can just query the path for its state or specific
treated differently by the system, since it is activated   variables. Given the lack of path support in Click and
only after the initial classification, that knowledge is    Crossbow, the mechanism for determining this state
useless to the system as a whole.                          must be different. Click provides a notion of a flow-
   Click, by its very nature, precludes isolating indi-    based router context that provides the ability to query
vidual flows through the system. Click makes each el-       elements up-stream and down-stream of the current
ement with more than one output responsible for clas-      element. Using flow-based router context, it is possi-
sification. Since the decision can only be made once        ble to identify the set of possible elements that could
an element has seen the packet, the set of elements        act on the packet, but not the sequence that will act,
operating on a packet cannot be known until the ac-        or in fact has acted, on it. Crossbow mandates that
tual processing has already completed. This makes it       each plugin must support a simple communication in-
impossible to treat individual paths through the sys-      terface between plugins, so it is possible to get infor-
tem as separate entities. Furthermore, it is impossi-      mation about particular packets if any is stored. It is
not possible, however, for a plugin to determine which     only allow application programs to control conven-
other plugins were activated on the same packet, so it     tional router behavior; they do not support adding or
is still not known with whom to communicate.               controlling new forwarders.
                                                              Overlay networks are becoming an increasingly
5 Applications                                             popular way to introduce new functionality into the
                                                           network. Within an overlay, each router’s view of the
Active networks are networks that allow users to           network is restricted to include only those nodes that
specify the forwarding functions applied to pack-          are also participating in the overlay. These nodes can
ets, thereby making the network configurable on an          either be conventional routers or end hosts. In the lat-
application-specific basis. Active network imple-           ter case, the resulting end-system nodes forward pack-
mentations exhibit obvious similarities to extensible      ets while remaining oblivious to the underlying net-
routers since both approaches are intended to provide      work topology.
the means to modify router functionality.                     Extensible routers provide a platform for imple-
   The primary difference between the two approaches       menting overlay networks. Programming an overlay
is in the level of trust given the extension. The ex-      into an extensible router requires modifying the clas-
tensible router systems described in this paper re-        sifier to recognize packets in the overlay and then im-
quire that the system trust the entity adding the ex-      plementing a forwarder to handle these packets in an
tension. The code is then fully trusted by the system      overlay-specific way. The advantage of implement-
and is executed with the same privileges as any other      ing an overlay on an extensible router, as opposed to
kernel-level code. The system does not try to pro-         a purely application-level implementation, is simply a
tect itself from the extension code. Active networks,      matter of performance; packets forwarded through the
on the other hand, are designed to let users execute       node can be processed in the kernel.
code on the router. The functions may be specified in
the packet or out-of-band, but they are untrusted and      6 Conclusion
should not be granted access to kernel level data struc-
tures and memory. As a result, active network systems      We have demonstrated that a simple model can de-
usually execute in user space.                             scribe the general operation of an extensible router—
   Since extensions from active network and exten-         routers classify packets, process them, and schedule
sible routers operate separate domains, it is proba-       them for transmission. We have used the model to
bly most accurate to view the two as complementary,        study three extensible routing architectures and shown
and in fact, active network architectures can be imple-    that there are at least three degrees of freedom along
mented using extensible routers. The resulting sys-        which extensible systems may be compared: extensi-
tems would provide both both user-level and kernel-        bility, process scheduling, and flow isolation. Based
level extensibility, and would be much more flexible        on this study, we draw a few conclusions about im-
than a system providing either level alone. To support     provements to these specific systems and to extensible
this possibility, the NodeOS interface [14] , a stan-      architectures in general.
dard interface for active routers, allows the program-        We believe that Scout has distinct advantages over
mer to name forwarders to use for packet processing.       the other two systems both in scheduling computa-
When implemented by an extensible router, the inter-       tions and isolating flows. However, Scout modules are
face provides the ability to name kernel-level exten-      fairly large, making it difficult to extend them incre-
sions as well as standard protocols for an active appli-   mentally. One way to solve this problem is for Scout
cation. Scout has an implementation of the NodeOS          to follow Click’s lead and decompose protocols into
interface; it’s easy to imagine Click and Crossbow         a larger collection of smaller modules. Alternatively,
supporting the same interface.                             Scout could pattern each module’s implementation af-
   In a similar way, one could imagine the control         ter Crossbow, with a well-defined substructure that
interfaces defined by programmable networks (e.g.,          supports incremental extensions.
IEEE P1520 [3] and Nortel’s Openet [10]) being im-            Crossbow is an excellent system for creating modi-
plemented on top of an extensible router. However,         fications to the IP processing path. However, its fixed
this case is less interesting because these interfaces     classifier does not provide the flexibility to implement
forwarders that require different types of classifica-                [7] S. Floyd and V. Jacobson. Random early detection
tion. Extensible routers need to offer some type of                      gateways for congestion avoidance. IEEE/ACM
                                                                         Transactions on Networking, 1(4):397–413, Aug. 1993.
mutable classification since the classifier limits the
type of extensions permitted. Crossbow could also                    [8] Intel Corporation. IXP1200 Network Processor Datasheet,
                                                                         Sept. 2000.
benefit from an actual scheduler after each gate. This
would allow it to scheduled computation, not just                    [9] E. Kohler, R. Morris, B. Chen, J. Jannotti, and M. F.
packet transmission.                                                     Kaashoek. The Click modular router. ACM Transactions
                                                                         on Computer Systems, 18(3):263–297, Aug. 2000.
   Click is the most flexible of the three architectures.
The ability to form virtually any configuration from                 [10] T. Lavian, R. Jaegeer, and J. Hollingsworth. Open
                                                                         programmable architecture for Java-enabled network
the set of elements gives the programmer great free-                     devices. In Hot Interconnects 7: Symposium on High
dom to modify routers incrementally and to add new                       Performance Interconnects, pages 265–272, Stanford, CA
services. The cost of this approach is that the architec-                USA, Aug. 1999. IEEE Computer Society.
ture provides very little guidance on what constitutes              [11] H. Massalin and C. Pu. Threads and input/output in the
a well-formed and meaningful configuration. On the                        Synthesis kernel. In Proceedings of the 12th ACM
other hand, Click includes a set of tools that help man-                 Symposium on Operating Systems Principles, pages
                                                                         191–201, Litchfield Park, AZ USA, Dec. 1989.
age the complexity of dealing with many elements.
                                                                    [12] D. L. Mills. The Fuzzball. In Proceedings of the
                                                                         SIGCOMM ’88 Symposium, pages 115–122, Stanford, CA
Acknowledgements                                                         USA, Aug. 1988.

                                                                    [13] D. Mosberger and L. L. Peterson. Making paths explicit in
We would like to thank the anonymous reviewers and                       the Scout operating system. In Proceedings of the Second
Scott Karlin for helping us improve the clarity and fo-                  USENIX Symposium on Operating System Design and
cus of the paper. This work supported in part by NSF                     Implementation (OSDI), pages 153–167, Seattle, WA USA,
                                                                         Oct. 1996.
grant ANI-9906704, DARPA contract F30602–00–2–
0561, and Intel Corporation.                                        [14] L. Peterson, Y. Gottlieb, M. Hibler, P. Tullmann,
                                                                         J. Lepreau, S. Schwab, H. Dandelkar, A. Purtell, and
                                                                         J. Hartman. An OS interface for active routers. IEEE
References                                                               Journal on Selected Areas in Communications,
                                                                         19(3):473–487, Mar. 2001.
 [1] D. S. Alexander, W. A. Arbaugh, M. W. Hicks, P. Kakkar,
     A. D. Keromytis, J. T. Moore, C. A. Gunter, S. M. Nettles,     [15] X. Qie, A. Bavier, L. Peterson, and S. Karlin. Scheduling
     and J. M. Smith. The SwitchWare active network                      computations on a software-based router. In Proceedings of
     architecture. IEEE Network, 12(3):29–36, May/June 1998.             the ACM SIGMETRICS 2001 Conference, pages 13–24,
                                                                         Cambridge, MA USA, June 2001.
 [2] B. Bershad, S. Savage, P. Pardyak, E. G. Sirer, D. Becker,
     M. Fiuczynski, C. Chambers, and S. Eggers. Extensibility,      [16] M. Shreedar and G. Varghese. Efficient fair queueing using
     safety, and performance in the SPIN operating system. In            deficit round-robin. IEEE/ACM Transactions on
     Proceedings of the 15th ACM Symposium on Operating                  Networking, 4(3):375–385, June 1996.
     System Principles, pages 267–284, Copper Mountain, CO
     USA, Dec. 1995.                                                [17] T. Spalink, S. Karlin, L. Peterson, and Y. Gottlieb. Building
                                                                         a robust software-based router using network processors.
 [3] J. Biswas, A. A. Lazarand, J.-F. Huard, K. Lim,
                                                                         In Proceedings of the 18th ACM Symposium on Operating
     S. Mahjoub, L.-F. Pau, M. Suzuki, S. Torstensson,
                                                                         Systems Principles (SOSP), pages 216–229, Chateau Lake
     W. Wang, and S. Weinstein. The IEEE P1520 standards
                                                                         Louise, Banff, Alberta, Canada, Oct. 2001.
     initiative for programmable network interfaces. IEEE
     Communications Magazine, 36(10):64–70, Oct. 1998.              [18] O. Spatscheck and L. L. Peterson. Defending against
 [4] D. Decasper, Z. Dittia, G. Parulkar, and B. Plattner. Router        denial of service attacks in Scout. In Proceedings of the
     plugins: A software architecture for next generation                Third Symposium on Operating Systems Design and
     routers. IEEE/ACM Transactions on Networking,                       Implementation, pages 59–72, New Orleans, LA USA, Feb.
     8(1):2–15, Feb. 2000.                                               1999.

 [5] A. Demers, S. Keshav, and S. Shenker. Analysis and             [19] D. Wetherall, J. Guttag, and D. Tennenhouse. ANTS: A
     simulation of a fair queueing algorithm. Journal of                 toolkit for building and dynamically deploying network
     Internetworking, 1(1):3–26, Jan. 1990.                              protocols. In The First IEEE Conference on Open
                                                                         Architectures and Network Programming, pages 117–129,
 [6] D. R. Engler, M. F. Kaashoek, and J. O’Toole, Jr.                   San Francisco, CA USA, Apr. 1998.
     Exokernel: An operating system architecture for
     application-level resource management. In Proceedings of
     the 15th ACM Symposium on Operating System Principles,
     pages 251–266, Copper Mountain, CO USA, Dec. 1995.

To top