A Comparative Study of Extensible Routers Yitzchak Gottlieb and Larry Peterson zuki,llp @cs.princeton.edu ¡ Department of Computer Science Princeton University Abstract customization of router functionality. These architec- tures accommodate changes to both the router’s con- Motivated by the demand for routers with new ca- trol plane, as required by programmable networks [3, pabilities, researchers have been building extensible 10], and the router’s data plane, as allowed by active routers that aid in the design and development of net- networks [1, 19]. work protocols and services. This paper evaluates and When considering these systems, it is evident that compares three such systems: (1) Princeton’s Scout- the designers addressed a common set of issues: what based Extensible Router, (2) MIT’s Click router, and functions should routers execute on packets, how to (3) Washington University’s Router Plugins. To pro- specify the function for a particular packet ﬂow, how vide a framework in which these three systems can to identify a packet as part of a ﬂow, and so on. Dif- be studied, the paper also presents a simple model ferent architectures address these issues in different of an extensible router based on four primitive ob- ways. A model that provides a framework for rea- jects: queues, classiﬁers, forwarders, and schedulers. soning about the functioning of routers would be use- By composing these primitive objects it is possible ful both in evaluating currently available extensible to model everything from a standard, best-effort IP routers and in designing future systems. router to an application-level proxy. The paper also The rest of the paper describes one such model brieﬂy discusses the role that extensible routers play and uses it to evaluate three software-based extensible in the construction of active, programmable, and over- routers. The paper is organized as follows. Section 2 lay networks. motivates our model from ﬁrst principles, and Sec- tion 3 uses the model to describe three software-based 1 Introduction extensible router architectures: Scout , Click , and Router Plugins . Section 4 then compares these Several novel network services have been proposed in three systems in terms of the model. Finally, Sec- the past few years. Examples include packet tagging, tion 5 brieﬂy discusses the relationship between ex- application-level proxies, application-speciﬁc packet tensible routers and research in active, programmable, dropping, performance monitoring, intrusion detec- and overlay networks. tion, and assorted ﬁlters and ﬁrewalls. Software based routers  provide an excellent platform for devel- 2 Model oping and deploying these new services. Although software is easily changed, reprogramming a router to This section describes a simple model for extensible support a new service is still a challenging endeavor. routers based on intuitive assumptions of how routers It is also important that the software have an architec- in general, and extensible routers in particular, ought ture that governs how the router may be extended. to work. We do not claim that the model is the only Motivated by earlier work on extensible operating one possible, only that it is useful for understanding systems [2, 6, 13], researchers have recently deﬁned the differences and similarities of the three systems extensible router architectures that support run-time we elect to study. Also, there are many details that we could add to the model to make it more precise. How- teed to a ﬂow. In a router that allows certain packets ever, we include only those details needed to under- to receive a different level of service, packets are not stand the important similarities and differences among necessarily sent in the same order in which they ar- the three example systems. rive. To allow modeling more complex types of queu- ing behavior, we add a second component that allows 2.1 Components us to abstract a router’s queuing discipline. We can then reason about it as a separate component of the In its simplest form, a router forwards packets from system. In our model, a scheduler, depicted in Fig- one of its input ports to one of its output ports. This ure 2, choses a packet from one of its input queues process involves reading the packet from one port, ex- and places it on its single output queue. amining it to decide whether and whither to forward it, and writing the packet out to an output port. Be- cause packets from multiple inputs may contend for a common output port, packets are queued on the output port. Because it is possible that packets arrive on the input link asynchronously with respect to the router, S we also model the input port as a queue. For our pur- poses, we assume both queues are serviced in ﬁrst-in, ﬁrst-out (FIFO) order. In this simple scenario, the router’s primary task is to decide the packet’s outgoing port based on its Figure 2: A packet scheduler destination address—a process typically called packet classiﬁcation. To reﬂect this behavior, the ﬁrst com- Simply connecting a set of classiﬁers (one per in- ponent of our model is a classiﬁer, which dequeues a put port) to a set of schedulers (one per output port) packet from a single input queue and places it on zero, is sufﬁcient to model a bridge or an Ethernet switch— one, or multiple of the available output queues. The network elements that do no processing on the packet classiﬁer places the packet on no queues if it elects beyond classiﬁcation. A router is distinguished in that not to forward the packet (e.g., to implement a ﬁlter), it modiﬁes the packet in some well-deﬁned way; we on one in the common case, and on more than one to say it applies a forwarding function to the packet. In implement multicast. Figure 1 shows a simple conﬁg- general, a router may support more than one forward- uration of a router with a classiﬁer associated with a ing function. For example, commercial routers tend to single input port. In general, a classiﬁer is bound to have at least two functions: one that handles option- each input port. free packets and one that processes IP options. Exten- sible routers extend this practice by allowing arbitrary Output processing. Thus any router model, especially one for Port extensible routers, should include an element encap- Output sulating forwarding functions. Input Port C The last component in our model, a forwarder, re- Port moves at most one packet from its input queue and places one packet on its single output queue. Al- Output though the expectation is that most of the data will Port be similar, the packet placed on the output need not be identical to that placed on the input. Between read- Figure 1: A simple, best effort IP router ing and writing the packet, a forwarder may engage in any amount of arbitrary computation. While it is possible to associate a single queue with Note that some forwarders implement control plane each output port, such a conﬁguration is not general functionality, while others are part of the data plane. enough to model complicated queuing strategies de- Control forwarders typically manage the other com- signed for preserving the quality of service guaran- ponents of the system; they set scheduling parameters, program classiﬁers to recognize new packet ﬂows, constructed by linking the output one classiﬁer to the and get or set state variables associated with data for- input of another. warders. The interface through which control for- warders interact with the rest of the router is beyond the scope of this paper. Forwarders implement a wide range of functional- 2.2 Extensibility ity. For example, the standard IP forwarder decre- ments the TTL ﬁeld, recomputes the checksum, and This model helps us understand that to extend a router, changes the link-layer header. However, routers are one has to change one of the four primitive compo- not limited to this base case. Applications such as nents. However, it does not prescribe exactly how various proxies, peer-to-peer and overlay networks, this “change” is accomplished. As we will see in the packet taggers, and denial-of-service detectors result following sections, different systems provide different in far more complex forwarders. For example, they mechanisms to manage the base components. may modify the IP and TCP headers to reroute the packet to a different host, or they may rewrite the con- For the purpose of this paper, a router may be ex- tents of the packet to request nearby, cached versions tended in three different ways: it is possible to insert of data. In general, several processing steps may be an additional component, modify an existing compo- composed to produce different behaviors. For exam- nent, or replace an existing component. While any ple, a proxy forwarder that rewrites TCP headers may component can be extended in any way, some ex- then pass the new packet to an IP forwarder. tensions are more common for particular components A rich combination of forwarders implies that clas- than others. For example, our model provides ex- siﬁcation may also need to be complex. For example, plicit support for inserting additional forwarders in the a best-effort IP router connected via Ethernet uses the router. In contrast, typically classiﬁers and schedulers Ethernet and IP type ﬁelds to determine the correct are either replaced with an alternative or modiﬁed to processing for the packet and preﬁx matching of the support some new behavior. Modifying a component, IP destination address to decide on the proper routing. in turn, involves either giving it a new set of parame- For many applications, classiﬁcation can be based on ters or extending its functionality with new code frag- both TCP ports and IP addresses. Other classiﬁers ments. The latter method implies that the component may require reading data deep in the packet. For in- has some externally visible structure that can be ex- stance, a router being used as a front end to a cluster of ploited, while the former consists of programming the load-balanced servers can read the HTTP GET com- component (e.g., with a new routing table, level-four mand from a packet and redirect it to an idle server. demultiplexing key, or QoS reservation) and therefore Thus, in general, the classiﬁer must choose not just requires a control interface. an output port, but also the appropriate sequence of As an example, application-speciﬁc droppers, like forwarders. those proposed to thin video streams in the face of Figure 3 shows a complete model of an extensi- congestion, extend the classiﬁer by providing it with a ble router. As mentioned above, this model is under- new function which is applied to each packet. On the speciﬁed; e.g., we do not specify how deep queues other hand, application-level proxies and peer-to-peer can be, whether other components can query its prop- networks add a whole new forwarder into the router; erties and contents, or how to model input and output these applications also modify the classiﬁer by pro- devices. These and other similar details are not re- gramming it to divert packets matching a particular quired to compare the extensible router systems we level-four key over to this forwarder. Finally, conges- describe later, so we leave them unstated. Also, this tion control mechanisms (e.g., RED ) and queuing ﬁgure shows only one possible combination of classi- disciplines used for QoS (e.g., Weighted Fair Queu- ﬁers, forwarders, and schedulers; other combinations ing  and Deﬁcit Round Robin ) replace the sys- are possible. For example, a forwarder may mod- tem’s default scheduler. Each of these new compo- ify packets then “loop back” and re-insert them for nents may allow modiﬁcation. For instance, a RED classiﬁcation. Also, sequences of forwarders can be implementation could offer a control interface for set- linked together and a hierarchy of classiﬁers can be ting its thresholds. F Input Output C F S Port Port F Figure 3: Modeling an extensible router 2.3 Engineering Trade-offs dropped in the middle of a forwarder, but since pack- ets that arrive with a TTL of 1 will be dropped by the The distinction between classiﬁers and forwarders is router, the classiﬁer could decide to drop the packet a matter of degree: classiﬁers may process packets af- by selecting no output queue for it. Another, similar ter they have determined in which queue to put them, choice involves multicast. Since forwarders can cre- and forwarders may make branching decisions. When ate packets, it is possible to model multicast behavior choosing how much processing to put into classiﬁers as a set of two components: a forwarder which cre- and how much decision making to put into forwarders, ates copies of a multicast packet and a classiﬁer would there are two possible extremes. One extreme is to de- then choose a single output for each copy. In order to clare that all branches are classiﬁcation and must be show that unicast is merely a degenerate form of mul- chosen before any processing is done on the packet; ticast, we chose instead to have the classiﬁer create the other extreme is to decide that only the classiﬁ- the copies. cation decision that chooses an output port is deﬁned as classiﬁcation and forwarders may branch arbitrar- 3 Systems ily. Choosing simpler forwarders leads to many clas- sifying components in a router, choosing complicated This section uses the model to describe three ex- ones restricts a router to just one classiﬁer. Similarly, tensible routers. The three systems: Scout, Click, if we permit classiﬁers to modify packets, we ﬁnd that and Router Plugins, were developed independently, some computations are simple enough to be included and each approaches extensibility in a different way. in a classiﬁer. Scout, developed at University of Arizona and later The system designer must choose the component Princeton University, was designed as a stand-alone in which to place a speciﬁc block of code. On real operating systems for network appliances but is also systems, in which compromises must be made for ef- available as a Linux kernel module. Click, developed ﬁcient operation, a middle ground is usually chosen. at MIT, uses a kernel module to replace the Linux One possible criterion is maximum computation time: kernel’s networking subsystem. The Router Plugins classiﬁers and schedulers must complete in a short, u architecture, developed by ETH Z¨ rich, Washington bounded time in order to send or receive packets at University and Ascom under the Crossbow project, line speed. Another criterion might be extensibility: extends the networking subsystem of a BSD kernel if it is difﬁcult to replace or modify the classiﬁer at to provide hooks for new forwarding functions. run time, then the function is best encapsulated in a forwarder. 3.1 Scout For the sake of this paper, we bar classiﬁers and schedulers from modifying packets. We also re- Scout is a modular, communication-oriented operat- quire any branches that change the type of process- ing system. Its major abstraction is the path: a linear ing that is done to a packet, as opposed to those that ﬂow of data that starts at a source device and ends at change the parameters used in the processing, go in a destination device. Paths are composed of stages, the classiﬁer—not the forwarder. One good illustra- which are instances of modules. Each Scout module tion of the choice is decrementing IP’s time-to-live implements a well understood protocol, such as IP or ﬁeld. IP routers must discard packets whose TTL TCP. A module may contribute stages to many paths, decrements to 0. The packet could be processed and but a stage is unique to a single path. The system cre- Path F Stage Stage Stage C F Input Output Stage Stage S Port Port F Stage Stage Stage Stage Figure 4: Modeling Scout ates paths on demand at run time. Scout’s infrastruc- ways. ture provides tools to create, modify, schedule, and First, Scout originally had no scheduler; it’s output control paths. queues were strictly FIFO. In order to schedule the In addition to providing stages to paths, each mod- output one scheduled the paths. This method proved ule also implements a module-speciﬁc, programmer- inefﬁcient since we could not effectively schedule a deﬁned, classiﬁcation function. As part of path cre- given packet’s execution to match its scheduled trans- ation, each module places a reference to the path in mission slot. We therefore added an explicit packet a private data structure that it can access during clas- scheduler to the output module. This allowed us to siﬁcation. If the module cannot uniquely identify the transmit packets whose processing was already com- path, the module stores a reference to another mod- plete immediately when the bandwidth was available. ule’s classiﬁcation function that might be able to dis- Second, to make packet classiﬁcation faster, we en- ambiguate an incoming packet. Scout’s hierarchical, hanced Scout’s classiﬁer with the ability to cache re- extensible packet classiﬁer queries each module for a sults from previous lookups at lower levels in the clas- partial decision based on the packet, then recurs to the siﬁcation hierarchy. For instance, Scout now keeps a next module. The recursion continues until a path is cache of the most recently seen IP addresses to make selected or it is determined that the packet belongs to forwarding lookups on those addresses faster. To clas- no path. sify a packet Scout ﬁrst does simple lookups in its Once a path has been selected, the packet is placed caches, and only if it fails to ﬁnd a match does the on that path’s input queue, and the path is scheduled to full classiﬁcation described earlier take place. run. When Scout’s thread scheduler decides that the Third, when we needed to extend Scout to the In- path’s thread is ready to execute, the thread dequeues tel IXP1200  network processor, we decided not the packet and hands it to the ﬁrst stage in the path, to port Scout directly, but instead designed a minimal which processes it and invokes the next stage. The version for this new environment. The architecture, stages process the packet in sequence, with the last called the Virtual Router Processor (VRP) , al- stage placing the packet on an output queue. Eventu- lows the user to specify a function to be applied to a ally, the scheduler selects the packet from this queue speciﬁc packet ﬂow on the line card. In effect, this al- and transmits it. lows a Scout path to run on one of a hierarchy of pro- cessors. The VRP also extends Scout’s classiﬁcation As shown in Figure 4, Scout has one classiﬁer with hierarchy to an even lower level—the network card. an internal hierarchical structure per input port, one Any packets that should be processed at the line card scheduler per output port, and paths implementing will match in the card’s classiﬁer. Any packet that re- forwarders. Alternatively, we could model each stage quires greater processing, or fails to match at the line as a separate forwarder, but equating paths and for- card, is diverted to Scout’s full classiﬁer running on a warders captures the important properties of Scout. more powerful processor. Given our experience with the Scout operating sys- tem, it is not surprising that it easily reﬂects our 3.2 Click model. However, this is true because we continued to evolve Scout as our notions of a useful design for Like Scout, Click is a modular architecture for build- an extensible router matured. This happened in three ing routers. Its design is based on composing many simple elements to produce a system that implements For example, elements with push-type interfaces have the desired behavior. Each element may have mul- the input chosen for them, so their scheduler is effec- tiple ports to connect it to other elements. Both in- tively null, while elements with pull-type interfaces put and output ports may be either push-type (through have their output selected for them, so their classiﬁer which a packet must be sent) or pull-type (from which is effectively null. a packet may be requested). Pull-type and push-type input ports can connect only to pull-type and push- 3.3 Router Plugins type output ports, respectively. Packets may be stored only in queues, an element that has a push-type input The Router Plugins architecture is designed to allow port and a pull-type output port. limited extensions to an IP router. Implemented in When a packet arrives at the router, the device the NetBSD operating system, Router Plugins allows driver pushes it through a series of elements. Since users to write extensions that can be placed at well elements may have more than one output port, each known points of the router’s IP execution. These element chooses the proper outgoing port as part of points, called gates, were chosen to suit a wide va- the packet processing. The packet is pushed through riety of applications, such as routing, packet schedul- elements until it hits a queue. Once enqueued, the ing, and security processing. The user invokes com- packet waits until it is pulled from its queue. When mand line tools to load plugins at run time. an output device is ready for transmission, the driver Router Plugins may be associated with distinct pulls a packet from one of its upstream neighbors. The ﬂows. Since not every plugin is appropriate for each neighbor will pull the data from one of its upstream packet, Crossbow1 allows the user to specify a ﬁxed neighbors until some element queries a queue. Each key with each plugin. The key speciﬁes IP address individual element decides which of its input ports to pairs, protocol number, ports, and incoming interface; query for a packet. The packet is then removed from any part of the key can be a wildcard meaning that the queue and will traverse the same sequence of mod- any value matches. At each gate, Crossbow attempts ules that selected it before reaching the device driver to match the packet to the keys registered for that gate. and being transmitted. If a key matches, the packet is passed to the plugin in- stance that registered that key. When the packet leaves Input Element Output the plugin, the next part of the IP processing path ex- ecutes until the next gate. This sequence of events repeats until there are no more gates and the IP pro- S F C cessing terminates. As shown in Figure 6, Crossbow can also be mod- eled as a sequence of our basic components. A plugin operates on a single packet then emits it into the pro- Figure 5: Modeling a Click element cessing path, and is, therefore, a forwarder. Crossbow must choose a plugin at each gate, so a gate is a clas- In terms of our model, each Click element encapsu- siﬁer. Upon leaving the gate, the packet must tran- lates one scheduler, one forwarder, and one classiﬁer, sit the next part of the IP processing code, which is as illustrated in Figure 5. The scheduler is necessary just another forwarder. Since all packets must execute because an element has to select a packet from poten- the same IP code after choosing only one forwarder, tially many input ports. Similarly, the classiﬁer is nec- our model dictates that a scheduler must be present. essary because the element may have multiple output Since Crossbow makes no scheduling decision at this ports. The forwarder simply connects the scheduler to time, we place a FIFO scheduler before the next frag- the classiﬁer. ment of IP code. For each deﬁned gate, Crossbow is By leaving any two of these three components a classiﬁer, a forwarder (plugin or null forwarder) and empty, a Click element can implement each of the a scheduler (FIFO). The scheduler’s output queue is primitive components of our model, making it a very 1 For the rest of the paper,we will use the project name, Cross- general architecture. It is more common, however, for bow, to refer to the infrastructure provided by the Router Plugins a given element to deﬁne two of the three components. architecture. F F Plug−ins Plug−ins Gate IP Code Gate IP Code Input C F S F C F S F Port F F Figure 6: Modeling Crossbow’s gates connected to the input queue of the the next classiﬁer compare how much ﬂexibility each system gives us by the next piece of NetBSD’s IP infrastructure. within its ﬁxed architecture. While this decomposition of Crossbow is accurate, in some instances there is a better one. Assuming Classiﬁers that the packet’s IP and TCP headers are not modi- ﬁed by any plugin, the packet need be classiﬁed only Many useful classiﬁers are possible. IP routing is once. The entire sequence of plugins executing on this based on a longest preﬁx match of the destination ad- packet is then known before the ﬁrst forwarder exe- dress. Transport-level packet ﬂows requires match- cutes. We can then deﬁne the forwarding function ex- ing IP addresses and TCP/UDP port pairs. Other ap- ecuting on this packet as the concatenation of all that plications may inspect the message content to differ- packet’s plugins and IP infrastructure code. We no entiate among packets. The classiﬁer fundamentally longer need the FIFO schedulers between gates, and limits the granularity at which ﬂows can be distin- the only remaining scheduler is the packet scheduler guished, and consequently, determines the set of pack- at the output port. Crossbow then looks like Scout, as ets on which a given forwarder operates. Crossbow depicted in Figure 4, except plugins replace stages. and Scout both prescribe a ﬁxed classiﬁer structure which may be modiﬁed, while Click allows new clas- 4 Comparison siﬁers to be inserted in arbitrary places. Crossbow’s classiﬁer can be modiﬁed only by giv- The previous section described three extensible router ing it new keys. It identiﬁes ﬂows by IP address, TCP operating systems and showed how we can model port pair, and incoming network interface, and it uses each of them as a collection of classiﬁers, forwarders, only these ﬁelds to ﬁnd the correct forwarder (plugin) and schedulers. This section compares the systems in at each gate. The classiﬁcation algorithm allows fast areas not restricted by the model. Primarily we will lookups even in the presence of full or partial wild- focus on three categories of differences: card values in any ﬁeld. The function itself, though, is ﬁxed. While Crossbow does not permit replacing Extensibility: Changing a system’s components. the core IP classiﬁcation function, the system will, in Scheduling: Scheduling packet processing. the future, provide a gate for replacing the IP preﬁx Flow Isolation: Segregating ﬂow processing. matching algorithm. It may also allow a plugin to modify the routing table lookup, but these gates will be restricted to operating on ﬂows identiﬁed by IP ad- 4.1 Extensibility dress and port numbers. We ﬁrst consider how each of the three systems can Scout’s single hierarchical classiﬁer can be mod- be extended by either inserting, replacing, or modify- iﬁed by programming new modules and giving new ing its components. It is important to note that nothing keys to existing modules. During path creation, prevents us from extending these systems by rewriting Scout’s classiﬁcation function is implicitly extended any code fragment we choose. However, doing so ob- by adding a call to the demultiplexing function of each viates the reasons for choosing the structure imposed module that contributed a stage to the path. Dele- by that particular architecture. Our goal is to under- gating classiﬁcation to the modules allows ﬂows to stand the structure of the extension mechanism, so we be readily identiﬁed on the basis of any criteria the module programmer chooses. Speciﬁc TCP ﬂows are provide no such facility, and the programmer who im- identiﬁed by their port numbers; IP ﬂows are iden- plements a connection-oriented protocol would have tiﬁed by destination address; Ethernet frames by their to create such a module from scratch. Scout eases the type number, and so on. Changing a speciﬁc module’s programmer’s burden by providing explicit interfaces classiﬁcation function is easier than altering Cross- speciﬁc to a given task, thereby eliminating tedious bow’s since the modiﬁcations are conﬁned to a single semantic checks needed if such calls must be encoded module, as opposed to affecting the entire system. in a simple data push/pull interface. Unlike the other two systems, Click has no global To properly interact with the structure provided classiﬁcation structure. Each element acts as a clas- by the underlying architecture, modules may also siﬁer to one of its outputs during packet processing, implement methods demanded by that architecture. so there is no extra classiﬁcation step. However, in- Router Plugins and Scout modules are both compli- serting a new element into the component graph cre- cated because they exist within a complex infrastruc- ates new possibilities for a packet’s route through the ture. Router Plugins are BSD loadable kernel mod- router and explicitly extends the classiﬁcation avail- ules; they must register themselves with the Plugin able. Like Scout, reprogramming a speciﬁc element’s Control Unit when they are loaded, and they must re- “routing” decision is fairly easy, but one must be care- spond to a small set of messages. Similarly, Scout ful that the module still implements the function for modules must implement an initialization function which it was designed. and a demultiplexing function, in addition to their data forwarding function. In effect, both Router Plugins and Scout modules are factories, creating dynamic, Forwarders run-time instances which are actually used in com- Scout, Click, and Crossbow were all designed to allow putation. The interface for creating and destroying easy insertion of forwarders. In this case, the more in- stages (or plugin instances) adds to the complexity of teresting question is how difﬁcult it is to program an the modules (plugins). entirely new service as one or more forwarders. Al- Click elements are much simpler. Click imposes though this discussion of the relative complexity of very little structure on an element, so initialization is programming a system is not very rigorous, it is still a simple. In order to create a working system, how- useful comparison point. ever, the Click infrastructure provides a mechanism to Perhaps the most convenient measure of the com- tie the elements together at conﬁguration time. Since plexity of creating a new module is the number of this mechanism runs at conﬁguration time, Click must methods a module must support. Modules support two reinitialize the entire conﬁguration each time a new el- classes of methods: those invoked by other modules ement is added, even if the code for the element is in the system, and those invoked by the architecture in already in the kernel. Neither Scout nor Crossbow which the module is embedded. To forward data be- stop processing while installing forwarders; modules tween peers, modules provide calls to transfer packet include the code needed to incrementally insert them- data. A system designer must balance the elegance selves into a running system. In Scout, path creation of a simpler interface with the convenience of a full- affects only those modules that contribute a stage to featured interface. the path. Similarly, when a user registers a Crossbow Each Router Plugin supports a single method to be plugin instance with a certain ﬂow, no other plugin is called when packets arrive for it; Click enhances this affected. simple push interface by adding a pull method. Scout While measuring number of methods is convenient, does not ﬁx the module interface. Instead, it sup- it is not wholly accurate. Simple functions, even if ports multiple interface types; e.g., a ﬁle system in- there are many of them, are easier to write than a terface, a framebuffer interface, and a protocol inter- smaller number of more complex ones. To quantify face. Even its protocol interface is more general and the amount of work needed to implement minimal provides methods for many useful operations. For ex- useful functionality, we also looked at how many lines ample, Scout provides an method that a connection- of code need to be written to create a module in each oriented protocol like TCP can use to determine when system. For the sake of uniformity, we will use the a peer is attempting to connect. Click and Crossbow “null” forwarder which does nothing but forward the packet unchanged. packets efﬁciently. It makes sense that smaller, less restrictive archi- Being designed around NetBSD’s IP core and net- tectures imply less code. The limited set of required work subsystem, Crossbow’s thread scheduling is lim- functionality allows Click elements to be very small; ited by BSD. On packet arrival, packets are classiﬁed the null element is just 8 lines of C++ code. The fact and processed through the gates and sent to the output that elements can be this small encourages program- ports. This process happens with no context switches, mers to write small, easily comprehensible modules no data copying, and no additional interrupt overhead. that are then composed into a complex system. By These are all laudable performance goals, but this de- comparison, the null Router Plugin is 96 lines long, sign may experience a type of head-of-line blocking. and Scout’s null module is 200 lines long. 2 Again, the Consider the situation where two packets arrive, the main reason for having to write more code to imple- ﬁrst packet belongs to a ﬂow that requires process- ment a null module in Scout and Crossbow is that the ing at each available gate, while the second matches module has to manage its own insertion into the larger no ﬁlters at any gate. The second packet can be for- system. warded very quickly with minimal overhead, yet it While allowing smaller modules eases extension, is delayed behind the ﬁrst packet’s processing. This there are compromises made to achieve this beneﬁt. is true even if the packet scheduler later decides that One trade off is that there are many more modules the second packet is more important and should be to understand in a full system. For example, imple- transmitted before the ﬁrst. As detailed in Section 3, menting a minimal IP router requires 38 elements in our model suggests that Crossbow can have a packet Click. Scout implements the same router with eight scheduler between gates, not just at the very end of modules; Crossbow uses just one. Thus, the effort processing. It is possible that such an addition would in programming a router from scratch is probably not enhance Crossbow’s ability to provide different qual- less in Click than in other systems. ities of service to different ﬂows. Click also has the notion that operations on a packet Schedulers should not be interrupted until its processing is com- plete. In particular, a packet is pushed or pulled from The ﬁnal extensible component in our model is the one queue to another. This observation might lead to packet scheduler. Scout has a single scheduler per the conclusion that Click, like Crossbow, must sched- output port. This scheduler can be replaced, but it ule execution based solely on packet arrival times, but is not possible to insert multiple schedulers. In many this is not the case. Click’s thread scheduler handles conﬁgurations, Click has a single packet scheduler per elements. In particular, the device drivers that poll the port; this is a separate component that rests between input and output ports place themselves on the sys- the output port and a queue. Just as with classiﬁers, tem’s scheduling queue. When a driver discovers a however, Click can have multiple components acting packet, the input process pushes it through the ele- as a scheduler, and therefore allows users to both in- ments until the packet is placed on some queue. Sim- sert and replace schedulers, as well as modify them ilarly, the output process pulls a packet from the out- with different parameters. Crossbow provides a spe- put device’s upstream neighbor, and from it’s neigh- cial gate for packet scheduling decisions, allowing the bor until a queue is reached. Thus, the placement of user to insert a new packet scheduler for a given ﬂow queues in the element graph determines when process- and modify previously registered instances. ing is scheduled. By placing queues early in the graph, it is possible to force packet processing to match avail- 4.2 Process Scheduling able output, thereby avoiding head-of-line blocking. Unfortunately, since a packet may come from any One aspect of a router that our model does not restrict reachable queue in the graph, it becomes nearly im- is the scheduling of packet processing. The choice of possible to predict how long the execution of a sin- how to schedule these computations can have a no- gle packet might take. A scheduler may decide that a ticeable impact on the ability of the router to forward packet ought to come from one of its neighbors only 2 The null VRP forwarder is completely empty since it has no to discover that the function operating on that packet interface to implement. takes too long to complete. Instead of scheduling individual modules, Scout ble to store ﬂow-speciﬁc state without needing a new schedules whole paths for execution. Since a path lookup at each element, since the packet is the only for a packet is chosen at packet arrival, Scout can item which is passed along. predict the processing time that will be spent on a For a given Click conﬁguration, however, it is pos- packet. This knowledge allows high priority packets sible to create another conﬁguration that can distin- to be scheduled ﬁrst for execution, not just transmis- guish ﬂows. To do this, the classiﬁcation decision sion. Also, packets may be dropped if the scheduler from all of the elements must be separated out and predicts that executing the path will degrade forward- placed at the beginning of the element graph. Each ing performance of other packets to unacceptable lev- possible outcome would feed a single queue. Each els. of these queues would feed an element graph with no branches, all of which feed into a scheduler at the out- put port. Thus, by manually enumerating each possi- 4.3 Flow Isolation ble path through the system, ﬂows could be isolated. It The ability of a system to explicitly recognize and ma- would probably be unwise to attempt to do this, how- nipulate the processing of packet ﬂows is important to ever. It would make the element graph more com- its ability to manage its resources. As noted in Sec- plex, hence more difﬁcult to read, thereby negating tion 4.2, a router can provide service guarantees if it Click’s primary advantage. Also, in the current design can schedule the processing of ﬂows individually . of Click, this effort would most likely result in code Processing sequences specialized for a known pur- bloat since elements would have to be repeated sev- pose can also be simpler and therefore execute more eral times in the element graph. Scout solves the con- quickly . A system that isolates the processing of ﬁguration problem by making each module a factory particular ﬂows can also more effectively monitor re- of stages for paths, thereby avoiding multiple declara- sources used by the ﬂow and therefore better detect tions in many cases. denial-of-service attacks . Packet ﬂows map very naturally onto Scout’s path Neither Crossbow nor Click has explicit support for abstraction. Paths are Scout’s unit of scheduling as isolating ﬂows. In both cases, the design of the archi- well as its unit of resource management. Paths allow tecture seems to preclude it. Crossbow’s monolithic Scout to isolate the processing of one ﬂow of packets infrastructure indicates the existence of a single ex- from the processing of any other ﬂow. Memory and ecution path for processing all ﬂows. As described CPU use are accounted to individual paths allowing earlier, however, a single lookup can be sufﬁcient to Scout to monitor at a ﬁne granularity the resources determine which sequence of plugins will operate on consumed by a packet ﬂow. Given our long experi- a packet. Knowing this sequence would allow Cross- ence with Scout, it is once again not unreasonable that bow to predict the execution time of the packet and we would endorse this particular abstraction. therefore schedule the packet’s processing more effec- Scout’s paths also give individual modules access tively. Unfortunately, this is not possible in Crossbow. to the current state of the ﬂow-speciﬁc module that This is because even though a plugin may be able to has acted, or will shortly act, on a given packet. The recognize different classes of packets that should be module can just query the path for its state or speciﬁc treated differently by the system, since it is activated variables. Given the lack of path support in Click and only after the initial classiﬁcation, that knowledge is Crossbow, the mechanism for determining this state useless to the system as a whole. must be different. Click provides a notion of a ﬂow- Click, by its very nature, precludes isolating indi- based router context that provides the ability to query vidual ﬂows through the system. Click makes each el- elements up-stream and down-stream of the current ement with more than one output responsible for clas- element. Using ﬂow-based router context, it is possi- siﬁcation. Since the decision can only be made once ble to identify the set of possible elements that could an element has seen the packet, the set of elements act on the packet, but not the sequence that will act, operating on a packet cannot be known until the ac- or in fact has acted, on it. Crossbow mandates that tual processing has already completed. This makes it each plugin must support a simple communication in- impossible to treat individual paths through the sys- terface between plugins, so it is possible to get infor- tem as separate entities. Furthermore, it is impossi- mation about particular packets if any is stored. It is not possible, however, for a plugin to determine which only allow application programs to control conven- other plugins were activated on the same packet, so it tional router behavior; they do not support adding or is still not known with whom to communicate. controlling new forwarders. Overlay networks are becoming an increasingly 5 Applications popular way to introduce new functionality into the network. Within an overlay, each router’s view of the Active networks are networks that allow users to network is restricted to include only those nodes that specify the forwarding functions applied to pack- are also participating in the overlay. These nodes can ets, thereby making the network conﬁgurable on an either be conventional routers or end hosts. In the lat- application-speciﬁc basis. Active network imple- ter case, the resulting end-system nodes forward pack- mentations exhibit obvious similarities to extensible ets while remaining oblivious to the underlying net- routers since both approaches are intended to provide work topology. the means to modify router functionality. Extensible routers provide a platform for imple- The primary difference between the two approaches menting overlay networks. Programming an overlay is in the level of trust given the extension. The ex- into an extensible router requires modifying the clas- tensible router systems described in this paper re- siﬁer to recognize packets in the overlay and then im- quire that the system trust the entity adding the ex- plementing a forwarder to handle these packets in an tension. The code is then fully trusted by the system overlay-speciﬁc way. The advantage of implement- and is executed with the same privileges as any other ing an overlay on an extensible router, as opposed to kernel-level code. The system does not try to pro- a purely application-level implementation, is simply a tect itself from the extension code. Active networks, matter of performance; packets forwarded through the on the other hand, are designed to let users execute node can be processed in the kernel. code on the router. The functions may be speciﬁed in the packet or out-of-band, but they are untrusted and 6 Conclusion should not be granted access to kernel level data struc- tures and memory. As a result, active network systems We have demonstrated that a simple model can de- usually execute in user space. scribe the general operation of an extensible router— Since extensions from active network and exten- routers classify packets, process them, and schedule sible routers operate separate domains, it is proba- them for transmission. We have used the model to bly most accurate to view the two as complementary, study three extensible routing architectures and shown and in fact, active network architectures can be imple- that there are at least three degrees of freedom along mented using extensible routers. The resulting sys- which extensible systems may be compared: extensi- tems would provide both both user-level and kernel- bility, process scheduling, and ﬂow isolation. Based level extensibility, and would be much more ﬂexible on this study, we draw a few conclusions about im- than a system providing either level alone. To support provements to these speciﬁc systems and to extensible this possibility, the NodeOS interface  , a stan- architectures in general. dard interface for active routers, allows the program- We believe that Scout has distinct advantages over mer to name forwarders to use for packet processing. the other two systems both in scheduling computa- When implemented by an extensible router, the inter- tions and isolating ﬂows. However, Scout modules are face provides the ability to name kernel-level exten- fairly large, making it difﬁcult to extend them incre- sions as well as standard protocols for an active appli- mentally. One way to solve this problem is for Scout cation. Scout has an implementation of the NodeOS to follow Click’s lead and decompose protocols into interface; it’s easy to imagine Click and Crossbow a larger collection of smaller modules. Alternatively, supporting the same interface. Scout could pattern each module’s implementation af- In a similar way, one could imagine the control ter Crossbow, with a well-deﬁned substructure that interfaces deﬁned by programmable networks (e.g., supports incremental extensions. IEEE P1520  and Nortel’s Openet ) being im- Crossbow is an excellent system for creating modi- plemented on top of an extensible router. However, ﬁcations to the IP processing path. However, its ﬁxed this case is less interesting because these interfaces classiﬁer does not provide the ﬂexibility to implement forwarders that require different types of classiﬁca-  S. Floyd and V. Jacobson. Random early detection tion. Extensible routers need to offer some type of gateways for congestion avoidance. IEEE/ACM Transactions on Networking, 1(4):397–413, Aug. 1993. mutable classiﬁcation since the classiﬁer limits the type of extensions permitted. Crossbow could also  Intel Corporation. IXP1200 Network Processor Datasheet, Sept. 2000. beneﬁt from an actual scheduler after each gate. This would allow it to scheduled computation, not just  E. Kohler, R. Morris, B. Chen, J. Jannotti, and M. F. packet transmission. Kaashoek. The Click modular router. ACM Transactions on Computer Systems, 18(3):263–297, Aug. 2000. Click is the most ﬂexible of the three architectures. The ability to form virtually any conﬁguration from  T. Lavian, R. Jaegeer, and J. Hollingsworth. Open programmable architecture for Java-enabled network the set of elements gives the programmer great free- devices. In Hot Interconnects 7: Symposium on High dom to modify routers incrementally and to add new Performance Interconnects, pages 265–272, Stanford, CA services. The cost of this approach is that the architec- USA, Aug. 1999. IEEE Computer Society. ture provides very little guidance on what constitutes  H. Massalin and C. Pu. Threads and input/output in the a well-formed and meaningful conﬁguration. On the Synthesis kernel. In Proceedings of the 12th ACM other hand, Click includes a set of tools that help man- Symposium on Operating Systems Principles, pages 191–201, Litchﬁeld Park, AZ USA, Dec. 1989. age the complexity of dealing with many elements.  D. L. Mills. The Fuzzball. In Proceedings of the SIGCOMM ’88 Symposium, pages 115–122, Stanford, CA Acknowledgements USA, Aug. 1988.  D. Mosberger and L. L. Peterson. Making paths explicit in We would like to thank the anonymous reviewers and the Scout operating system. In Proceedings of the Second Scott Karlin for helping us improve the clarity and fo- USENIX Symposium on Operating System Design and cus of the paper. This work supported in part by NSF Implementation (OSDI), pages 153–167, Seattle, WA USA, Oct. 1996. grant ANI-9906704, DARPA contract F30602–00–2– 0561, and Intel Corporation.  L. Peterson, Y. Gottlieb, M. Hibler, P. Tullmann, J. Lepreau, S. Schwab, H. Dandelkar, A. Purtell, and J. Hartman. An OS interface for active routers. IEEE References Journal on Selected Areas in Communications, 19(3):473–487, Mar. 2001.  D. S. Alexander, W. A. Arbaugh, M. W. Hicks, P. Kakkar, A. D. Keromytis, J. T. Moore, C. A. Gunter, S. M. Nettles,  X. Qie, A. Bavier, L. Peterson, and S. Karlin. Scheduling and J. M. Smith. The SwitchWare active network computations on a software-based router. In Proceedings of architecture. IEEE Network, 12(3):29–36, May/June 1998. the ACM SIGMETRICS 2001 Conference, pages 13–24, Cambridge, MA USA, June 2001.  B. Bershad, S. Savage, P. Pardyak, E. G. Sirer, D. Becker, M. Fiuczynski, C. Chambers, and S. Eggers. Extensibility,  M. Shreedar and G. Varghese. Efﬁcient fair queueing using safety, and performance in the SPIN operating system. In deﬁcit round-robin. IEEE/ACM Transactions on Proceedings of the 15th ACM Symposium on Operating Networking, 4(3):375–385, June 1996. System Principles, pages 267–284, Copper Mountain, CO USA, Dec. 1995.  T. Spalink, S. Karlin, L. Peterson, and Y. Gottlieb. Building a robust software-based router using network processors.  J. Biswas, A. A. Lazarand, J.-F. Huard, K. Lim, In Proceedings of the 18th ACM Symposium on Operating S. Mahjoub, L.-F. Pau, M. Suzuki, S. Torstensson, Systems Principles (SOSP), pages 216–229, Chateau Lake W. Wang, and S. Weinstein. The IEEE P1520 standards Louise, Banff, Alberta, Canada, Oct. 2001. initiative for programmable network interfaces. IEEE Communications Magazine, 36(10):64–70, Oct. 1998.  O. Spatscheck and L. L. Peterson. Defending against  D. Decasper, Z. Dittia, G. Parulkar, and B. Plattner. Router denial of service attacks in Scout. In Proceedings of the plugins: A software architecture for next generation Third Symposium on Operating Systems Design and routers. IEEE/ACM Transactions on Networking, Implementation, pages 59–72, New Orleans, LA USA, Feb. 8(1):2–15, Feb. 2000. 1999.  A. Demers, S. Keshav, and S. Shenker. Analysis and  D. Wetherall, J. Guttag, and D. Tennenhouse. ANTS: A simulation of a fair queueing algorithm. Journal of toolkit for building and dynamically deploying network Internetworking, 1(1):3–26, Jan. 1990. protocols. In The First IEEE Conference on Open Architectures and Network Programming, pages 117–129,  D. R. Engler, M. F. Kaashoek, and J. O’Toole, Jr. San Francisco, CA USA, Apr. 1998. Exokernel: An operating system architecture for application-level resource management. In Proceedings of the 15th ACM Symposium on Operating System Principles, pages 251–266, Copper Mountain, CO USA, Dec. 1995.
Pages to are hidden for
"A Comparative Study of Extensible Routers"Please download to view full document