Internet2 Network Service Descriptions

Document Sample
Internet2 Network Service Descriptions Powered By Docstoc
					Internet2 Network Service

     v1.1 — February 7, 2007
                                                Table of Contents

Executive Summary................................................................................................................................ 1
1. Network Infrastructure and Services ........................................................................................... 3
  1.1      Introduction............................................................................................................................... 3
  1.2      Architectural Overview ............................................................................................................ 5
  1.3      Circuit Service Descriptions ..................................................................................................... 7
     1.3.1      The Static Circuit Service................................................................................................. 8
     1.3.2      The Dynamic Circuit Service ........................................................................................... 9
     1.3.3      The HOPI Test Service................................................................................................... 10
  1.4      IP Services .............................................................................................................................. 11
     1.4.1      IP Backbone Network..................................................................................................... 11
     1.4.2      Commercial Peering Service .......................................................................................... 13
     1.4.3      Commodity Transit Service............................................................................................ 13
     1.4.4      MPLS Services ............................................................................................................... 14
2. Details of Multiservice Switching Infrastructure and Dynamic Circuit Services .................. 15
  2.1      Multiservice Switching Infrastructure Architecture ............................................................... 15
     2.1.1      Multiservice Switch Description .................................................................................... 15
     2.1.2      Interconnections between Boxes and Users ................................................................... 16
     2.1.3      Multiservice Switch Control Plane................................................................................. 18
  2.2      Dynamic Circuit Service......................................................................................................... 19
     2.2.1      Dynamic Circuit Details ................................................................................................. 19
     2.2.2      Dynamic Circuit Control ................................................................................................ 20
  2.3      Connecting Internet2 Dynamic Circuit Services to Users...................................................... 20
    2.3.1       Connecting Two Regional Networks and Their Users ................................................... 21
     2.3.2      Connecting Internet2 with other National and International R&E Networks ............... 22
  2.4      Use Cases................................................................................................................................ 24
     2.4.1      Applications.................................................................................................................... 24
     2.4.2      Models ............................................................................................................................ 25
Appendix A. Dynamic Circuit Service — Detailed Description....................................................... 27
  A.1      Connecting to the Dynamic Circuit Service ........................................................................... 27
  A.2      Transport Service Definition .................................................................................................. 27
  A.3      Encapsulating Ethernet in SONET ......................................................................................... 28
  A.4      Control Plane Overview ......................................................................................................... 28
Appendix B. Bandwidth Used by Physical Interface......................................................................... 37
Appendix C. Ethernet Issues in the Dynamic Circuit Service.......................................................... 39
Appendix D. Control Plane Development and Deployment Schedule ............................................. 41
Appendix E. One Example of How to Connect a Campus to the Dynamic Circuit Services ....... 43
                                       Executive Summary
The new Internet2 Network will be a hybrid network, offering three production services plus the HOPI
test service.

The IP service is the successor to the Abilene Network. It will retain all of Abilene's capabilities (e.g.,
IPv6, multicast) while notably improving on Abilene in the areas of performance and performance
measurement. Commercial peering and commodity transit for the IP service are currently under
consideration. We currently expect the transition from Abilene to be complete by April 2007.

The dynamic circuit service will enable users to set up dedicated point-to-point lightpaths. This will be
done manually, at first. Control plane software is under development, building on the DRAGON
project, with the goal of enabling automated reservations by May 2007. Dynamic circuits are
envisioned to have lifetimes ranging from a few minutes to a few months.

A static circuit service will be available for those who need dedicated lightpaths with lifetimes of
several months or longer.

Finally, the HOPI test service will be available for engineers and researchers to experiment with new
hybrid networking technologies including, but not limited to, technologies slated for deployment in the
wider Internet2 Network.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                   1

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                  2
Network Infrastructure and Services
       1.1     Introduction

This document describes the infrastructure and services of the Internet2 hybrid network. To do so, it
first gives an overview of the Internet2 network architecture showing how the different services and
infrastructures fit into the total picture. One important goal in showing the architecture is to show the
distinction between the infrastructure being rolled out and the services offered on the infrastructure.

Internet2 has standardized the new network connection model around a 2 x 10Gbps connection.
(Slower connections from 1Gbps will be supported during a migration period. More information is
available on the Fee Structure page at One connection will be
used for IP services; the other for dynamic circuit services. The bandwidth of the physical connection
to the network and the bandwidth used on the network connection do not need to match. The
expectation is that most connectors will connect at 10Gbps, with their traffic rate-limited, if necessary,
based on the terms of their connection agreement.

In addition to the two connections included in the standard network connection model, some users will
want to have additional connections to support static point-to-point circuits across the Internet2
network. Such circuits will be provided by the static circuit service; they will run from one Internet2
Point of Presence (PoP) to another.

Part 1 of this document presents an overview of the network architecture and services. Part 2 describes,
in detail, the new circuit services and infrastructures that are offered. The circuit service description
includes details of physical connections to the network, the bandwidth of the circuits across the
network, and other details of circuit connections.

The Internet2 physical implementation is made up of several related infrastructures. Each of these
infrastructures has its own architecture. The infrastructures are the:
1) Core infrastructure
2) IP infrastructure
3) Multiservice switching infrastructure
4) HOPI testbed infrastructure

In the initial rollout, these infrastructures will be kept as independent as possible, both to limit the
amount of interaction between technologies while we gain experience with new technology and to
make transition from the previous network infrastructure as straightforward as possible.

The mapping of circuit services to infrastructure for the first rollout is:
1) Static circuits – provided on the core infrastructure or on the multiservice switching infrastructure
2) Dynamic circuits – provided on the multiservice switching infrastructure
3) HOPI test service – provided on the HOPI testbed infrastructure

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                     3
Note that infrastructures can and do use services provided by other infrastructures. For example, the IP
network, the multiservice switching infrastructure and the HOPI testbed use static circuits provided by
the core infrastructure to interconnect their boxes. In the future, the IP network and the HOPI testbed
may use static circuits from the multiservice switching infrastructure as well as from the core
infrastructure. The HOPI testbed might use MPLS paths over the IP network. In addition, services
provided may use multiple infrastructures in interesting combinations.

For example, the dynamic circuit service initially is provided over the multiservice switching
infrastructure but, in the future, the service may be enhanced so that the core and multiservice
switching infrastructures are used in an integrated way, such that, when the multiservice switching
infrastructure does not have enough bandwidth to satisfy a request, it will attempt to get more
bandwidth from the core infrastructure. Other combinations are possible as we gain experience with
the network and the ways people want to use it. The ability to combine infrastructures and services to
create new capabilities and services is what makes the hybrid network unique and powerful. The form
of this future integration in services and infrastructure is a subject for a later document. This document
focuses on describing the initial basic infrastructure and services.

In Part 1 of this document, all infrastructures and services are described at a functional level. In
addition to the overall architecture, Part 1 includes a section that describes the IP network and the IP
services provided on it, and a section that describes circuit infrastructures and circuit services,
including the dynamic circuit service and multiservice switching infrastructure.

In Part 2, the dynamic circuit services and the multiservice switching infrastructure are described in
more detail. This is because the dynamic circuit service is new and complex. The IP service, while
complex, is an enhancement of the Abilene network, with some additional services added. The static
circuit service is simply providing a point-to-point circuit across the network. The HOPI test service is
a continuation of the earlier HOPI network, using some new infrastructure. Part 2 has four sections:

2.1) Multiservice Switching Infrastructure
This section includes an architectural diagram of the multiservice switch, and a description of how
multiservice switches physically interconnect with each other. The multiservice switches are controlled
by independent control plane software, and a brief overview is given, showing the interaction between
the data plane that creates the actual circuits, and the control plane that sets them up and tears them

2.2) Dynamic Circuit Service
The dynamic circuit service section describes how dynamic circuits are set up between users and
Internet2. Two subsections describe the connectivity at the circuit level and at the control level.

2.3) Connecting Internet2 Dynamic Circuit Services to Users
To show how dynamic circuits may connect to users and other networks, a section showing possible
connection scenarios is included. Users of dynamic circuit services include regional networks, end-
users, and other provider networks. By connecting to Internet2 dynamic services, users are able to
connect to others who are also connected to Internet2 dynamic services. If Internet2 is connected to
other provider networks, then a user on Internet2 may create circuits to a user on a different provider

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                    4
network. Several connection scenarios show how Internet2 may be used in providing point-to-point
circuits across several domains. This also shows the coordination of control planes that is needed to
create such circuits between domains.

2.4) Use Cases
This section presents a number of use cases showing applications where dynamic circuits can be used.
This section has been written before most of the applications have been deployed. Therefore, the use
cases include some that are in a planning stage as well as some that are presented as possible cases for
consideration. This section includes models of how network resource allocation may be managed and
the contrasting benefits of different models for different applications. The model part of the section is
to provide ideas about what might be done, and to help shape thinking on how resource sharing in the
dynamic circuit environment can be most effective.

The expectation is that this will be a living document that will be modified over time as the network,
and our understanding of how to best describe it, evolves. For example, the document could describe
the Infinera, Ciena, and HOPI hardware in more detail. It could also add the need for testing and
debugging circuit services, the need to support performance testing between segments of circuits, the
need to support perfSONAR on cross-domain circuits, and other debugging and setup issues.

       1.2     Architectural Overview

This section describes the architecture of the Internet2 network. The network consists of multiple
interconnected infrastructures, each with independent architectures. The infrastructures are used by
Internet2 to provide network services to users. The Internet2 network provides the following broad
classes of services: 1) IP service and 2) circuit service.

The IP service is an IP network that replaces and enhances the Abilene network. The details of the
services provided by the IP network are described in Section 1.4.

The circuit service, described in Section 1.3, provides point-to-point circuits between ports on the
Internet2 network. These circuits provide guaranteed bandwidth and deterministic behavior in terms of
bounds on jitter, latency and packet loss. Three different circuit services are provided:
1) Static circuit service
2) Dynamic circuit service
3) HOPI test service

There are four architecturally-distinct infrastructures in the Internet2 network:
1) Core infrastructure, consisting of fiber segments interconnected with wave equipment
2) IP network
3) Multiservice switching infrastructure
4) HOPI testbed infrastructure

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                  5
Figure 1 is a diagram of the overall architecture of the Internet2 network. In the initial deployment,
each service will be supplied by a particular infrastructure. Note that this diagram does not show all the
nodes and connectors, but shows how the different services interact architecturally. It also shows some
of the similarities and differences between the services.

                                                 Figure 1

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                6
Architecturally, the base of the Internet2 network is a core infrastructure, which consists of fiber
segments carrying waves that are connected with wave equipment. Each of the fibers carries a number
of waves; the Infinera wave equipment connects a wave on one fiber to a wave on another fiber. By
connecting waves at sequential wave equipment nodes, a circuit from a port on local wave equipment
to a port on remote wave equipment may be instantiated.

The other Internet2 infrastructures use static circuits created on the core infrastructure to interconnect
their devices. The IP network uses the core infrastructure to create links between routers. The routers
form an IP network functionally identical to the previous Internet2 IP network (Abilene).

The multiservice switching infrastructure and HOPI testbed infrastructure also use static circuits
created on the core infrastructure to interconnect their nodes. As shown in Figure 1, the IP, dynamic
circuit and HOPI infrastructures use the core infrastructure in the same way. All infrastructures are
architecturally independent of each other, although they may use services from one another to
implement their architectures. Future documents will describe how the infrastructures and services
may integrate more extensively in the future.

       1.3     Circuit Service Descriptions

For the initial deployment of Internet2, each circuit service will be deployed in a way that limits
interaction between infrastructure elements providing different services. As the network matures, the
level of integration of both the services and the infrastructure will increase. This integration is a subject
for a future document. However, it should be noted that separation of the architecture of the services is
likely to remain even as the services begin to interoperate. This architectural independence is valuable
for the maintenance of the integrity of each service, independently of the others.

Two major aspects of the circuit service are the:
1) Physical connection between the Internet2 device and the user device
2) Bandwidth of the circuit across Internet2

The physical connection is defined by the type and speed of the device connecting to the service, while
the bandwidth of the circuit across Internet2 may be any multiple of STS-1 (approximately 50Mbps)
from STS-1 to STS-192.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                   7
                                                 Figure 2

Figure 2 is an illustration of a user A connecting to the circuit service with a 10Gbps interface, user B
connecting with a 1Gbps interface, and a circuit between users A and B of 150Mbps. This illustrates
that the speed of the interfaces and the bandwidth of the circuit do not have to match.

               1.3.1 The Static Circuit Service

The static circuit service allows users to create point-to-point circuits across the Internet2 network.
Users connect at an Internet2 PoP to either the Infinera core infrastructure wave equipment or the
Ciena multiservice switching infrastructure. The capabilities and requirements of each differ slightly.

For static circuits provided by the Infinera infrastructure, the physical interface to the static circuit
service is at either 10GE or OC-192 SONET. Connection across Internet2 is Ethernet circuit or
SONET circuit and matches the type of the physical interface. Bandwidth of static circuits provided by
Infinera is 10Gbps.

If the connection to the static service is provided by the Ciena multiservice switching infrastructure,
the physical connection may be 1GE, 10GE, or SONET at OC-48 or OC-192. In the Ciena
infrastructure, connection across Internet2 is always SONET. If the physical device connecting to the
multiservice switch is Ethernet, the Ethernet frames are encapsulated in SONET using Generic
Framing Protocol (GFP). When using the Ciena multiservice switch, the bandwidth across the network
can be any multiple of STS-1. Note that the bandwidth of the physical interface and the bandwidth of
dynamic circuits carried over these interfaces do not have to be the same. For example, a user could
connect using a 1GE interface and request a ~100 Mbps (2 x STS-1) circuit across Internet2.

In all cases of users connecting to the static circuit service using SONET, the incoming SONET may
be channelized or not; for example, the connection may be OC-48 or OC-48c.

Static circuits are provisioned by the Internet2 NOC. Connections between Internet2 and a user are
coordinated with the user's NOC. Static circuits are long-lasting connections (e.g., a year) that are

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                   8
setup between two specific users; they bring a connection specific to that circuit to an Internet2 PoP.
(More information is available on the Fee Structure page at

               1.3.2 The Dynamic Circuit Service

The dynamic circuit service allows users to create point-to-point circuits across Internet2 for shorter
periods of time than with static circuits. The dynamic circuit service uses control plane software to set
up the circuits. A user connects physically to the dynamic infrastructure either as a single circuit to be
switched, or as a point-to-point circuit that multiplexes multiple circuits over the physical connection.
The current expectation is that most users will connect in multiplexing mode, but either mode is

On request, a dynamic circuit is created across Internet2 from one user to the other. The Internet2
circuit connects at either end to a user circuit. The user circuit is, as described above, either a single
circuit or a multiplexed circuit.

Dynamic circuits are expected to be set up for periods of time ranging from minutes to months. The
dynamic circuit service is essentially a switching service that creates circuits between users. As such,
its value depends significantly on having a set of users physically connected to it that want to make
connections between each other periodically. Topologically, it is similar to a telephone exchange.

The dynamic circuit service is provided by the Ciena multiservice switching infrastructure. Physical
connection to the multiservice switch is provided at an Internet2 PoP. The physical interface to the
multiservice switch may be Ethernet — with or without VLANs — or SONET.

The dynamic service uses control plane software to create and tear down circuits. The ability to set up
circuits is permitted to a limited set of people. Initially, permissions will be granted only to certain
Internet2 staff, but in the longer term, others will be given permissions to make at least some
connections. In all cases, the circuits will be made using control plane software.

Most circuits that cross Internet2 are expected to be a segments of longer point-to-point circuits,
including segments in regional networks and perhaps in other provider networks as well as in
Internet2. When setting up such cross-domain circuits, it is necessary to coordinate the setup of all the
segments with each of the organizations. Coordination may be done in two ways: 1) staff at different
organizations using email, phone, or other manual methods, or 2) using software at each participating
organization to set up cross-domain connections automatically.

To make cross-domain connections automatically, users will need appropriate control plane software.
Internet2 will provide experimental software for regional networks that will allow such
interconnection. Support is also planned for circuits that connect with other networks automatically, as
well as connecting with other networks using staff to staff communications.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                      9
               1.3.3 The HOPI Test Service

The HOPI test service is a set of equipment that allows experimentation in implementing and using
dynamic point-to-point circuits. The test service is carried over from the original experimental HOPI
network that ran on a different set of waves. At the start, the new HOPI service is expected to be
functionally identical with the previous service. It is expected that the HOPI test service will expand,
though perhaps in unexpected ways.

HOPI is experimental in a number of ways; it:
1) Has a footprint limited relative to the total Internet2 footprint.
2) Uses experimental control plane software to create circuits.
3) Uses its control software to interconnect with other domains using their control software.
4) Allows other networks to interconnect with it to test their inter-domain circuit capabilities.

As with the dynamic circuit service, users connect physically to an Internet2 PoP that contains a HOPI
access point. As with the dynamic circuit service, a physical connection carries one or more sub-
circuits. Each sub-circuit may be connected over a circuit across the HOPI infrastructure to another
circuit connected to a HOPI access point.

The kinds of physical connections available on this service will change over time as devices are added.
For the initial service, the connections will consist of the existing HOPI Force10 Ethernet switches and
DRAGON software.

The physical interfaces are 1GE or 10GE. Connections across HOPI are point-to-point Ethernet
VLAN-based circuits. Circuit bandwidth is in increments of 100Mbps. Requests for guaranteed
bandwidth circuits may be made by using GMPLS-style Peer Mode, GMPLS-style UNI Mode, Web
Service API, or email/phone to the NOC. User input devices must support standard Ethernet 802.1q
VLAN capabilities.

The control plane software used in HOPI is developed by a set of collaborators including Internet2,
DRAGON, University of Southern California/Information Sciences Institute East (USC/ISI-East),
Mid-Atlantic Crossroads (MAX), and others. In addition to using the waves of the core infrastructure,
HOPI will connect to the dynamic services of Internet2 as a separate domain. The network control
software on HOPI will interoperate with dynamic services as a separate domain, setting up circuits
automatically that use segments in both the HOPI and the dynamic circuit domains, as well as,
perhaps, other domains. In this way, HOPI provides a way to demonstrate control plane

HOPI also may be used to exercise and test new control plane software being developed by Internet2
and others as well as being used for testing dynamic services being developed by other domains. It is
likely that this will include HOPI interconnects to test labs and with other organizations, such as
regional networks, that are planning to deploy or develop dynamic circuit provisioning control plane
software and infrastructure.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                    10
Finally, HOPI may be used to allow testing of applications prior to using them on the dynamic or static
circuit services. For example, performance measuring software for dynamic services networks may be
tested on HOPI before rolling it out on the dynamic circuit infrastructure.

       1.4     IP Services

Internet2 provides IP services that are a continuation and enhancement of services provided by the
Abilene Network. IP services are provided by the IP infrastructure, which uses circuit service from the
previously-described core infrastructure to form a new Internet2 IP backbone network. The IP services
provided consist of a standard IP backbone network service that can be viewed as a replacement for
Internet2’s existing Abilene backbone network, and new services that are contemplated to be available
upon request on the IP backbone network. These services, described below, are the commercial
peering service (currently undergoing a beta trial using the Abilene backbone) and the commodity
transit service.

               1.4.1 IP Backbone Network

The new Internet2 IP backbone network consists of routers at nine locations, connected with multiple
physically-diverse 10Gbps backbone links. Each connection is provided on a dedicated 10Gbps
wavelength between the two interconnecting locations. As traffic needs increase beyond 10Gbps,
Internet2 will allocate additional waves on the core backbone to augment the capacity for the IP
backbone network. The routers are located at:

   •   New York
   •   Washington
   •   Chicago
   •   Atlanta
   •   Houston
   •   Kansas City
   •   Los Angeles
   •   Salt Lake City
   •   Seattle

All of the services provided by Abilene will be provided on the new Internet2 network. Figure 3 shows
the Internet2 IP network overlaid on the core infrastructure.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                              11
                                                Figure 3

Connecting to the Internet2 IP Network

The connector must build out the capabilities to reach the Internet2 suite in one of the Level(3) PoPs
where Internet2 has equipment. Internet2 will carry the connection to one of the IP network backbone
routers for IP connectivity. Generally, 10Gbps connections will be carried over the core infrastructure.
Connections of less than 10Gbps may be carried over the multiservice switching infrastructure.

In anticipation of the potential addition of the other services mentioned below, which will require
separate BGP peerings, Internet2 is encouraging connectors to connect initially with VLANs enabled
(in the case of Ethernet connections) or a frame relay Data Link Connection Identifier (DLCI) set up
(in the case of SONET connections).

A dedicated backup circuit to another router may be purchased, so long as the aggregate usage across
the two links remains below the connection fee allowance. The redundant network costs shown in the
price schedule include the cost of the router interface and transport to the alternate node. Other
redundancy options will be considered upon request. (More information is available on the Fee
Structure page at

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                12
               1.4.2 Commercial Peering Service

Internet2 is creating a content peering exchange fabric as a benefit to its members. This service will be
available to connectors that choose to participate. The service consists of peering with providers such
as Google, YouTube, or Microsoft. A beta trial is currently underway with several Internet2 connectors
to help understand the implications before deploying this across a national infrastructure. Early results
with a few peers show 20-30% traffic offload from existing commodity connections to this service.

As configured for the trial, the service is a separate peering between Internet2 and the participating
connectors, allowing the connectors to control how the peering service is distributed to their members.
This service is made available over the connector’s access to the IP backbone network. However, to
allow connectors to better differentiate this service, the connection will multiplex via the use of
VLANs (in the case of Ethernet) or frame relay DLCI (in the case of SONET). Internet2 anticipates no
additional fee for this service. The Network Technical Advisory Committee (NTAC) and Internet2
staff will be seeking community input on how this service should be expanded to additional connectors
and rolled out to the entire Internet2 community. (The NTAC is an advisory body similar to the
Abilene TAC, and may be contacted by sending email to or by communicating
with the chair. The current NTAC chair is Paul Schopis of OARnet.) The NTAC has formed a
commodity and peering working group to address this issue.

The Internet2 IP backbone network will connect at 10Gbps to commercial colocation facilities in
Chicago, IL and Ashburn, VA, with additional locations contemplated in the near future.

               1.4.3 Commodity Transit Service

Internet2 intends to offer commodity transit service to connectors through their IP connection.
Internet2 has negotiated with Level(3) the ability to provide commodity Internet services to its
connectors and members at a reduced rate. To ensure that this agreement does not undermine the
significant progress the Quilt has made in this area, Level(3) has agreed that purchases by Internet2 for
its members will count towards the Quilt’s aggregate purchase levels. Level(3) will also allow some
existing 1GE ports that do not connect to the Abilene backbone, but provide commodity transit directly
to Internet2 connectors or members, to be transitioned to the new rate as direct connections from the
Level(3) backbone to the members or connectors. To ensure that Internet2 is not dependent on only
one supplier, we will engage other commodity transit providers in the near future, and add their
services to the overall service offering. (For details on current commodity transit pricing, see the Fee
Structure page at

Commodity transit is an optional service to Internet2 connectors, respecting the preference that
regional connectors are the primary access points to the network for community participants. Internet2
will consider providing commodity transit directly to a member institution whose connector declines
the service, or where there otherwise is agreement between a member and connector. Internet2 will
continue to be sensitive to the wide variety of business models among the regional networks, and will
support the option to use this direct connection service in coordination with and in support of the

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                13
connectors’ and regional network operators’ desire to serve their members. The commodity transit
service is also planned for discussion by the NTAC commodity and peering working group (see last

Internet2 provides the optional commodity transit service as a separate BGP peering, allowing the
connectors to control distribution to their members. As with the commercial peering service, we
anticipate that, in general, this service will be made available over the IP backbone connection, and
that, in such cases, it will require the use of VLANs (for an Ethernet connection) or frame relay DLCI
(for a SONET connection).

               1.4.4 MPLS Services

Internet2 network staff will work with connectors to implement MPLS tunnels through the Internet2
Network on a case-by-case basis. In the past, Internet2 has created MPLS tunnels for connectors to
fulfill a specific need. If you have a situation where you think an MPLS tunnel might be appropriate,
contact Internet2 network staff at

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                 14
2. Details of Multiservice Switching Infrastructure and Dynamic Circuit
       2.1     Multiservice Switching Infrastructure Architecture

The multiservice switching infrastructure provides fixed-bandwidth circuits across the Internet2
backbone, between multiservice switches at the edge of the network. The following section describes
the multiservice switch functionality as deployed in Internet2, and then shows the architecture of the
boxes’ interconnections with each other and with users.

               2.1.1 Multiservice Switch Description

Each multiservice switch has two or more core infrastructure circuits that connect the box to other
multiservice switches. The circuits that connect to other infrastructure boxes are all SONET, and each
SONET circuit consists of multiple OC-1 channels.

Each box also has one or more user interfaces. The user interfaces may be Ethernet or SONET. If the
connections are Ethernet, the box encapsulates the Ethernet frames into SONET streams using GFP.
The user SONET streams are divided into one or more OC-1 channels at the interface, using the
Virtual Concatenation (VCAT) capabilities of the box.

The Add/Drop Multiplexer (ADM) in the multiservice switch breaks out each OC-1 stream in a
SONET connection and inserts it into a different SONET connection.

Subrate circuits are carried in one or more OC-1 streams within each SONET connection. VCAT is
used to define which sets of OC-1 streams are parts of the same subrate circuit. To combine multiple
OC-1 streams into a subrate circuit, VCAT signaling is carried within the SONET stream. Figure 4
shows a multiservice switch.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                  15
                                                         Figure 4

               2.1.2 Interconnections between Boxes and Users

The multiservice switching infrastructure is used to provide circuits for both the static and dynamic
circuit services. The two types of user circuits use different core circuits to carry the user circuits
internally. This is to prevent any interference between static and dynamic services on the internal

Section 2.3 contains further discussion of how this service is used to interconnect with other networks
and provide connections between users on connected networks.

Figure 5 shows the architecture of the multiservice switch data plane infrastructure.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                   16
                                                Figure 5

Figure 5 shows a set of multiservice switches interconnected with SONET circuits to provide dynamic
circuits. User multiplexers are attached to each multiservice switch (note that multiple users could be
connected; to simplify the picture, only one is shown). Each user may multiplex multiple circuits to the
multiservice switch. Each user circuit is connected to a circuit that is made up of one or more segments
between multiservice switches.

Note that the bandwidth on a segment between any two multiservice switches is shared by all users of
that segment. Thus, the connection between boxes A and B is used for circuits between A and B, but
also for circuits between A and C where the circuit between A and C is made up of segments A-B and
B-C. Also note that the use of multiservice switches for static circuits is very similar to the dynamic
circuit setup, except that static connections between multiservice switches are only made when there
are static user circuits that need them to make a circuit.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                               17
               2.1.3 Multiservice Switch Control Plane

The setting up of dynamic circuits will be managed by control plane software that keeps track of
bandwidth, allows reservation of future bandwidth, authenticates users requesting bandwidth, and
reports on the status of the infrastructure as a whole, as well as on individual multiservice switches.

Figure 6 shows the separation of control plane and data plane capabilities. In the multiservice
switching infrastructure, control of the data plane is done using control infrastructure that is different
than the infrastructure that supports the circuits. The control plane is used to communicate between
control elements, and the data plane is the infrastructure that provides circuits between multiservice
switches. Control elements communicate with multiservice switches to effect control.

                                                  Figure 6

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                      18
In this model, a connection request is sent to the control plane, which then determines if and how the
circuit can be implemented. The control plane elements signal the multiservice switches to find the
state of each box and to set the boxes to make the circuit.

The control plane for the multiservice switching infrastructure is the Ciena software that manages the
Ciena Core Director devices. Additional software will be integrated with the Ciena software to provide
additional capabilities for dynamic circuit services, such as interoperation with other networks offering
dynamic circuits.

       2.2     Dynamic Circuit Service

Two major differences between static circuits and dynamic circuits are: 1) dynamic circuits assume a
fabric of interconnected devices that want to make periodic connections to each other, and 2) dynamic
circuits give users the ability to create services directly rather than having them initiated by network
engineers. These concepts were introduced in section 1.3.2 and are expanded in this section.

               2.2.1 Dynamic Circuit Details

To use the dynamic circuit service, a user must make a physical connection to the Ciena multiservice
switching infrastructure using an Ethernet or SONET client interface. The Ethernet interfaces may be
1GE or 10GE. SONET interfaces may be OC-48 or OC-192.

Over the physical interface, the user may carry a single circuit or multiplex multiple subcircuits. An
Ethernet interface multiplexes VLANs. A SONET interface multiplexes streams, where each stream is
made up of multiples of OC-1 streams.

The bandwidth of the physical interface and the bandwidth of the circuit provided across the network
do not need to be the same. Each incoming, possibly multiplexed, circuit is connected to a dynamic
circuit across Internet2. The physical interface to the Internet2 circuit has a specific bandwidth, and
this bandwidth sets the upper limit for the bandwidth of the circuit (or circuits) carried by this
interface. The dynamic circuit may be configured in increments of OC-1.

For Ethernet connections, the circuit provided to a remote Ethernet device is a point-to-point Ethernet
circuit. The circuit may be a tagged VLAN at one end of the circuit and untagged Ethernet at the other
end, or be the same at both ends.

Ethernet circuits are carried on Internet2 dynamic circuit services as GFP-encapsulated SONET. GFP
encapsulation and de-encapsulation of Ethernet frames is done at network ingress and egress.

When the Internet2 segment of the dynamic circuit connects with a segment from another network
using a SONET interface, the data carried between Internet2 and the other network is GFP-
encapsulated SONET. This means that the GFP encapsulation must be done at the termination of the

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                   19
SONET part of the circuit. A good example occurs when the circuit includes a segment from Internet2
and a segment from GÉANT, where the users at each end connect using Ethernet, but Internet2 and
GÉANT connect using SONET. In this case, the circuit between Internet2 and GÉANT is GFP-
encapsulated SONET.

Physical connections that use Ethernet must support 802.3x (flow control). Physical connections using
Ethernet VLANs must support 802.1q (VLAN).

For SONET connections, the circuits provided are point-to-point SONET circuits at multiples of OC-1
bandwidth. Incoming physical SONET circuits may be channelized or unchannelized.

As noted, the Ciena infrastructure connection across Internet2 is always SONET. Dynamic circuits'
SONET connections are always channelized. User SONET connections may be channelized or
unchannelized. If unchannelized, they are converted to channelized at the edges.

               2.2.2 Dynamic Circuit Control

The dynamic circuit service creates circuits using control plane software. In the initial deployment,
dynamic circuits are provisioned only by the control plane, and are requested by users interacting with
the Internet2 NOC. The NOC uses control plane software to implement the circuit. In future
deployments, software will be added to allow circuits to be created at user request, if the request meets
the policy requirements of the networks providing the circuit segments. We expect that several
incremental deployments will create capabilities to make automated requests using a web form, then to
make requests directly from applications, then to check policy but leave final approval to the NOC.
Other enhancements will be made to support the ability to interact with dynamic circuit capabilities
from other domains.

We expect that research on, and initial deployment of, much of the control plane software will take
place on the HOPI test service, before the software is moved to the dynamic circuit service.

The ability to create dynamic circuits across domains using shared software is a topic of research in
many areas. Internet2 expects to bring its experience in deploying such networks to IETF and perhaps
other standards bodies.

       2.3     Connecting Internet2 Dynamic Circuit Services to Users

This section examines a number of ways that dynamic circuits may be connected to other users and
networks. The methods of interconnecting are undergoing intense investigation. To provide insight for
actual implementations, here are best guesses as to how this will happen.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                20
               2.3.1 Connecting Two Regional Networks and Their Users

The dynamic circuit service provides the ability for a regional network to connect to Internet2 and
provide dynamic circuit services across Intenet2 to a user on another regional network.

In this case, the regional network makes a physical connection to Internet2. In the most common case,
this will be an Ethernet connection that supports VLANs. The regional network on each end creates a
VLAN circuit to its user and then makes a VLAN connection to Internet2. The VLAN segments are
joined to create an end-to-end circuit between the users.

Figure 7 shows such a connection. The regional network provides its own circuit multiplexing
capability that takes circuits from its users and multiplexes them over its connection to Internet2. Each
multiplexed connection is sent by Internet2 on a separate SONET Virtual Container connection to a
multiplexed connection on a different RON.

                                                 Figure 7

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                21
Some notes about this setup. The physical connection between the regional network and Internet2 is
permanent. The dynamic circuits carried over the physical connection change over time, depending on
the requirements of the users. The dynamic circuits are set up and managed by control plane software,
and, if the regional network and Internet2 have compatible control plane software, the circuits may be
set up automatically across domains. Cross-domain control plane software is a heavily investigated
research topic at this time.

Internet2 is working with others to develop software that will allow regional networks to control
Ethernet switches and interface with the Internet2 control plane software. Internet2 is also participating
in research and standardization efforts to create control plane designs and specifications that will allow
interoperability of control plane software developed by different organizations.

                                                 Figure 8

               2.3.2 Connecting Internet2 with other National and
               International R&E Networks

Internet2 dynamic circuit services connect with similar services provided by other national and
international R&E networks, such as ESnet, GÉANT, and CANARIE. Each interconnection is a
physically permanent connection over which dynamic circuits are multiplexed. These interconnections
take place either directly with other networks on a one-to-one basis, or — as proposed by the Global
Lambda Integrated Facility (GLIF) — at exchange points where multiple networks come together and
circuits may be switched among any of the connected networks. Following GLIF, we refer to these
exchange points as Global Optical Lambda Exchanges (GOLEs).

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                 22
Using these interconnections and appropriate manual and automatic control of circuit switching,
Internet2 will be able to be a partner in creating circuits from users on regional networks in the United
States to users on networks connected to other core networks.

The data interface between core networks may be done using either Ethernet or SONET/SONET
interfaces. The control plane interface is logically very similar to the interface between Internet2 and
regional networks. At the present time, the biggest differences in control plane software interfaces
appear to be in defining how to authorize users requesting services, and how to share information
about reachability between networks. Internet2 is involved with groups working on ways to provide
and standardize these.

Two models of how networks could interconnect are described below.

A core provider dynamic circuit network architecture
Figure 9 shows a possible architecture of a set of networks that allow interoperation among
themselves. In this case, there is a set of core networks that are interconnected such that they can create
dynamic circuits among themselves, just as in the previous example.

                                                 Figure 9

Connected to each core network is a set of regional or provider networks. These regional networks
provide dynamic circuit service to organizations that have connections to them.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                    23
Interconnecting networks at GOLEs
Figure 10 shows the addition of exchange points, or GOLEs, to which networks connect with dynamic
circuits. By connecting to a GOLE, a network can create a circuit to any other network that also
connects to a GOLE.

                                               Figure 10

       2.4    Use Cases
              2.4.1 Applications

TeraGrid provides an integrated set of computing resources to users for a scheduled period of time.
Users connect to these resources during their scheduled time. Dynamic circuit services can be used to
create a high-performance circuit from the user to the TeraGrid infrastructure during each user’s
scheduled time.

High-performance videoconferencing
Dynamic circuits can be used to set up high-definition video conferences. This would be valuable for
scientists and researchers working on a joint project from widely separated locations. An application
currently being developed will permit groups in Michigan, Indiana, Maryland, and Virginia to create
dynamic video conferences. This application will use 2 x 1GE circuits to transmit data.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                 24
eVLBI (electronic Very Long Baseline Interferometry)
This is an application in which multiple radio telescopes in widely separated locations are used
simultaneously to provide sensitivity equal to that of a single gigantic (but impractical to build) radio
telescope. Dynamic circuits will be used to carry signals from many sources to a central processing
site. The radio telescopes may also be used sequentially by multiple cooperating processing sites.

Remote medicine
Dynamic circuits can be used to create temporary high-performance connections between hospitals at
various locations. A specialist at one location could create and use a connection to another location to
view real-time images from remote diagnostic equipment (e.g. MRI) or perform tele-surgery. High-
performance videoconferencing over dynamic circuits will be useful for medical diagnosis and

IP load shifting
Two sites that normally connect over IP networks may occasionally have the need to transfer very high
volumes of traffic, such that this would be disruptive to other users. In this case, the IP routers at each
site may be configured to create a direct path between the sites, using a temporary circuit directly
between the local routers.

File transfer
Dynamic high-bandwidth circuits can be used for file transfer. The protocols and timing for doing
these transfers are being investigated in a number of places, mostly using the IP network for transport.
The dynamic circuit service will eliminate congestion problems with high-speed networks, but not
other problems caused by high-speed long-distance transfer. Internet2 projects such as Phoebus and
VFER may use dynamic circuits as one way of improving performance.

               2.4.2 Models

The following are a few models describing how the dynamic circuit service may choose to allocate
resources. The actual methods for allocating resources are being worked out in the network as
applications come on line. These models are presented to give an idea of the different methods that are
possible when circuits can be allocated relatively quickly (in minutes, seconds, or less).

Note that where circuits consist of segments from multiple networks, all the networks must have an
interoperable method of allocating resources. This is a subject of current research.

Telephone model
The telephone model allocates resources at the time of request, on a first-come-first-served basis. If a
resource is not available, then the request is refused and the user tries again later. Hopefully the
resource will be available shortly, and within a small number of tries. This method depends on the
resource being used for short periods of time, and the rate of requests not being such that the requests
cannot be satisfied by existing resources.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                     25
We will use this model initially as we bring applications on line. We will supplement or replace this
model with other models as requirements become clear.

Doctor's office scheduling
In doctor's office scheduling, a user requests a resource at a specific time. A scheduling program
checks to see if the resource is available at this time, schedules an appointment if time is available, or
suggests an alternative time. When the appointment time arrives, the resource is likely to be available,
but there is a chance that the user may have to wait some (usually small) amount of time.

This type of scheduling is relatively simple and does not require aborting existing applications to meet
a timetable. Scheduling is done such that the waiting time for the resource is less than a specified time
for a specified percentage of the time.

Classroom scheduling
In this type of scheduling, a resource is allocated for a specific amount of time. Like a classroom, it is
used for a period and must be vacated after that period is complete, so that a new user may take over
the resource.

This type of scheduling is needed when other resources (e.g. supercomputing clusters) are also
scheduled for fixed amounts of time, or when some other resources need to be scheduled

Virtual organization
In this method of resource allocation, a set of resources is assigned to a virtual organization and the
resources are allocated by that organization. This lets an organization use a set of communication
resources as it sees fit, given the total set of resources, including computation and storage, that it has
available to solve a particular problem.

Multiple simultaneous circuits
This is a method of scheduling multiple circuits to be used simultaneously in an application. In some
cases, exactly when a circuit is available may not be as important as having a number of circuits
available at the same time. The eVLBI application, where multiple sources must be correlated
simultaneously, is a good example of such a scheduling need.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                      26
Appendix A. Dynamic Circuit Service — Detailed Description

        A.1    Connecting to the Dynamic Circuit Service

Connections to the dynamic circuit service are either SONET-based or Ethernet-based.

Type                           Ethernet                               SONET
Interface             1GE                10GE                OC-48          OC-192
                     1310ns             1310ns               1310ns          1310ns
                     1550ns             1550ns               1550ns          1550ns
Other                                  LAN/PHY             Channelized     Channelized
                    VLAN or            VLAN or
                    Untagged           Untagged

These interface types are supported at all locations where Ciena multiservice switch equipment is

        A.2    Transport Service Definition

The service infrastructure of the dynamic circuit service is transported using SONET framing and
incorporating GFP encapsulation for carrying Ethernet. Both VCAT and LCAS capabilities will be
implemented. The initial deployment of the service will use a set of waves on the infrastructure
dedicated to the dynamic circuit service, and separate from the set of waves used for the static circuit
service. Note that 10GE connections to the Ciena equipment will be carried by SONET and will be
restricted to SONET bandwidth limitations.

Service Requirements:
   • All SONET connectors must support VCAT and LCAS.
   • All SONET connectors providing Ethernet services must support GFP.
   • All Ethernet connectors must be capable of supporting 9K (MTU) payload frames.
   • Ethernet participants may be tagged with VLANs or untagged, and VLANs may be switched
       internally on the transport. That is, a VLAN tag on one end need not be the same as a VLAN
       tag on the far end.
   • Transport bandwidth will be allocated in increments of OC-1 channels in SONET or OC-3
       channels, as needed. Path setup must specify bandwidth.

It is strongly recommended that all Ethernet connectors support IEEE 802.1p (flow control).

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                    27
       A.3     Encapsulating Ethernet in SONET

When a circuit attaching to the dynamic circuit service is carried over an Ethernet physical interface,
the Ethernet frames are encapsulated using GFP and carried over SONET. When the circuit is
converted back from SONET to Ethernet, Ethernet frames are de-encapsulated from SONET back to

When connecting to Internet2, signaling on a SONET circuit will indicate whether or not it carries
GFP-encapsulated Ethernet.

       A.4     Control Plane Overview

A.4.1 Introduction

The Internet2 Dynamic Circuit Services (DCS) described earlier in this document will be realized via
the implementation of a DCS Control Plane. This control plane is being derived from technologies
developed as part of the NSF-funded Dynamic Resource Allocation via GMPLS Optical Networks
(DRAGON) project with extensions to interoperate with the DCS network elements.

In addition, it is anticipated that technology from the OSCARS (On-demand Secure Circuits and
Advance Reservation System) and BRUW (Bandwidth Reservation for User Work) projects will be
incorporated into the overall Internet2 DCS control plane to address issues associated with multi-
domain AAA (Authentication, Authorization, and Accounting).

It should be noted, that while this appendix discusses the Internet2 DCS specifically, the HOPI control
plane and associated services are very similar. The one major difference is that HOPI only offers
Ethernet services, while DCS provides for both Ethernet and SONET services.

This appendix provides an overview of the Internet2 DCS control plane from both a user interaction
and architecture perspective. The information presented in this appendix represents the current state of
the architecture definition and development for this control plane. It is expected that various features
and details will change and evolve over the next several months as part of the ongoing development
and implementation activities. The main objective of this appendix is to communicate the basic DCS
environment, architecture, and user environment such that initial planning activities can occur.

A.4.2 Internet2 DCS Services

This section provides an overview of the “services” that a client can request from the DCS. In this
context “client” is a very broad term. It generally means any device that finds itself on the edge of the
DCS and desiring dynamic services. This could include a host, a computational cluster, a router, a
radio telescope, a regional network, or any networked device. Therefore, a “client” is any system
requesting network services.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                   28
The base service provided is a point-to-point circuit connecting two clients attached at different points
on the network. The physical attachment point can be via either Ethernet or SONET. The subsequent
circuit provision can be requested in increments of approximately 50Mbps, up to the physical speed of
the attached port. Below is a summary of the Internet2 DCS from client perspective:

               •   Physical Connection
                       o 1 or 10 Gigabit Ethernet
                       o OC-48 or OC-192 SONET
               •   Circuit Service Type
                       o Point-to-point Ethernet Framed SONET Circuit
                       o Point-to-point SONET Circuit
               •   Clients Request Options
                       o Client must specify SRC Address, DST Address, Bandwidth, VLAN
                           Requirements [VLAN ID | ANY ID | Untagged], Identification Information
                       o Request mechanism options are via Web Services, GMPLS Style UNI,
                           Application Specific Topology Builder (allows for instantiation of multiple
                           individual circuits via one request)

                        Figure A. 1 Internet2 DCS Client "Service" Perspective

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                29
Figure A.1 depicts a high level view of a client service request and instantiation flow. Item 1 in this
figure represents the client making a request to the Domain Controller (DC) for an Ethernet Service.
This client request includes the following parameters:

               •   User Identification: Certificate information to allow user authentication
               •   Source Address: Near end of the requested circuit
               •   Destination Address: Remote end of the requested circuit
               •   Bandwidth: The requested bandwidth for this circuit
               •   VLAN Tag: User VLAN requirements
               •   Schedule: Time when this service is desired

The Internet2 DCS DC is responsible for authenticating the user and evaluating the user request in the
context of domain policies. If the request is approved, the DC must forward request-specific
information to the appropriate network element attachment point such that service instantiation can be
accomplished. This is depicted in item 2 of Figure A.1. Items 1 and 2 in Figure A.1 represent the
process of a client service request, evaluation, and approval. This is distinct from the process of
instantiating a service which is represented by items A and B in Figure A.1. With a scheduled service,
these two activities might also have a significant time separation as well. The use of message flow
identification scheme 1,2 for request/approval processing and A,B for service instantiation
(“signaling”) is intended to reflect the uncoupled nature of these events.

The request phase could happen well in advance of the signaling phase. As an example, there may be a
standing policy which allows for a user to attempt signaling at its own convenience up to a specified
total bandwidth utilization. As a result, the request and authorization processing can occur once for
multiple subsequent signaling instances. In this mode, signaling is the only process that needs to occur
at instantiation time.

It should be noted that the details of establishing domain policies and applying these to specific user
requests is still under development. In addition, the ability to schedule and reserve services for future
time slots is also under development. Additional details regarding control plane AAA and scheduling
features are presented in the subsequent control plane architecture section of this appendix.

The CSA (Client System Agent) element in Figure A.1 is software that runs on (or on behalf of) any
system that terminates the data plane link of the provisioned service. The CSA includes the capabilities
for both Web Service and GMPLS style UNI interactions as required to accomplish the request and
service instantiation processing. In addition, the CSA can be run in a “proxy” configuration where the
CSA is located on a machine that is physically separate from the element where the data plane link is

The CSA may also interact with the Application Specific Topology Builder (ASTB) if a more
complicated topology is to be built. The ASTB is a stand-alone process that accepts requests for
multiple circuits which an application domain desires to be set up as a group. The ASTB utilizes the
capabilities of the control plane to request the instantiation of the individual circuits and maintains the
mapping of individual service groupings to specific topologies on behalf of the user. In this scenario,
the CSA would interact with the ASTB, and the ASTB would interact with the DC.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                     30
The above is a high-level overview of service instantiation from a user perspective. The example
shown was for a single-domain case. The future control plane will also allow for similar service
instantiation across multiple networks and domains. From a user perspective, very little should change
for the multi-domain service provisioning as compared to the single domain case. However, additional
control plane functions are required. Details on the control plane architecture are provided in the
following sections.

A.4.3 Internet2 DCS Control Plane Architecture

The Internet2 DCS control plane is being developed via extension of the DRAGON control plane. The
DRAGON architecture utilizes Generalized Multiprotocol Label Switching (GMPLS) as the basic
building block for network element control and provisioning. There are several key functional
elements identified in the control plane architecture:

               •   Virtual Label Switch Router (VLSR)
               •   Network Aware Resource Broker (NARB)
               •   Client System Agent (CSA)
               •   Application Specific Topology Builder (ASTB)

These elements are being modified to enable the control of the DCS network elements, which include
the Ciena Core Director platform. In addition, the OSCARS (On-demand Secure Circuits and Advance
Reservation System) and BRUW (Bandwidth Reservation for User Work) systems are being evaluated
as the basis for the addition of multi-domain AAA features to augment the basic network provisioning
and control capabilities of the DRAGON control plane.

A more detailed description of the control plane architecture and these components is presented in the
sections that follow.

Virtual Label Switch Router (VLSR)
The VLSR provides a mechanism to integrate non-GMPLS equipment and network regions into the
end-to-end GMPLS provisioned services. The VLSR translates standard GMPLS protocols into
device-specific protocols, to allow dynamic reconfiguration of non-GMPLS aware devices. The
combination of a PC that runs the GMPLS based control plane software and a switch fabric is referred
to as a VLSR. The VLSR PC consists of a control plane stack that includes OSPF-TE and RSVP-TE
and acts as a proxy agent for the non-GMPLS capable devices. This allows non-GMPLS devices to be
included in end-to-end path instantiations.

The primary use for the VLSR on the DRAGON project is to control Ethernet switches via the
GMPLS control plane. Our application of the VLSR to Ethernet environments has included
modifications to OSPF-TE and RSVP-TE to allow the provision of Ethernet circuits based on VLAN
configurations. At each Ethernet switch, the VLSR translates RSVP-TE signaling messages into local
switch commands and creates the desired VLAN-ports associations along with the requested
bandwidth guarantees. The VLSR PC uses a combination of RFC 2674 SNMP and Command Line
Interface (CLI) commands for local control of the Ethernet switch fabric.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                              31
                           Figure A. 2 Single Domain Service Provisioning

For the Internet2 DCS control plane, the VLSR is being adapted to interact with the Ciena Core
Directors via the standard OIF UNI 1.0 interface. In this manner, an end-to-end path can be
instantiated across the Internet2 DCS Core Director region. Figure A.2 depicts this configuration
where the client service model described in section A.4.1 is applied to the Internet2 DCS network.
While this figure shows one VLSR per Core Director, this is actually a configuration decision. A single
VLSR can take responsibility for one or more Core Directors. The exact configuration will be
determined as part of testing and evaluation of VLSR loading conditions. The VLSR does rely on the
standard UNI 1.0 interface to instantiate paths across the Core Director domain. The main
responsibilities of the VLSR in this configuration are to process value added reservation and signaling
information, such as that associated with AAA, and to provide a seamless end-to-end provisioning
environment for clients or regional networks connected to the DCS edges.

Network Aware Resource Broker (NARB)
The NARB is an entity that represents the local Autonomous System (AS) or domain. The NARB
serves as a path-computation engine that other devices can query to find out about availability of traffic
engineered paths between specified source and destination pairs. The NARB is also responsible for
inter-domain routing. NARBs peer across domains and exchange topology information to enable inter-
domain path computation and Label Switched Path (LSP) provisioning. This inter-domain topology
exchange can be based on the actual topology as discovered by listening to the local OSPF-TE
protocol, or optionally based on an “abstracted” view of the domain topology (generated by
configuration file, automatic synthesis, an OSPF link state database, or other source of topology
information). Domain abstraction provides mechanisms for an administrative domain to advertise to
the outside world a highly simplified view of its topology. This allows domains to hide their real

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                32
topologies as well as minimize the amount of external updates required. The trade-off is reduced
accuracy for path computations. Each administrative domain can utilize configuration parameters to
tailor its domain abstraction to the level desired.

                           Figure A. 3 Multi-Domain Service Provisioning

For the Internet2 DCS control plane, the NARB functionality will be augmented with AAA features to
allow single domain and multi-domain user authentication and policy application to service requests. It
is anticipated that technology from the OSCARS (On-demand Secure Circuits and Advance
Reservation System) and BRUW (Bandwidth Reservation for User Work) projects will be adapted for
this purpose. However, this analysis is ongoing and final architectures in this area will be reported in
future documents and revisions. The combination of the NARB functionalities with the AAA features
will constitute the DC as represented in the figures in this appendix. Figure A.3 shows how a multi-
domain topology would include peering Domain Controllers for the purpose of extending a circuit
across multiple domain boundaries.

Client System Agent (CSA)
The CSA is software that runs on (or on behalf of) any system that terminates the data plane link of the
provisioned service. This is the software that participates in the Web Service and GMPLS protocols to
allow for on-demand end-to-end provisioning from client system to client system. In this context,
Client System is a very broad term. It generally means any device that finds itself on the edge of the
DCS dynamically-provisioned network. This could include a host, a computational cluster, a router, a
radio telescope, a regional network, or any other networked device. Therefore, a “client” is any system
requesting network services. The CSA may also interact with the ASTB if a more complicated
topology is to be built.

The control plane architecture identifies two distinct types of CSA modes of operation for service
instantiation (signaling):

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                               33
               •   UNI CSA
                   o UNI (RSVP based) Signaling
               •   WebService CSA
                   o XML Based Signaling

Web Services are always used for service requests (which happen prior to signaling). It should be
noted that this request phase could happen well in advance of the signaling phase. Depending on
policies, there may be a standing authorization that allows for a user to attempt signaling at its own
convenience. In this mode, signaling is the only process that needs to happen at instantiation time.

In addition any of these modes can be run in the “proxy” configuration where the CSA is located on a
proxy box that is physically separate from the CS where the data plane link is terminated.

Application Specific Topology Builder (ASTB)
The DRAGON architecture includes the notion of establishing Application Specific Topologies (AST).
These are requested by an end-user and are, generally, a set of circuits that an application domain
desires to be set up as a group. The Application Specific Topology Builder (ASTB) is responsible for
coordinating this in response to application requests. The ASTB utilizes the capabilities of the other
control plane elements (DC, VLSR, CSA) to request instantiation of the individual circuits, maintain
the mapping of individual circuit groupings to specific topologies, and interact with user requests.

A.4.4 Dynamic Service Provisioning with a Global Reach

Internet2 is not alone in its plans to develop and deploy technologies that allow for multi-domain
dynamic provision of dedicated resource paths. In fact, many other R&E network organizations in the
United States and around the world are also developing dynamic service provisioning capabilities
similar in requirements to the Internet2 DCS. A few of these efforts are listed below:

       •   ESNet Science Data Network (SDN) and the OSCARS project
       •   DANTE/GEANT JRA3 project
       •   Netherlands SURFnet and collaboration with Nortel on the DRAC project
       •   European Union PHOSPHORUS Project. This is a newly funded project that aims to
           demonstrate on-demand service delivery across access-independent multi-domain/multi-
           vendor research network testbeds on a European and worldwide scale. Their testbed will
           include GÉANT2, the EU NRENs SURFnet (Netherlands), CESNET2 (Czech Republic),
           and PIONIER (Poland) as well as the national test-beds VIOLA (Germany), OptiCAT
           (Spain/Catalonia), UKLight (United Kingdom), NetherLight (Netherlands), CzechLight
           (Czech Republic), and GLIF (International Consortium).
       •   The Canadian R&E network CANARIE and their User Controlled LightPath Provisioning
           (UCLP) project.
       •   G-Lambda project from Japan. Project members include: AIST (National Institute of
           Advanced Industrial Science and Technology), KDDI R&D Laboratories, and NTT
           Network Innovation Laboratories.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                  34
       •   Enlightened Computing project, which is collaboration between MCNC, Center For
           Computational Technology at Louisiana State University, North Carolina State University,
           Renaissance Computing Institute.

The larger goal for all these communities is to be able to provide these services on a truly global basis.
Much work is underway to develop the required inter-domain peering, provisioning, and AAA
architectures for this to occur. For Internet2, this includes current collaborations and/or discussions
with all the above projects and organizations. In addition, there are specific interoperation testing plans
between Internet2 DCS and ESNet, GÉANT2, CANARIE, and the University of

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                  35

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                  36
Appendix B. Bandwidth Used by Physical Interface
The bandwidth used by each circuit on a physical connection is assigned by the control plane. Each
physical connection may be set up to carry a single circuit on the backbone or may be multiplexed to
carry multiple connections. When it carries multiple connections, each connection to the network has a
different path in the network. Each path may go to a different egress multiservice switch on the

The control plane allocates bandwidth to each path. The maximum that can be allocated to all circuits
coming in on a physical circuit is limited only by the speed of the physical interface. The minimum
bandwidth of a circuit is OC-1.

Each physical interface may multiplex multiple circuits over the interface. 1GE and 10GE physical
connections may multiplex VLANs. SONET interfaces may multiplex SONET sub-channels. In both
cases, the bandwidth of the path allocated to the multiplexed circuits is allocated in increments of
OC-1 channels (approximately 50Mbps). For example, a 1GE connection may multiplex four VLANs:
one with a bandwidth of one channel, one with a bandwidth of three channels, and two with a
bandwidth of six channels each, with the remaining five channels available for additional VLANs.

Note that the bandwidth of the circuit across the backbone and the bandwidth of the interface to the
user may be different. It is up to applications to be able to handle this difference in bandwidth.

For example, a regional network may have a physical connection to Internet2 over which it multiplexes
a number of circuits for its users. One of these users connects to the regional network with a GE; the
regional network creates a VLAN to Internet2 to carry that circuit to a remote location. The VLAN is
allocated 250Mbps of bandwidth. In this case, an application that sends data at a regular rate less than
250Mbps will work well, but an application that sends data at a bursty rate, even if less than 250Mbps,
may not work well.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                37

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                  38
Appendix C. Ethernet Issues in the Dynamic Circuit Service
A number of performance issues with Ethernet buffering and flow control are being studied and tested.
There are two specific projects underway to test Ethernet performance: one testing performance of
point-to-point paths crossing multiple domains that include both Ethernet and SONET switching, and
another testing the ability the of the dynamic circuit service to handle bursty Ethernet traffic, where the
offered traffic may temporarily exceed the configured bandwidth of the network path that has been
provisioned to carry the traffic.

This Appendix will be expanded to include information about these tests and other information as it
becomes available over the next few months.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                  39

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                  40
Appendix D. Control Plane Development and Deployment Schedule
The following describes a tentative timeline regarding control plane features and implementation. Final
design work is ongoing and a revised schedule is planned for March 2007.

The initial deployments will include the ability to provision circuits across the Internet2 DCS Ciena
equipment region using the production Ciena Management Software. This capability will be available
from the time of initial deployment and will require coordination with the NOC. A web form or other
facilities will be provided to process such requests.

We anticipate testing initial versions of the automated control plane in the February 2007 timeframe.
Deployments and trials with early adopters at select regional networks will begin in April 2007. This
will involve a basic multi-domain dynamic provisioning capability with minimal AAA features and no
advanced scheduling capability. This early control plane deployment will allow for dynamic circuit
provisioning on a first-come-first-served basis. After completion of this early deployment testing,
automated control plane software will be made available to regional networks and others for general
use with the Internet2 DCS. This is anticipated for the May 2007 timeframe.

Advanced AAA and scheduling features will follow these initial deployment activities. More detailed
schedules on these features will be included in future documents and revisions.

In addition, Internet2 is working with a group of core networks (GÉANT, ESnet, and CANARIE) to
expand these automated services to have truly a national and global reach. Progress on this requires
developing consensus and agreements regarding an inter-domain peering, provisioning, and AAA
model. Discussions between these groups are underway, and a more detailed timeline regarding this
expanded capability should be available sometime in summer 2007.

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                41

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                  42
Appendix E. One Example of How to Connect a Campus to the
Dynamic Circuit Services

This appendix outlines a simplified process for connecting a campus infrastructure to the dynamic
circuit services portion of the Internet2 Network. It describes a step-by-step procedure for first
connecting to the Hybrid Optical and Packet Infrastructure (HOPI) and then evolving that connection
to the dynamic circuit services. Clearly, one could connect directly to the dynamic circuit services,
right from the beginning. See the Dynamic Services Description (
specifying these types of connections. Alternate connections are possible, but Ethernet is the simplest
way to begin.
Phase 1. Connect the campus infrastructure through the regional network to the HOPI switch using a
fixed set of VLANs. The campus can deploy an inexpensive 1GE or 10GE switch that can connect to
campus applications expected to use the dynamic services. The campus also can deploy a Virtual Label
Switch Router (VLSR) on a PC and connect it to the switch. The VLSR will participate in a
Generalized Multi-Protocol Layer Switching (GMPLS) control plane through the IP network that
parallels the data plane. The DRAGON software first can be deployed on the VLSR; then it can begin
participating in the HOPI control plane. At this point, the campus is connected to HOPI and
participating in the HOPI control plane administrative domain. Note: the regional network does not
have to set up the control plane capabilities to begin this process. The following illustration describes
the setup:

Phase 2. Add a Network Aware Resource Broker (NARB). This can be done on the same PC as above,
or it can be deployed on a separate PC. The function of the NARB is to separate the campus domain
from the HOPI domain. Once this is accomplished, the campus is running its own administrative
GMPLS control plane and it is peering with the HOPI administrative GMPLS control plane. The
picture below describes the setup:

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                43
Phase 3. When it is appropriate for the RON to implement its own GMPLS capabilities, they can do
so, implementing an infrastructure similar to the one that exists on the campus, as pictured in the
diagram below:

Phase 4. When campus, regional network, and dynamic circuit services are functional, Internet2
moves the connection of the regional network from the HOPI testbed to the Dynamic circuit services.
Note that the dynamic circuit services will be capable of carrying Ethernet traffic even thought it is a
SONET-based transport infrastructure. The following illustration describes the transport services:

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                44
Phase 5. The final phase is for the campus, regional network, and dynamic circuit services to expand
their respective infrastructures. For example, the regional network might move their service to a wave-
based grooming capability similar to the dynamic circuit services backbone and the campus might
expand its services by using SONET capabilities or by providing greater access to applications.

Once again, it should be emphasized that the first phase connection could be directly to the dynamic
circuit services. Moreover, there are a variety of alterations to the above connections. The above
example uses simple and inexpensive Ethernet switches that easily can be added to the campus
infrastructure and that are not part of the campus production environment. It assumes that the campus
is connected to a regional network that is not yet capable of connecting to the dynamic circuit services
It also assumes the use of Ethernet throughout, although campus and regional network engineers will
be able to use alternate equipment. It is required, however, that VLANs be capable of being set up
across the infrastructure. The campus must connect to the regional network equipment at either 1GE or
10GE, and the regional network must connect to the HOPI testbed at either 1GE or 10GE. For
questions or further clarification, contact

Internet2 Network Service Descriptions
v1.1 — February 7, 2007                                                                                45
Internet2 Network Service Descriptions
v1.1 — February 7, 2007                  46

Shared By: