; newman
Learning Center
Plans & pricing Sign in
Sign Out
Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>



  • pg 1
									   The Motivation, Architecture and
Demonstration of the UltraLight Network

                  Dan Nae
     California Institute of Technology
 UltraLight is an NSF funded collaboration of experimental
  physicists and network engineers whose purpose is to
  provide the network advances required to enable petabyte-
  scale analysis of globally distributed data.
 Current Grid-based infrastructures provide massive
  computing and storage resources, but are currently limited
  by their treatment of the network as an external, passive,
  and largely unmanaged resource. The goals of UltraLight
  are to:
    Develop and deploy prototype global services which
     broaden existing Grid computing systems by promoting
     the network as an actively managed component.
    Integrate and test UltraLight in Grid-based physics
     production and analysis systems currently under
     development in ATLAS and CMS.
    Engineer and operate a trans- and intercontinental
     optical network testbed
        The LHC Data Grid Hierarchy Concept:
        Refined in UltraLight

                   2.5 - 30 Gbps

Emerging Vision: A Richly Structured, Global Dynamic System
UltraLight Backbone

 The UltraLight testbed is a non-standard core network
  with dynamic links and varying bandwidth inter-
  connecting our nodes.
 The core of UltraLight is dynamically evolving as function
  of available resources on other backbones such as NLR,
  HOPI, Abilene and ESnet.
 The main resources for UltraLight:
    US LHCnet (IP, L2VPN, CCC)
    Abilene (IP, L2VPN)
    ESnet (IP, L2VPN)
    UltraScienceNet (L2)
    Cisco Research Wave (10 Gb Ethernet over NLR)
    NLR Layer 3 Service
    HOPI NLR waves (Ethernet; provisioned on demand)
    UltraLight nodes: Caltech, SLAC, FNAL, UF, UM,
     StarLight, CENIC PoP at LA, CERN, Seattle
UltraLight Points-of-Presence
 StarLight (Chicago)
        HOPI (2 x 10GE), USNet (2 x 10GE), NLR (4 x 10GE)
        UM (3 x 10GE), TeraGrid, ESnet, Abilene
        FNAL, US-LHCNet (2 x 10GE)
   MANLAN (New York)
        HOPI (2 x 10GE), US-LHCNet (2 x 10GE), BNL,
        Buffalo (2 x 10GE), Cornell, Nevis
   Seattle
        CENIC/PWave, GLORIAD, JGN2, NLR (2 x 10GE)
   CENIC (Los-Angeles)
        HOPI (2 x 10GE), NLR (4 x 10GE)
        Caltech (2 x 10GE), PWave
   Level3 (Sunnyvale)
        USNet (2 x 10GE), NLR, SLAC
UltraLight TestBed Overview

 UltraLight Network Engineering
 GOAL: Determine an effective mix of bandwidth-management
  techniques for this application-space, particularly:
       Best-effort and “scavenger” using “effective” protocols
       MPLS with QOS-enabled packet switching
       Dedicated paths provisioned with TL1 commands, GMPLS
 PLAN: Develop, Test the most cost-effective integrated
  combination of network technologies on our unique testbed:
   Exercise UltraLight applications on NLR, Abilene and campus
  networks, as well as LHCNet, and our international partners
   Deploy and systematically study ultrascale protocol stacks
  (such as FAST) addressing issues of performance & fairness
   Use MPLS/QoS and other forms of BW management, to
  optimize end-to-end performance among a set of virtualized disk
   Address “end-to-end” issues, including monitoring and end-
UltraLight: Effective Protocols

The protocols used to reliably move data are
a critical component of Physics “end-to-end”
use of the network
TCP is the most widely used protocol for
reliable data transport, but is becoming ever
more ineffective for higher and higher
bandwidth-delay networks.
UltraLight is exploring extensions to TCP
(HSTCP, Westwood+, HTCP, FAST, MaxNet)
designed to maintain fair-sharing of networks
and, at the same time, to allow efficient,
effective use of these networks.
FAST Protocol Comparisons
 Gigabit WAN                Random packet loss
  5x higher utilization     10x higher throughput
  Small delay               Resilient to random loss

           FAST: 95%


            Reno: 19%

 Optical Path Developments
Emerging “light path” technologies are arriving:
    They can extend and augment existing grid
        computing infrastructures, currently focused on
        CPU/storage, to include the network as an
        integral Grid component.
       Those technologies seem to be the most
        effective way to offer network resource
        provisioning on-demand between end-systems.
We are developing a multi-agent system for secure light path
provisioning based on dynamic discovery of the topology in
distributed networks (VINCI)
We are working to further develop this distributed agent system
and to provide integrated network services capable to efficiently
use and coordinate shared, hybrid networks and to improve the
performance and throughput for data intensive grid applications.
This includes services able to dynamically configure routers and
to aggregate local traffic on dynamically created optical
 GMPLS Optical Path Provisioning
 Collaboration efforts between UltraLight and
  Enlightened Computing.
 Interconnecting Calient switches across the US for the
  purpose of unified GMPLS control plane.
 Control Plane: IPv4 connectivity between site for
  control messages
 Data Plane:
    Cisco Research wave: between LA and Starlight
    EnLIGHTened wave: between StarLight and MCNC
    LONI wave: between Starlight and LSU Baton
     Rouge over LONI DWDM.
GMPLS Optical Path Network
  Monitoring for UltraLight
Realtime end-to-end Network monitoring is essential for
We need to understand our network infrastructure and
track its performance both historically and in real-time to
enable the network as a managed robust component of our
       Caltech’s MonALISA: http://monalisa.cern.ch
       SLAC’s IEPM: http://www-iepm.slac.stanford.edu/bw/
We have a new effort to push monitoring to the “ends” of
the network: the hosts involved in providing services or
user workstations.
MonALISA UltraLight Repository

  The UL repository: http://monalisa-ul.caltech.edu:8080/
      SC|05 Global Lambdas for Particle Physics
 We previewed the global-scale data analysis of the LHC Era
   Using a realistic mixture of streams:
     Organized transfer of multi-TB event datasets; plus
     Numerous smaller flows of physics data that absorb
          the remaining capacity
 We used Twenty Two [*] 10 Gbps waves to carry bidirectional traffic
   between Fermilab, Caltech, SLAC, BNL, CERN and other partner Grid
   sites including:
   Michigan, Florida, Manchester, Rio de Janeiro (UERJ) and Sao Paulo
   (UNESP) in Brazil, Korea (KNU), and Japan (KEK)
 The analysis software suites are based on the Grid-enabled UltraLight
   Analysis Environment (UAE) developed at Caltech and Florida, as well
   as the bbcp and Xrootd applications from SLAC, and dcache/SRM from
 Monitored by Caltech’s MonALISA global monitoring
     and control system
 [*] 15 at the Caltech/CACR Booth and 7 at the FNAL/SLAC Booth
         HEP at SC2005
Global Lambdas for Particle Physics

                                  Monitoring NLR,
                                  LHCNet, USNet,
                                  PWave, SCInet,
                                  Gloriad, JGN2,
                                  WHREN, other
                                  Int’l R&E Nets,
                                  and 14000+
                                  Grid Nodes at
                                  250 Sites (250k

                                       I. Legrand
 Global Lambdas for Particle Physics
 Caltech/CACR and FNAL/SLAC Booths

 151 Gbps peak, 100+ Gbps of throughput sustained
  for hours: 475 Terabytes of physics data
  transported in < 24 hours
    131 Gbps measured by SCInet BWC team on 17
      of our waves
    Sustained rate of 100+ Gbps translates to > 1
      Petayte per day
 Linux kernel optimized for TCP-based protocols,
  including Caltech’s FAST
 Surpassing our previous SC2004 BWC Record
    of 101 Gbps
Above 100 Gbps for Hours
475 TBytes Transported in < 24

Sustained Peak Projects to > 1 Petabyte Per Day
 It was the first time: a struggle
 for the equipment and the team

We will stabilize, package and more widely
 deploy these methods and tools in 2006
Thank You

To top