Docstoc

johnston

Document Sample
johnston Powered By Docstoc
					                                      ESnet4:
                              Networking for the Future
                                  of DOE Science

                                             CEFNet Workshop, Sept. 19, 2007



                                                   William E. Johnston
                                            ESnet Department Head and Senior
                                                         Scientist

                                             Energy Sciences Network
                                       Lawrence Berkeley National Laboratory




                                       wej@es.net, www.es.net
                                       This talk is available at www.es.net




Networking for the Future of Science
                                                                               1
  DOE’s Office of Science: Enabling Large-Scale Science
• The Office of Science (SC) is the single largest supporter of basic
  research in the physical sciences in the United States, … providing
  more than 40 percent of total funding … for the Nation’s research
  programs in high-energy physics, nuclear physics, and fusion energy
  sciences. (http://www.science.doe.gov) – SC funds 25,000 PhDs and PostDocs
• A primary mission of SC’s National Labs is to build and operate very
  large scientific instruments - particle accelerators, synchrotron light
  sources, very large supercomputers - that generate massive amounts of
  data and involve very large, distributed collaborations
• ESnet - the Energy Sciences Network - is an SC program whose
  primary mission is to enable the large-scale science of the Office of
  Science that depends on:
   –   Sharing of massive amounts of data
   –   Supporting thousands of collaborators world-wide
   –   Distributed data processing
   –   Distributed data management
   –   Distributed simulation, visualization, and computational steering
   –   Collaboration with the US and International Research and Education
       community
• In addition to the National Labs, ESnet serves much of the rest of DOE,
  including NNSA – about 75,000-100,000 users in total
                                                                               2
                             What ESnet Is
• A large-scale IP network built on a national circuit
 infrastructure with high-speed connections to all major US and
 international research and education (R&E) networks
• An organization of 30 professionals structured for the service
  – The ESnet organization builds and operates the network

• An operating entity with an FY06 budget of $26.6M
• A tier 1 ISP (direct peerings will all major networks)
• The primary DOE network provider
  – Provides production Internet service to all of the major DOE Labs* and
    most other DOE sites
  – Based on DOE Lab populations, it is estimated that between 50,000 -
    100,000 users depend on ESnet for global Internet access
     • additionally, each year more than 18,000 non-DOE researchers from
       universities, other government agencies, and private industry use Office of
       Science facilities
                                           * PNNL supplements its ESnet service with commercial service
                                                                                                     3
                       ESnet History
ESnet0/MFENet    ESnet0/MFENet                  56 Kbps microwave and
mid-1970s-1986                                  satellite links
ESnet1           ESnet formed to serve the      56 Kbps, X.25 to
1986-1995        Office of Science              45 Mbps T3
ESnet2           Partnered with Sprint to       IP over 155 Mbps ATM
1995-2000        build the first national       net
                 footprint ATM network

ESnet3           Partnered with Qwest to        IP over 10Gbps SONET
2000-2007        build a national Packet over
                 SONET network and optical
                 channel Metropolitan Area
                 fiber
ESnet4           Partner with Internet2 and     IP and virtual circuits on
2007-2012        US Research& Education         a configurable optical
                 community to build a           infrastructure with at
                 dedicated national optical     least 5-6 optical
                 network                        channels of 10-100
                                                Gbps each                  4
                                                         2700 miles / 4300 km
     Japan (SINet)         KAREN/REANNZ         CA*net4                             SINet (Japan)                             GÉANT
                                                                MREN
     Australia (AARNet)    ODN Japan            France                              Russia (BINP)                             - France, Germany,
                                                                StarTap
     Canada (CA*net4       Telecom America      GLORIAD                                                  CERN                   Italy, UK, etc
                                                                Taiwan (TANet2,
     Taiwan (TANet2)       NLR-Packetnet         (Russia, China) ASCC)                                 (USLHCnet
     Singaren                                   Korea (Kreonet2                                     DOE+CERN funded)
                           Abilene/I2

                                                                                                                             NSF/IRNC
                                                                                                                              funded
                           LIGO
                   PNNL
  AU

                                                                                                                                           MIT
                                                Denver                                                             Lab DC
                                                                                                                                        BNL




                                                                                                                                                   1625 miles / 2545 km
                                                                                                                   Offices
                                                                                                                                 general
      JGI              LNOC
                       LLNL                                                                                                          PPPL
 LBNL                  SNLL
                                                                                                        NETL                 MAE-E
 NERSC
  SLAC                 PAIX-PA                                               Equinix                                            Equinix
                       Equinix           NREL                                                          OSC GTN
                       MAE-W                                                                                NNSA
                                                                              KCP
            NASA
            Ames




                                                                                            OSTI         ORAU
                              YUCCA MT
                                                                       ARM
                                                                                            NOAA
                                                  SNLA                                                                 SRS
AU                                                 Allied
                                                   Signal                                                                          AMPATH
                            AMPATH
                          (S. America)                                                                                           (S. America)




                                                                                                          International (high speed)
     ESnet Provides Global High-Speed Internet                                                            10 Gb/s SDN core
Connectivity for DOE Facilities and Collaborators (2007)                                                  10G/s IP core
                                                                                                          MAN rings (≥ 10 G/s)
     ~45 end sites: Office Of Science Sponsored       Other Sponsored                                     Lab supplied links
            commercial peering points       R&E
                                           network
                                                   Specific R&E network peers                             OC12 / GigEthernet
                    ESnet core hubs           s
                                                   Other R&E peering points                               OC3 (155 Mb/s)
                                                                                                          45 Mb/s and less
                   high-speed peering points with Internet2
        ESnet FY06 Budget is Approximately $26.6M
                               Approximate Budget Categories
                                                                Special projects    Target
                                                           (Chicago and LI MANs): carryover:
             SC R&D:             Carryover:                         $1.2M           $1.0M
              $0.5M                 $1M             Management and
SC Special
 Projects:                                         compliance: $0.7M
  $1.2M                Other                        Collaboration
                       DOE:                         services: $1.6
                       $3.8M

                                                        Internal
                                                 infrastructure,
                                              security, disaster
                                                                                          Circuits & hubs:
                                               recovery: $3.4M
                                                                                              $12.7M

                        SC operating:
                           $20.1M
                                               Operations:
                                                    $1.1M



                                                         Engineering &
                                                       research: $2.9M
                                                                                WAN
                                                                             equipment:
                                                                               $2.0M

                Total funds:                                             Total expenses:
                  $26.6M                                                      $26.6M
                                                                                                             6
A Changing Science Environment is the Key Driver of
            the Next Generation ESnet
• Large-scale collaborative science – big facilities, massive data,
 thousands of collaborators – is now a significant aspect of the
 Office of Science (―SC‖) program

• SC science community is almost equally split between Labs
 and universities
  – SC facilities have users worldwide

• Very large international (non-US) facilities (e.g. LHC and ITER)
 and international collaborators are now a key element of SC
 science

• Distributed systems for data analysis, simulations, instrument
 operation, etc., are essential and are now common (in fact
 dominate data analysis that now generates 50% of all ESnet
 traffic)                                                          7
       Science Networking Requirements Aggregation Summary
Science           End2End           Connectivity      Today      5 years          Traffic          Network Services
Drivers           Reliability                        End2End    End2End       Characteristics
                                                      Band        Band
Science Areas
                                                      width       width
/ Facilities

Magnetic           99.999%       • DOE sites          200+       1 Gbps      • Bulk data        • Guaranteed
Fusion Energy                                         Mbps                                       bandwidth
                  (Impossible    • US Universities                           • Remote control
                  without full
                                 • Industry                                                     • Guaranteed QoS
                 redundancy)
                                                                                                • Deadline scheduling
NERSC and              -         • DOE sites         10 Gbps     20 to 40    • Bulk data        • Guaranteed
ACLF                                                              Gbps                           bandwidth
                                 • US Universities                           • Remote control
                                 • International                             • Remote file      • Guaranteed QoS
                                 • Other ASCR                                 system sharing    • Deadline Scheduling
                                  supercomputers                                                • PKI / Grid
NLCF                   -         • DOE sites         Backbone   Backbone     • Bulk data
                                                       Band     band width
                                 • US Universities     width      parity
                                                                             • Remote file
                                                                              system sharing
                                 • Industry            parity
                                 • International
Nuclear                -         • DOE sites         12 Gbps     70 Gbps     • Bulk data        • Guaranteed
Physics (RHIC)                                                                                   bandwidth
                                 • US Universities
                                 • International                                                • PKI / Grid


Spallation           High        • DOE sites         640 Mbps    2 Gbps      • Bulk data
Neutron Source
                    (24x7
                  operation)
      Science Network Requirements Aggregation Summary
Science          End2End             Connectivity          Today         5 years              Traffic        Network Services
Drivers          Reliability                              End2End       End2End           Characteristics
                                                           Band        Band width
Science Areas
                                                           width
/ Facilities

Advanced              -        • DOE sites                1 TB/day      5 TB/day      • Bulk data      • Guaranteed
Light Source                                                                                             bandwidth
                               • US Universities          300 Mbps      1.5 Gbps      • Remote control
                               • Industry                                                              • PKI / Grid

Bioinformatics        -        • DOE sites                625 Mbps      250 Gbps      • Bulk data      • Guaranteed
                                                                                                         bandwidth
                               • US Universities            12.5                      • Remote control
                                                           Gbps in
                                                                                      • Point-to-      • High-speed
                                                          two years                                      multicast
                                                                                          multipoint
Chemistry /           -        • DOE sites                    -          10s of       • Bulk data           • Guaranteed
Combustion                                                            Gigabits per                           bandwidth
                               • US Universities                        second
                               • Industry                                                                   • PKI / Grid

Climate               -        • DOE sites                    -       5 PB per year   •   Bulk data         • Guaranteed
Science                                                                                                      bandwidth
                               • US Universities                         5 Gbps       •   Remote control
                               • International                                                              • PKI / Grid

                                      Immediate Requirements and Drivers
High Energy      99.95+%       • US Tier1 (FNAL, BNL)     10 Gbps     60 to 80 Gbps   • Bulk data           • Guaranteed
Physics (LHC)                                                                                                bandwidth
                  (Less        • US Tier2                             (30-40 Gbps     • Coupled data
                  than 4        (Universities)                        per US Tier1)       analysis          • Traffic isolation
                 hrs/year)                                                                processes
                               • International (Europe,                                                     • PKI / Grid
                                Canada)
The Next Level of Detail In Network Requirements
•   Consider the LHC networking as an example,
    because their requirements are fairly well
    characterized




                                                   10
LHC will be the largest scientific experiment and generate the most data
           the scientific community has ever tried to manage.
  The data management model involves a world-wide collection of data
 centers that store, manage, and analyze the data and that are integrated
through network connections with typical speeds in the 10+ Gbps range.




                                                                   closely
                                                              coordinated and
                                                              interdependent
                                                                 distributed
                                                                systems that
                                                                  must have
                                                                 predictable
                                                             intercommunicati
                                                               on for effective
                                  [ICFA SCIC]                    functioning
 The current, pre-production data movement sets the scale of the LHC
 distributed data analysis problem: The CMS experiment at LHC is
already moving 32 petabytes/yr among the Tier 1 and Tier 2 sites.

   Accumulated data (Terabytes) moved among the CMS Data Centers (“tier1” sites)
  and analysis centers (“tier2” sites) during the past four months (8 petabytes of data)




                                                                                 [LHC/CMS]
            Estimated Aggregate Link Loadings, 2007-08
                                                                                                              unlabeled links are 10 Gb/s
                                                                          9
                                                                                            12.5
                       Seattle
                                                                                                      13
                                                                                                                   13
                                                                                                                               9
            Portland
                                            Existing site
                                             supplied
                                                                                                                                                  2.5
                                 Boise        circuits                                                                                   Boston

                                                                                  Chicago                  Clev.
                                                                                                                                      NYC
Sunnyvale




                                                      Denver
                                  Salt                                    KC                                                       Philadelphia
                                  Lake                                                                       Pitts.
                                  City                                                                                             Wash DC


                                                                                                                          Raleigh             8.5
                  LA                                            Tulsa              Nashville
                                                       Albuq.                                         OC48
                                                                                             (1(3))
San Diego                                    (1)
                                                                                                                                     6
                                                                                            Atlanta                        6
                                                                                                                      Jacksonville
                                           El Paso
               ESnet IP switch/router hubs
                                                                                   Baton
                                                                        Houston
               ESnet IP switch only hubs                                           Rouge
                                                                                                      ESnet IP core (1)
               ESnet SDN switch hubs                                                                  ESnet Science Data Network core
                                                                                    2.5               ESnet SDN core, NLR links
               Layer 1 optical nodes at eventual ESnet Points of Presence                             Lab supplied link
                                                                                                      LHC related link
               Layer 1 optical nodes not currently in ESnet plans
                                                                                                      MAN link
               Lab site                  2.5    Committed bandwidth, Gb/s                             International IP Connections    13
            ESnet4 2007-8 Estimated Bandwidth Commitments
                                                               600 W. Chicago
                                                             West Chicago MAN                                               Long Island MAN
unlabeled links are 10 Gb/s                                                                                                            CERN
                                                                                                                                                          5
                              Seattle
                                                                                                                            USLHCNet
                  (28)                                                                                                                                   BNL
            Portland
                                                                                                                                                                    10
                       CERN                                                                                             32 AoA, NYC
                (29)
                                        Starlight
                                             Boise                                                                                                               13            Boston
                                                                                                                                                              (9)
                                                  (7)
                                                                           29
                                                                                                              Chicago                Clev.                                    (10)
                                                                         (total)                                             (11)
                                                                                                                                                                              NYC
Sunnyvale




                                 (32)                                                                     (13)                                                         (25)
                                                                        Denver
                USLHCNet                 Salt                                                      KC                                                                  Philadelphia
                                                                               10   (15)                                                                            (26)
                                         Lake                                                                                              Pitts.
                                         City
                                  San Francisco                                                                                                                        Wash DC
                                                                                                                     (21)
               (23)                                               (0)                       (22)                                                                (30)
                                  Bay Area MAN
                                                                                                                                                               Raleigh
                      LA                   FNAL
                                         JGI                                        Tulsa                      Nashville
                                                                                       ANL
                                                                        Albuq.                                                      OC48
                       (24)                                                                                                 (3)
                                                                                                                         (1(3))                    (4)
San Diego                                               LBNL(1)
                                                                                                                        Atlanta                          Newport News - Elite
                                        SLAC
                                           (19)
                                                         NERSC                             (20)                                              (2)

                                                                                                                                                     Jacksonville                    Wash.,
                                                          El Paso                                                                                                                     DC
                                                                        (17)
                                        LLNL                                                                                  (6)
               ESnet IP switch/router hubs                                                                                                               MATP
                                                                                                        (5)    Baton
                                       SNLL                                                   Houston              MAX
               ESnet IP switch only hubs                                                                       Rouge
                                                                                                                                    ESnet IP core             JLab
               ESnet SDN switch hubs                                      All circuits are 10Gb/s.                                                      ELITE
                                                                                                                                    ESnet Science Data Network core
                                                                                                                                    ESnet SDN core, NLR links (existing)
               Layer 1 optical nodes at eventual ESnet Points of Presence                                                           Lab suppliedODUlink
                                                                                                                                    LHC related link
               Layer 1 optical nodes not currently in ESnet plans
                                                                                                                                    MAN link
               Lab site                  2.5    Committed bandwidth, Gb/s                                                           International IP Connections      14
             Are These Estimates Realistic? YES!
Mbytes/sec




                    FNAL Outbound CMS Traffic
       Max= 1064 MBy/s (8.5 Gb/s), Average = 394 MBy/s (3.2 Gb/s)
                                                                    15
   Footprint of Largest Office of Science Labs Data Sharing
                         Collaborators
               Large-Scale Science is an International Endeavor




• Top 100 data flows generate 50% of all ESnet traffic (ESnet handles about 3x109 flows/mo.)
• 91 of the top 100 flows are from the Labs to other institutions (shown) (CY2005 data)
    Observed Evolution of Historical ESnet Traffic Patterns

3000                                                               ESnet total traffic
                                                                        passed
                                                                 2 Petabytes/mo about
           Terabytes / month
2500                                                                mid-April, 2007

                                   ESnet Traffic has Increased by
2000
                               10X Every 47 Months, on Average, Since
                                                1990
1500
                                                                    top 100
1000                                                              sites to site
                                                                   workflows
 500

   0
       Jan, 00




                                 Jan, 01




                                            Jan, 02




                                                       Jan, 03




                                                                   Jan, 04




                                                                                  Jan, 05




                                                                                            Jan, 06




                                                                                                                  Jan, 07
                                                                                                       site to
                                                                                                        site
                                                                                                      workflow
                                                                                                      data not
                                                                                                      available



         ESnet Monthly Accepted Traffic, January, 2000 – June, 2007
• ESnet is currently transporting more than1 petabyte (1000 terabytes) per
  month
• More than 50% of the traffic is now generated by the top 100 sites 
  large-scale science dominates all ESnet traffic
                                             ESnet Traffic has Increased by
                                      10X Every 47 Months, on Average, Since 1990

   10000.0                                                                                                                                                                           Apr., 2006
                                                                                                                                                                                     1 PBy/mo.
                                                                                                                                              Nov., 2001
                                                                                                                                             100 TBy/mo.
           1000.0
                                                                                                               Jul., 1998
                                                                                                              10 TBy/mo.
                    100.0
Terabytes / month




                                                                                                                                                                           53 months
                                                             Oct., 1993
                                                             1 TBy/mo.
                     10.0                                                                                                          40 months
                                   Aug., 1990
                                  100 MBy/mo.
                      1.0                                                                 57 months



                      0.1                        38 months



                      0.0
                            Jan, 90

                                       Jan, 91


                                                  Jan, 92

                                                            Jan, 93

                                                                      Jan, 94


                                                                                Jan, 95

                                                                                          Jan, 96


                                                                                                    Jan, 97

                                                                                                               Jan, 98


                                                                                                                         Jan, 99

                                                                                                                                   Jan, 00

                                                                                                                                             Jan, 01


                                                                                                                                                       Jan, 02

                                                                                                                                                                 Jan, 03


                                                                                                                                                                           Jan, 04

                                                                                                                                                                                        Jan, 05


                                                                                                                                                                                                  Jan, 06

                                                                                                                                                                                                            Jan, 07
                                      Log Plot of ESnet Monthly Accepted Traffic, January, 1990 – June, 2007
           Summary of All Requirements To-Date
Requirements from science case studies:
•   Build the ESnet core up to 100 Gb/s within 5 years
•   Provide high level of reliability
•   Deploy initial network to accommodate LHC collaborator footprint
•   Implement initial network to provide for LHC data path loadings
•   Provide the network as a service-oriented capability
Requirements from observing traffic growth and change trends in the network:
•   Most of ESnet science traffic has a source or sink outside of ESnet
     – Requires high-bandwidth peering
     – Reliability and bandwidth requirements demand that peering be redundant
     – Bandwidth and service guarantees must traverse R&E peerings
•   Provide 15 Gb/s core within four years and 150 Gb/s core within eight years
•   Provide a rich diversity and high bandwidth for R&E peerings
•   Economically accommodate a very large volume of circuit-like traffic
•   Large-scale science is now the dominant user of the network
     – Satisfying the demands of large-scale science traffic into the future will require
        a purpose-built, scalable architecture
Requirements from SC Programs:
•   Provide ―consulting‖ on system / application network tuning                         19
                   The Evolution of ESnet Architecture




    ESnet IP                   Metro Area
     core                        Rings

                                   ESnet IP
ESnet to 2005:                      core
• A routed IP network with
  sites singly attached to a
  national core ring
                                ESnet Science Data
                                Network (SDN) core   ESnet from 2006-07:
                                                     • A routed IP network with sites dually
     ESnet sites                                       connected on metro area rings or dually
     ESnet hubs / core network connection points       connected directly to core ring
     Metro area rings (MANs)                         • A switched network providing virtual
     Other IP networks                                 circuit services for data-intensive science
      Circuit connections to other
                                                     • Rich topology offsets the lack of dual,
      science networks (e.g. USLHCNet)                 independent national cores
                                                                                                     20
 ESnet Metropolitan Area Network Ring Architecture for High Reliability Sites
                                                    SDN
  US          US
                                                    core              ESnet production IP core hub                     IP core
LHCnet      LHCnet
switch      switch                                  east                  IP core                                        east
                              SDN                                                                  IP core
                                                                           west
                              core                                                                 router
SDN           SDN            switch
core          core
west         switch

                  ESnet SDN                                                    ESnet
                   core hub                                                    IP core hub

                       MAN fiber ring: 2-4 x 10 Gbps channels provisioned initially,
                                    with expansion capacity to 16-64
                                                                                                             ESnet managed
         Large Science Site                                                                               virtual circuit services
                                               ESnet managed                  ESnet production
                                                                                                          tunneled through the
                     ESnet MAN                λ / circuit services               IP service                    IP backbone
                       switch
                                  Independent
                                     port card
                                    supporting
                                multiple 10 Gb/s                                                         Site
                                 line interfaces                     ESnet switch



                                                                                             Virtual Circuits
                                                                  Virtual                         to Site
                        Site                                     Circuit to             Site gateway router
 SDN circuits          router                                      Site
to site systems                                    Site LAN          Site edge router                                            21
                               ESnet4
• Internet2 has partnered with Level 3 Communications Co.
    and Infinera Corp. for a dedicated optical fiber infrastructure
    with a national footprint and a rich topology - the ―Internet2
    Network‖
    – The fiber will be provisioned with Infinera Dense Wave Division
      Multiplexing equipment that uses an advanced, integrated optical-
      electrical design
    – Level 3 will maintain the fiber and the DWDM equipment

•   ESnet has partnered with Internet2 to:
    – Share the optical infrastructure, but build independent L2/3 networks
    – Develop new circuit-oriented network services
    – Explore mechanisms that could be used for the ESnet Network
      Operations Center (NOC) and the Internet2/Indiana University NOC to
      back each other up for disaster recovery purposes


                                                                              22
         L1 / Optical Overlay - Internet2 and ESnet Optical Node
                           ESnet                                                                          Internet2
                            IP
    ESnet                  core
  metro-area               M320                                          T640
  networks
                  SDN
                  core                                                            grooming
                 switch                                                      Ciena device
                               dynamically                             CoreDirector
support devices:              allocated and
•measurement
•out-of-band access           routed waves
•monitoring                       (future)
•security
                                                                                     optical
                                                                                    interface
                                                                                       to            support devices:
                                                                                                     •measurement
            Network Testbed                                                           R&E            •out-of-band access
                                                                                    Regional         •monitoring
           Implemented as a                                                                          •…….
                                                                                      Nets
            Optical Overlay
                                  BMM BMM




   various equipment and
    experimental control
    plane management
          systems



       fiber west                                                                            fiber east


                                            Infinera DTN            Internet2/Level3/Infinera
                                                                 National Optical Infrastructure
                                             fiber north/south                                                             23
Conceptual Architecture of the Infinera DTN Digital Terminal

                                                                              •   Band Mux Module (BMM)
                      intra- and                                                   – Multiplexes 100Gb/s bands onto
                      inter-DLM              DLM                                       400 Gb/s or 800 Gb/s line
                     switch fabric        10x10 Gb/s                          •   Digital Line Module (DLM)
         TOM
               TAM                                                                  – 100Gb/s DWDM line module (10  x
         TOM                                           100 Gb/s                          10Gb/s)




                                  OE/EO
        10 Gb/s                                                                     – Integrated digital switch enables
                                                                                         add/drop & grooming at 2.5Gb/s (ODU-1)
                                                                                         granularity
                                                                              •   Tributary Adapter Module (TAM)
                                                       100 Gb/s                     – Provides customer facing optical
                                                                                        interface




                                                                  BMM
                                                                                    – Supports variety of data rates and
                                                                        WAN             service types
                                                       100 Gb/s                     – Up to 5 TAMs per DLM
                                                                              •   Tributary Optical Module (TOM)
                                                                                    – Pluggable client side optical module
                                                                                        (XFP, SFP)
                                                       100 Gb/s                     – Supports 10GE, OC48&OC192 SONET
                                                                              •   Optical Supervisory Channel (OSC)
                                                         OSC                       – Management Plane Traffic—Includes
                                                                                        traffic from the remote management
                       Control                                                          systems to access network elements for
                      Processor                                                         the purpose of managing them
                                                                                   – Control Plane Traffic—GMPLS routing
                                                                                        and signaling control protocol traffic
                                                                                   – Datawire Traffic—Customer
                                                                                        management traffic by interconnecting
Notes:                                                                                  customer’s 10Mbps Ethernet LAN
1) This conceptual architecture is based on W. Johnston’s                               segments at various sites through AUX
                                                                                        port interfaces
    understanding of the Infinera DTN
2) All channels are full duplex, which is not made explicit here
3) The switch fabric operates within a DLM and also interconnects
    the DLMs
4) The Management Control Module is not shown
                                                                                                                            24
            ESnet 4 Backbone September 30, 2007




                                                                        Boston
               Boise
                                          Clev.




                            Kansas City
 Los Angeles




                               Houston    10 Gb/s SDN core (NLR)
                                          10/2.5 Gb/s IP core (QWEST)
Future ESnet Hub                          10 Gb/s IP core (Level3)
ESnet Hub                                 10 Gb/s SDN core (Level3)
                                          MAN rings (≥ 10 G/s)
                                          Lab supplied links
                                                                             25
           Typical ESnet4 Wide Area Network Hub
                                           To                                                                        To AoA,
                                        Cleveland                                                                     NYC




                                                SD




                                                                                                          SD
                                                                                                                                         Washington, DC




                                                                                                                           10 G
                                                                                                             N
                                                   N




                                                                                                         10
                                                     10




                                                                                    10




                                                                                                                               E to
                                                                                                    GE
                                                                                                                 WDC L(3)




                                                                                      GE
                                                       GE
                                                                                                                  3 racks




                                                                                               toA
                                                                                          to




                                                                                                                         AOA
                                                          to




                                                                                           CL
                                                                                                  OA V
                                                             C




                                                                                                   E
                                                               LE
                                                                 V
                                                                                                                                                         DClabs
                                                                                                                                               2xT1        LLNL-DC
                   NGIX-E                                                                                                                         2xT1
                                                                                                                                                              ORAUDC
                                                                                                                                                         T1
To GEANT   10 GE    6509              10                   WDC-SDN1
                                   MAX GE
                    MAX                L a mb
                                                da                                         10 GE
                                                                                                                                      1GE SX   WDC-AR1
                   MAX NGIX-E                                        7609                  10 GE              WDC-CR1                          7206VXR
                   Coillege Park
                                                                                                                  M320
                                                           E
                                                          G                                                                      1GE LX? GE to Eqx-ASH
                                                     10
                                                                                                                            OC3c             OC3 SM to DOE
                                      MATP                                                                                DS3               T3 to DOE-RT1
                                                                                                          1GE       DS3                  T3 to NGA
                                      7609                                                                 SX
                                                               L
                                                             AT




                                       E                                                                          WDC-PR1
                                    LIT                                                                              M7i
                                                          to




                                   E
                                                     GE




                                                                                      L
                                                 10




                                                                                    AT                                            MAE-E DS3
                                                                               to
                                               N




                         JLAB
                                            SD




                                                                           E
                                                                          G
                        Foundry                                      10


              To
            Atlanta                                                                                                                                                    26
                   Typical Medium Size ESnet4 Hub
   DC power                                            DC power
   controllers                                         controllers


                                                    local Ethernet


     10 Gbps
  network tester                                    secure terminal
                                                         server
                                                      (telephone
                                                    modem access)
 Cisco 7609,
 Science Data
Network switch                                       Juniper M7i,
(about $365K,                                         IP peering
     list)                                              router,
                                                     (about $60K,
                                                          list)

                                                     Juniper M320,
                                                     core IP router,
                                                    (about $1.65M,
                                                          list)
                                                                     27
             Note that the major ESnet sites are now directly on the ESnet
                                    “core” network
                                                          West Chicago MAN                                                 Long Island MAN
                                                                               600 W.
         e.g. the                                                                                                                      USLHCNet
               Seattle                                                        Chicago
   bandwidth into
         out
  and (28) of FNAL                                   Starlight                                                                     32 AoA, NYC
    is equal to, or
   Portland                         (>1 )
                                                                                                                                                                BNL
  greater, than5    the
       (29)                           Boise
      ESnet core                                                                                                                                                           Boston
                                           USLHCNet                                                                                                           (9)
                                                                                                                                                         5
      bandwidth
            4
                                             (7)                                                              Chicago        (11)        Clev.                             5 (10)
                                                                                                                                                                        NYC
                                                          5 FNAL
Sunnyvale




                           (32)                                                        ANL                (13)                                                      (25)
                                                                 Denver                                                           5
                                          Salt                                                    KC                                                                Philadelphia
                                                                                   (15)                                                                             5 (26)
                                          Lake                                                                                                   4
                                          City                                                                      (21)                                            Wash. DC
                     4                                                           5
             (23)           San Francisco                 4
                                                                                           (22)                     3                                          (30)
                                                                        (0)
                            Bay Area MAN
                                                                                                                                              5              Raleigh
               LA          4      JGI                                             Tulsa                       Nashville
                                                                     Albuq.                                                            OC48
                    (24)             4              4                                                                 (3) 3                     (4)
San Diego                                          LBNL        (1)                                 3
                                                                                                                         Atlanta
                                  SLAC
                                     (19)                                                 (20)                                                (2)
                                                    NERSC
                                                                                                                           4                         Jacksonville             Wash.,
                                       El
             ESnet IP switch/router hubs Paso                        (17)        4
                                                                               Atlanta      MAN                                                                                 DC
                                                                                                    ORNL                         (6)   ESnet IP core (1)
             ESnet IP switch only hubs
                                     LLNL                                                               (5)
                                                                                                      Baton
                                                                                                                                                    MATP
                                                                                                                                       ESnet Science Data Network core
                                                                                                   Nashville
                                                                                             Houston  Rouge
             ESnet SDN switch hubs SNLL                                       56 Marietta              Wash.,            DC            ESnet SDN core, NLR links (existing)
                                                                                                                                                                 JLab
                                                                                (SOX)                                                  Lab supplied link
             Layer 1 optical nodes at eventual ESnet Points of Presence                                                                                    ELITE
                                                                                                                                       LHC related link
             Layer 1 optical nodes not currently in ESnet plans                                           180 Peachtree                MAN link      ODU
                                                                                                                                       International IP Connections
             Lab site                                                                             Houston                     (20)     Internet2 circuit number
                   Aggregate Estimated Link Loadings, 2010-11
                                                                                             30
                          Seattle                                                                                                                                    45
                                                                                                                                      50
             (28)
       Portland                     (>1 )                                                                                                   15                                        20

            (29)              5        Boise                                                                                                                                Boston
                                                                                                                                                               (9)
                                                                                                                                                          5
                                               (7)                                                           Chicago                       Clev.
                     4                                                                                                     (11)                                             5 (10)
                                                                                                                                                                          NYC
Sunnyvale




                                                          5                                               (13)                                                      (25)
                             (32)                                                                                                5
                                                                 Denver
                                             Salt                                             KC                                                                     Philadelphia
                                                                               (15)                                                                                  5 (26)
                                             Lake                                                                                                  4
                                             City                                                                   (21)                                             Wash. DC
                    4
                                                          4
                                                                              5                                                                                                       10
            (23)                    5                                   (0)            (22)                         3                                           (30)

                                                                                                                                               5              Raleigh
              LA             4                                                Tulsa                           Nashville
                                                                     Albuq.                            5                              OC48
                   (24)                 4           4                                                                (3) 3                       (4)
                                                                                                                                                                            20
San Diego                                                      (1)                                3
                                                                                                                         Atlanta
 20                                     (19)                                          (20)                                                     (2)
                                                                                                           5               4                           Jacksonville
                                                     El Paso                  4
                                                                     (17)
                                                                                                                                (6)
            ESnet IP switch/router hubs
                                                                                                       (5)     Baton
            ESnet IP switch only hubs                  20                                Houston               Rouge
                                                                                                                                      ESnet IP core (1)
                                                                                                                                      ESnet Science Data Network core
            ESnet SDN switch hubs                                                                                                     ESnet SDN core, NLR links (existing)
                                                                                                                                      Lab supplied link
            Layer 1 optical nodes at eventual ESnet Points of Presence                                                                LHC related link
            Layer 1 optical nodes not currently in ESnet plans                                                                        MAN link
                                                                                                                                      International IP Connections
            Lab site                                                                                                        (20)      Internet2 circuit number          29
 ESnet4 2010-11 Estimated Bandwidth Commitments
                                                                600 W. Chicago
        CMS                                                                                    25                                          CERN
                                                                                                                                                          40
                          Seattle                                                                                                                                    20
                                                                                                                             USLHCNet
                                                                                                                                 25
             (28)                                                                                                                                         BNL
       Portland                       (>1 )                                                                                                   15
                   CERN                                                                                                    32 AoA, NYC
            (29)              5         Boise
                                    Starlight                                                                                                                    65           Boston
                                                                                                                                                               (9)
                                                                                                                                                          5
                                                (7)                                                            Chicago                    Clev.
                     4                                                                                                       (11)                                            5 (10)
                                                                                                                                                                          NYC
Sunnyvale




                                                           5                                                (13)                                                     (25)
                             (32)                                                                                                  5
                                                                   Denver
                                              Salt                                              KC                                                                    Philadelphia
             USLHCNet                         Lake
                                                                                 (15)                100
                                                                                                     80
                                                                                                     80                                                               5 (26)
                                                                                                                                                  4
                                              City                                                                    (21)                                            Wash. DC
                    4                                                          5
                                     5                     4
            (23)                                                                         (22)                         3                                          (30)
                                                                          (0)

                                                                                                                                               5               Raleigh
              LA             4                                                 Tulsa                            Nashville
                                         FNAL                          Albuq.                            5                              OC48
                   (24)                  4           4                      40 ANL                                     (3) 3                     (4)
                                                                                                                                                                             10
San Diego                                                        (1)                                3
                                                                                                                           Atlanta
 20                                      (19)                                           (20)                                                    (2)
                                                                                                             5               4                        Jacksonville
                                                      El Paso                   4
                                                                       (17)
                                                                                                                                  (6)
            ESnet IP switch/router hubs
                                                                                                         (5)     Baton
            ESnet IP switch only hubs                   10                                 Houston               Rouge
                                                                                                                                        ESnet IP core (1)
                                                                                                                                        ESnet Science Data Network core
            ESnet SDN switch hubs                                                                                                       ESnet SDN core, NLR links (existing)
                                                                                                                                        Lab supplied link
            Layer 1 optical nodes at eventual ESnet Points of Presence                                                                  LHC related link
            Layer 1 optical nodes not currently in ESnet plans                                                                          MAN link
                                                                                                                                        International IP Connections
            Lab site                                                                                                          (20)      Internet2 circuit number          30
                                  ESnet4 IP + SDN, 2011 Configuration

                          Seattle

             (28)
       Portland                     (>1 )


            (29)              5      Boise                                                                                                                          Boston
                                                                                                                                                        (9)
                                                                                                                                                   5
                                             (7)                                                        Chicago                    Clev.
                     4                                                                                                (11)                                          5 (10)
                                                                                                                                                                  NYC
Sunnyvale




                                                        5                                          (13)                                                      (25)
                             (32)                                                                                           5
                                                               Denver
                                          Salt                                              KC                                                                Philadelphia
                                                                             (15)                                                                             5 (26)
                                          Lake                                                                                             4
                                          City                                                                (21)                                            Wash. DC
                    4                                                      5
                                                        4
            (23)                                                                     (22)                     3                                          (30)
                                                                      (0)

                                                                                                                                        5              Raleigh
              LA             4                                              Tulsa                       Nashville
                                                                   Albuq.                                                        OC48
                   (24)              4            4                                                             (3) 3                     (4)
San Diego                                                    (1)                             3
                                                                                                                   Atlanta
                                      (19)                                          (20)                                                (2)
                                                                                                                     4                         Jacksonville
                                                   El Paso                  4
                                                                   (17)
            ESnet IP switch/router hubs                                                                                    (6)
                                                                                                  (5)    Baton                   ESnet IP core (1)
            ESnet IP switch only hubs                                                  Houston           Rouge
                                                                                                                                 ESnet Science Data Network core
            ESnet SDN switch hubs                                                                                                ESnet SDN core, NLR links (existing)
                                                                                                                                 Lab supplied link
            Layer 1 optical nodes at eventual ESnet Points of Presence                                                           LHC related link
            Layer 1 optical nodes not currently in ESnet plans                                                                   MAN link
                                                                                                                                 International IP Connections
            Lab site                                                                                                   (20)      Internet2 circuit number          31
                                                                        ESnet4 Planed Configuration
                                                              40-50 Gbps in 2009-2010, 160-400 Gbps in 2011-2012
                                                                                            Canada                            CERN (30 Gbps, each
                       Asia-Pacific                        Canada                          (CANARIE)
                                                           (CANARIE)            Asia Pacific                                           path)
                                                                                                   GLORIAD
                                                                                                   (Russia and                          Europe
                                                                                                     China)                             (GÉANT)

                                                                                Science Data
                                                                                Network Core
                                                                                                                                         Boston
1625 miles / 2545 km
                           Australia




                                                   Boise                    IP Core
                                                                                                                                        New York



                                                                       Denver
                                                                                                                                   Washington
                                                                                                                                    DC
                                                                                                       Tulsa
                   Australia




                                           LA                                        Albuquerque
                                                                                                                                   South America
                                       San Diego                                                                                       (AMPATH)

                                                       South America                                                    Jacksonville
                                                                           El Paso
                                                           (AMPATH)

                                IP core hubs
                                                                                                                 Production IP core (10Gbps) ◄
                                SDN (switch) hubs
                                                                                                                 SDN core (20-30-40Gbps) ◄
                                Primary DOE Labs
                                                                         Core network fiber path is              MANs (20-60 Gbps) or
                                High speed cross-connects
                                with Ineternet2/Abilene                 ~ 14,000 miles / 24,000 km               backbone loops for site access
                                 Possible hubs                                                                   International connections
                                                                                       2700 miles / 4300 km
          New Network Service: Virtual Circuits
• The control plane and user interface for ESnet version of light paths
                         Service Requirements
•   Guaranteed bandwidth service
    – User specified bandwidth - requested and managed in a Web Services
      framework
• Traffic isolation and traffic engineering
    – Provides for high-performance, non-standard transport mechanisms that
      cannot co-exist with commodity TCP-based transport
    – Enables the engineering of explicit paths to meet specific requirements
        • e.g. bypass congested links, using lower bandwidth, lower latency paths
• End-to-end (cross-domain) connections between Labs and collaborating
    institutions
•   Secure connections
    – Provides end-to-end connections between Labs and collaborator institutions
    – The circuits are ―secure‖ to the edges of the network (the site boundary)
      because they are managed by the control plane of the network which is
      isolated from the general traffic
• Reduced cost of handling high bandwidth data flows
    – Highly capable routers are not necessary when every packet goes to the
      same place
    – Use lower cost (factor of 5x) switches to relatively route the packets
                                                                                    33
                     The Mechanisms Underlying OSCARS
           Based on Source and Sink IP addresses, route of LSP between ESnet border routers is determined
         using topology information from OSPF-TE. Path of LSP can be explicitly directed to take SDN network.
          On the SDN Ethernet switches all traffic is MPLS switched (layer 2.5), which stitches together VLANs


                                  VLAN 1                    VLAN 2                         VLAN 3
 On ingress to ESnet,
   packets matching
reservation profile are           SDN                       SDN                            SDN
filtered out (i.e. policy
     based routing),
  policed to reserved                                                   RSVP, MPLS
    bandwidth, and                                                       enabled on                              Sink
  injected into a LSP.         Label Switched Path                   internal interfaces




 Source
                   IP Link             IP                     IP                            IP

   high-priority
         queue

      standard,
     best-effort
         queue
                                               MPLS labels are attached to packets from Source and
                                               placed in separate queue to ensure guaranteed bandwidth.

                    Interface queues           Regular production traffic queue.


                                                                                                                        34
 Conclusions from “Measurements On Hybrid
 Dedicated Bandwidth Connections,” Rao, et al
• Test scenario is ESnet-USN
   –   MPLS tunnels through routed IP network
   –   Ethernet over SONET
   –   hybrid 1Gbps connections
   –   Not tested: “native” Ehthernet
• Basic Conclusions
   – Transport Throughputs:
      • UDP : all modalities are very similar
      • TCP: SONET is slightly better for multiple streams ~6%
   – Jitter Measurements:
      • Ping, tcpmon, tcpcliser: all modalities are comparable
      • Differences are minor and likely only to effect “finer”
         control applications

   “Measurements On Hybrid Dedicated Bandwidth Connections,” Nagi Rao, Bill Wing – Oak Ridge
      National Laboratory, Qishi Wu – University of Memphis, Nasir Ghani – Tennessee Technological
      University, Tom Lehman – Information Science Institute East, Chin Guok, Eli Dart – ESnet,
      raons@ornl.gov. May 11, 2007, INFOCOM2007 Workshop on High Speed Networks, Anchorage,
      Alaska
                                                          References
1.        High Performance Network Planning Workshop, August 2002
      –        http://www.doecollaboratory.org/meetings/hpnpw
2.        Science Case Studies Update, 2006 (contact eli@es.net)
3.        DOE Science Networking Roadmap Meeting, June 2003
      –        http://www.es.net/hypertext/welcome/pr/Roadmap/index.html
4.        DOE Workshop on Ultra High-Speed Transport Protocols and Network Provisioning for Large-Scale Science
          Applications, April 2003
      –        http://www.csm.ornl.gov/ghpn/wk2003
5.        Science Case for Large Scale Simulation, June 2003
      –        http://www.pnl.gov/scales/
6.        Workshop on the Road Map for the Revitalization of High End Computing, June 2003
      –        http://www.cra.org/Activities/workshops/nitrd
      –        http://www.sc.doe.gov/ascr/20040510_hecrtf.pdf (public report)
7.        ASCR Strategic Planning Workshop, July 2003
      –        http://www.fp-mcs.anl.gov/ascr-july03spw
8.        Planning Workshops-Office of Science Data-Management Strategy, March & May 2004
      –        http://www-conf.slac.stanford.edu/dmw2004
9.        For more information contact Chin Guok (chin@es.net). Also see
      -        http://www.es.net/oscars
[LHC/CMS]
      http://cmsdoc.cern.ch/cms/aprom/phedex/prod/Activity::RatePlots?view=global
[ICFA SCIC] ―Networking for High Energy Physics.‖ International Committee for Future Accelerators (ICFA), Standing
       Committee on Inter-Regional Connectivity (SCIC), Professor Harvey Newman, Caltech, Chairperson.
      -        http://monalisa.caltech.edu:8080/Slides/ICFASCIC2007/
[E2EMON] Geant2 E2E Monitoring System –developed and operated by JRA4/WI3, with implementation done at DFN
      http://cnmdev.lrz-muenchen.de/e2e/html/G2_E2E_index.html
      http://cnmdev.lrz-muenchen.de/e2e/lhc/G2_E2E_index.html
[TrViz] ESnet PerfSONAR Traceroute Visualizer
       https://performance.es.net/cgi-bin/level0/perfsonar-trace.cgi


                                                                                                                     36

				
DOCUMENT INFO