ATLAS Tier 2 Paths Within ESnet

Document Sample
ATLAS Tier 2 Paths Within ESnet Powered By Docstoc
					ATLAS Tier 2 Paths Within
         ESnet
             Mike O’Connor
     ESnet Network Engineering Group
      Lawrence Berkeley National Lab
              moc@es.net
  ATLAS Tier 2 Connectivity
• Typical ATLAS tier 2 site connectivity
  should be via Abilene and ESnet in order
  to reach Brookhaven Lab.
• “ESnet’s Goal is that DOE Lab ↔ Univ.
  connectivity should be as good as Lab ↔
  Lab and Univ. ↔ Univ.” W. E. Johnston
         ATLAS Tier 2 Support
• The role of ESnet is to provide networking that supports and
  anticipates the needs of the Office of Science Labs and their
  collaborators.

• ESnet support for ATLAS tier 2 site connectivity makes high
  bandwidth and fault tolerant paths between BNL and Abilene an
  absolute priority.

• Planning is underway to upgrade BNL’s ESnet OC48 circuit to a
  multi 10G lambda fault tolerant Metropolitan Area Network (MAN).
  (Targeted for Q1 2006)

• Today ESnet and Abilene peer in Atlanta, Chicago, New York and
  Sunnyvale with a combined bandwidth of 18.2Gbps.These four
  locations comprise a high bandwidth and fault tolerant interconnect
  between these two networks.
                                            ESnet Provides Global High-Speed Internet Connectivity for
                                           DOE Facilities and Collaborators (Spring, 2006 Configuration)
                                   Japan (SINet)                                                                       SINet (Japan)                   GÉANT
ESnet Science Data Network (SDN)




                                   Australia (AARNet)                       CA*net4              MREN                  Russia (BINP)                   - France, Germany,
                                   Canada (CA*net4                          France               Netherlands                                             Italy, UK, etc
                                   Taiwan (TANet2)                          GLORIAD              StarTap                   CERN
                                   Singaren                                  (Russia, China)     Taiwan (TANet2,         (USLHCnet
                                                                            Korea (Kreonet2       ASCC)               CERN+DOE funded)
              core




                                                           LIGO
                                                    PNNL
                                                                         ESnet IP core
                                                                                                                                                                       MIT
                                     AU
                                                 LVMR-                                                                                                                BNL
                                                  NOC
                                    JGI                                                                                     FNAL
                                                         LLNL                                                                              Lab DC
                                                                                                                                                                  PPPL
                         LBNL                            SNLL                                            AMES                ANL           Offices
                        NERSC                                                                                                                                 MAE-E
                          SLAC
                                                         PAIX-PA              NREL                                                       OSC GTN                Equinix
                                                         Equinix, etc.                                                                     NNSA
                                                                                                                KCP
                                                                                                                                                               JLAB
                                          NASA
                                          Ames




                                                                                                                                  ORNL
                                                              YUCCA MT                                                     OSTI          ORAU
                                                                            LANL
                                                                                                          ARM
                                                                                                                           NOAA
                                                                               SNLA                                                                  SRS
  AU                                                         GA                    Allied                                                                  ESnet IP core: Packet
                                                                                   Signal
                                                                                                                                                            over SONET Optical
                                                                                                                                                              Ring and Hubs
                                            South America
                                                West                                                                  South America       International (high speed)
                                                                                                                          East            10 Gb/s SDN core
       42 end user sites                                                                    international R&E peers                       10G/s IP core
          Office Of Science Sponsored (22)                                                                                                2.5 Gb/s IP core
                                                                                       commercial and R&E peering points                  MAN rings (≥ 10 G/s)
          NNSA Sponsored (12)
                                                                                                   ESnet core hubs                        OC12 ATM (622 Mb/s)
          Joint Sponsored (3)
                                                                                                                                          OC12 / GigEthernet
          Other Sponsored (NSF LIGO, NOAA)                                                   high-speed peering points                    OC3 (155 Mb/s)
          Laboratory Sponsored (6)                                                           with Internet2/Abilene                       45 Mb/s and less
ESnet & Abilene Peering points
                                ESnet AS 293




             Sunnyvale CA        Chicago IL    Atlanta GA          New York NY




                             Abilene AS 11537

                                              *paths do not include all router hops
   10 Gbps       2.4 Gbps   1 Gbps
                                              **Example site logos chosen at random.
Abilene Routing Toward ESnet
                               ESnet AS 293
Abilene Interior Gateway Protocol (IGP) mileage based
metrics determine the best exit path toward the ESnet.
This common method also known as “hot potato routing”
it will always compute an exit path to the nearest peering
point.




            Sunnyvale CA       Chicago IL         Atlanta GA   New York NY




                            Abilene AS 11537
  ESnet Routing Toward Abilene
                                  ESnet AS 293
Abilene Multi Exit Discriminators (MEDs) are used by ESnet
to choose the best exit path toward Abilene. These MEDs are
based on the same IGP metrics used in the interior of their
network. This creates a symmetric path back to Abilene.




                 Sunnyvale CA     Chicago IL        Atlanta GA   New York NY




                                Abilene AS 11537
32 Avenue Of the Americas NYC
                           ESnet Core
                  OC48
                           AOA (NYC)     OC192
                   DC                     CHI           ESnet
                                                       Peering
                                                       Router
            BNL                           2GE
                   OC48


                           10GE
                                  MANLAN


  Peerings@MANLAN                                          1GE
  nysernet 3754                                  Abilene
  abilene 11537
  sinet 2907
                                        10GE
  geant 20965



  The MANLAN peering is the primary path from ESnet to Abilene at
  AOA. The directly connected 1Gbps peering is a secondary
  failover connection.
                       Chicago
                               ESnet Core
                     OC192
                                Chicago       OC192
Peerings@Starlight    NY                       CA
ANL 683
ASCC 9264
CANET 6509
CERN 513
DREN 668
FNAL 3152                                         OC48
GLORIAD 5568                   10GE
KREONET 17579
MREN 22335
NASA 297
NWU 103
OPENTRANSIT 5511
SINET 2907
TANET 7539                        ESnet                    Abilene
UIUC 38                          Starlight
ULTRALIGHT 32361
USLHCNET 1297
UWMADISON 59           Abilene is the only external peer connected directly to
WISCNET 2381                     the ESnet Core router in Chicago.
Current Load in Chicago
                 ESnet Core
         OC192
                  Chicago     OC192
          NY                   CA




                                 OC48
                 10GE




                   ESnet                Abilene
                  Starlight




 1.9 Gbps peak                           0.8 Gbps peak
   19% Load                                33% Load
         Sunnyvale & Atlanta
         ESnet Core              El Paso             ESnet Core
OC192
        Sunnyvale CA                TX               Atlanta GA   OC48
 CHI                                                               DC
                       OC48                  OC48




         OC48                                        OC48




          Abilene                                      Abilene

         ESnet and Abilene have direct OC48 peerings in Atlanta
         GA and Sunnyvale CA.
      Future Directions

• Long Island MAN Ring
• ESnet Lambda Infrastructure
                    ESnet LI MAN Ring
                                                111th & 8th
• 2 λs (10 G circuits)
  delivered as two 10
  GigEther circuits
      -8 X current BNL                             λ2 SDN/circuit
      bandwidth
                                                                       Circuits based on
• Physically diverse fiber                             10 Gb/s            WDM rings
  paths forming a ring                                 optical
  topology.                                           channels
• Will be used as a10 Gb/s                        λ1 production IP
  production IP and a10 Gb/s                                                 BNL
  path for SDN circuit
                                                        Manhattan         Long Island
  services                        IP Core
                                to Chicago                MANLAN
• Services switch over to the                             Abilene
                                                          GEANT
  other service path during a
                                                              32 AOA
  circuit failure.
                                                       60 Hudson

                                IP Core to DC


*SDN – Science Data Network
 ESnet 2010 Lambda Infrastructure and LHC T0-T1 Networking




                                                                                                                                   CERN-1 CERN-2
             TRIUMF    Vancouver
                                                             CANARIE
                      Seattle

                                                                                         Toronto




                                                                                                                                                CERN-3
                                Boise
                                                                                                                             BNL
                                                                          Chicago                  Clev
                                                                                                                             New York
Sunnyvale




                                                Denver




                                                                                                                                                GÉANT-1
                                                                     KC
                                                                                               Pitts
                                                                                  FNAL
                                                                                                                      Wash DC
                                                                                                    Raleigh




                                                                                                                                                GÉANT-2
              LA                                Albuq.       Tulsa
                                Phoenix
     San Diego                                                                           Atlanta
                                                          Dallas


                                   El Paso -                                                                  Jacksonville
            NLR PoPs               Las Cruces                                          Pensacola
                                                                     Houston    Baton Rouge
             ESnet IP core hubs                          San Ant.
                                                                               ESnet Production IP core (10-20 Gbps)
             ESnet SDN/NLR hubs                                                ESnet Science Data Network core (10G/link)
             Tier 1 Centers                                                    (incremental upgrades, 2007-2010)
                                                                               Other NLR links
            Cross connects with Internet2/Abilene
                                                                               CERN/DOE supplied (10G/link)
             New hubs                                                          International IP connections (10G/link)
               Conclusions
• ESnet provides an infrastructure and services
  that are critical to ATLAS science
• ESnet is well positioned to provide essential
  network services to the LHC tier 2 community
• ESnet is working on providing the DOE
  mission science networking requirements with
  several new initiatives and a new architecture