Docstoc

johnston

Document Sample
johnston Powered By Docstoc
					                Esnet:
         DOE’s Science Network
                 GNEW March, 2004

William E. Johnston, ESnet Manager and Senior Scientist
            Michael S. Collins, Stan Kluz,
Joseph Burrescia, and James V. Gagliardi, ESnet Leads
                 and the ESnet Team
        Lawrence Berkeley National Laboratory



                                                          1
                        Esnet Provides
•   High bandwidth backbone and connections for 
    Office of Science Labs and programs
•   High bandwidth peering with the US, European, and 
    Japanese Research and Education networks
•   SecureNet (DOE classified R&D) as an overlay 
    network
•   Science services – Grid and collaboration services
•   User support: ESnet “owns” all network trouble 
    tickets (even from end users) until they are resolved
    o   one stop shopping for user network problems
    o   7x24 coverage
    o   Both network and science services problems
                                                            2
                  ESnet Connects DOE Facilities and Collaborators
 Australia                                      CA*net4                                   CA*net4                                        GEANT
 CA*net4                                        KDDI (Japan)                              CERN                                           - Germany
 Taiwan                                         France                                    MREN                                           - France
  (TANet2)                                      Switzerland                               Netherlands                                    - Italy
 Singaren                                       Taiwan                                    Russia                                         - UK
                                                 (TANet2)                                 StarTap                                        - etc
                                                                                          Taiwan                                         Sinet (Japan)
                                                                                           (ASCC)                                        Japan – Russia(BINP)
                  PN
                      W
                      G




         SEA HUB
                                 LIGO
                          PNNL
                                                         ESnet IP




                                                                                                                                                       e
                                                                                                                                                   ilen
                                                                                               Star




                                                                                                                                                 N
 Japan




                                                                                                                                                  YC
                                                                                                                                                  Ab
                                                                                                            ne
                                                                                                                                                            MIT




                                                                                                                                                    H
                                                                                                 light




                                                                                                                                                       U
                                                                                                            ile




                                                                                                                                                        B
     TW
         C                              INEE                                         Ch                                                                BNL
                                            L                                            i N




                                                                                                         Ab
                      SNLL                                                                                                    ANL-DC
                                                                                            AP
             Ab




     JGI                                                  QWEST                                               FNAL          INEEL-DC               NY-NAP
               ilen




                                                           ATM                                                              ORAU-DC                 PPPL
                              LLNL                                                AMES                                   LLNL/LANL-DC
                                                                                                                  ANL
 LBNL
                  e




                           SNV HUB                                                                CHI HUB                  4xLAB-DC            MAE-E
 NERSC
                             MAE-W                                                                                       GTN&NNSA                PAIX-E
  SLAC                                     EL
                                        HT
                             Fix-W B EC                                                  KCP
              W                                                                                                                DC HUB            JLAB
                                                           NRE




          IX-             YUCCA MT                                                                                ORNL
     PA           x                                                                                                           ORAU
              ini
                                                               L




                                                        LANL                                                 OSTI
         E uq                SDSC
                                                                                   ARM
                                                  ALB                                                        NOAA
                                                           SNLA                                                                         SRS
                                                  HUB
                                                             Allied
                                                                       PA
                                                                           NT
                                                                                                                    Abilene
42 end user sites      GA                                    Signal          EX                                          ATL HUB
   Office Of Science Sponsored (22)                        DOE-ALB
                                                                                                                              International (high speed)
   NNSA Sponsored (12)                                             H  UB                                                      OC192 (10G/s optical)
                                                               ELP
   Joint Sponsored (3)                                                                                                        OC48 (2.5 Gb/s optical)
                                                                                                                              Gigabit Ethernet (1 Gb/s)
   Other Sponsored (NSF LIGO, NOAA)                                                                                           OC12 ATM (622 Mb/s)
    Laboratory Sponsored (6)                                                                                                  OC12
                                                                                                                              OC3 (155 Mb/s)
        peering points                                                      ESnet backbone: Optical                           T3 (45 Mb/s)
SNV HUB   hubs                                                                  Ring and Hubs                                 T1-T3
                                                                                                                              T1 (1 Mb/s)                         3
              ESnet is Driven by the Needs of DOE Science
                                         August 13-15, 2002
                                         Organized by Office
                                         of Science
                                          Mary Anne Scott, Chair
                                          Dave Bader
                                          Steve Eckstrand
                                          Marvin Frazier
                                          Dale Koelling
                                          Vicky White
                                         Workshop Panel Chairs 
                                           Ray Bair and Deb Agarwal
                                           Bill Johnston and Mike Wilde
                                           Rick Stevens
                                           Ian Foster and Dennis Gannon
                                           Linda Winkler and Brian Tierney
                                           Sandy Merola and Charlie Catlett
                                       Focused on science
                                         requirements that drive
                                       • Advanced Network
                                         Infrastructure
                                       • Middleware Research
                                       • Network Research
Available at www.es.net/#research      • Network Governance Model
Eight Major DOE Science Areas Analyzed at the August ’02 Workshop
    Feature         Driven by                                                                  Requirements

              Vision for the Future            Characteristics that
              Process of Science             Motivate High Speed Nets             Networking               Middleware
Discipline


                                   • A few data repositories, many                                  • Server side data
            Analysis of model data distributed computing sites                                      processing (computing
                                                                             • Authenticated data
Climate by selected                                                          streams for easier     and cache embedded in
            communities that have • NCAR - 20 TBy                                                   the net)
(near term) high speed networking • NERSC - 40 TBy                           site access through
                                                                             firewalls              • Information servers for
            (e.g. NCAR and NERSC)
                                   • ORNL - 40 TBy                                                  global data catalogues

                                       • Add many simulation
                                       elements/components as
                                       understanding increases
Climate       Enable the analysis of
                                                                              • Robust access to    • Reliable data/file
              model data by all of the • 100 TBy / 100 yr generated
(5 yr)        collaborating            simulation data, 1-5 PBy / yr (just at large quantities of   transfer (across system /
                                       NCAR)                                  data                  network failures)
              community
                                          o Distribute large chunks of data
                                          to major users for post-
                                          simulation analysis

                                       • 5-10 PBy/yr (at NCAR)               • Robust networks
                                                                             supporting             • Quality of service
                                       • Add many diverse simulation         distributed            guarantees for
              Integrated climate       elements/components, including                               distributed, simulations
Climate                                                                      simulation -
              simulation that          from other disciplines - this must be
                                                                             adequate bandwidth     • Virtual data catalogues
(5+ yr)       includes all high-impact done with distributed,
                                                                             and latency for        and work planners for
              factors                  multidisciplinary simulation
                                                                             remote analysis and    reconstituting the data
                                       • Virtualized data to reduce storage visualization of        on demand
                                       load                                  massive datasets
                                                                                                                              5
 Evolving Qualitative Requirements for Network Infrastructure
              S                                                                               S                               C
                                          C
                                                                             guaranteed




                                                                   2-4 yrs
1-3 yrs



                  1-40 Gb/s,                  I
                                                                             bandwidth                                             I
                                                                             paths
                  end-to-end
          C                                                                               C
                                          C           S storage                                                           C
                          S                                                                                S
                                                   C compute
In the near term applications                         I
                                                      instrument   2-4 yrs requirement is for high bandwidth and QoS.
need high bandwidth                               C&C cache &
                                                      compute


                      S                           C                                       S                                C




                                                                                                               C&
                                                                                                      C
                                     C&




                                C
                              C&                                                                  C&
                                                                   4-7 yrs




                                                                                                               C
3-5 yrs




                                     C




                                                          I                                                                    I




                                                                                                                    C&C
                                          C&C




                                                                                              C&
                          C&




                                         C                                                                        &C




                                                                                                  C
                                      C&
                          C




                               C&C                                                    C               C&C       C
                  C
      100-200 Gb/s,                           C                                                                        C
                    S                                                                                  S
       end-to-end
3-5 yrs requirement is for high bandwidth                          4-7 yrs requirement is for high bandwidth and QoS
and QoS and network resident cache and                             and network resident cache and compute
compute elements.                                                  elements, and robust bandwidth (multiple paths)
                                                                                                                                       6
Evolving Quantitative Science Requirements for Networks
Science Areas      Today             5 years          5-10 Years        Remarks
                   End2End           End2End          End2End
                   Throughput        Throughput       Throughput
High Energy        0.5 Gb/s          100 Gb/s         1000 Gb/s         high bulk 
Physics                                                                 throughput


Climate (Data &    0.5 Gb/s          160-200 Gb/s     N x 1000 Gb/s     high bulk 
Computation)                                                            throughput


SNS NanoScience    Not yet started   1 Gb/s           1000 Gb/s +       remote control 
                                                      QoS for control   and time critical 
                                                      channel           throughput
Fusion Energy      0.066 Gb/s        0.198 Gb/s       N x 1000 Gb/s     time critical 
                   (500 MB/s         (500MB/                            throughput
                   burst)            20 sec. burst)
Astrophysics       0.013 Gb/s        N*N multicast    1000 Gb/s         computational 
                   (1 TBy/week)                                         steering and 
                                                                        collaborations
Genomics Data &    0.091 Gb/s        100s of users    1000 Gb/s +       high throughput 
Computation        (1 TBy/day)                        QoS for control   and steering
                                                      channel
                                                                                             7
  New Strategic Directions to Address Needs of DOE Science
                                      June 3-5, 2003
                                      Organized by the ESSC
                                      Workshop Chair
                                        Roy Whitney, JLAB
                                      Report Editors
                                        Roy Whitney, JLAB
                                        Larry Price, ANL

                                      Workshop Panel Chairs 
                                        Wu-chun Feng, LANL
                                        William Johnston, LBNL
                                        Nagi Rao, ORNL
                                        David Schissel, GA
                                        Vicky White, FNAL
                                        Dean Williams, LLNL

                                    Focused on what was needed
                                     to achieve the science driven
                                     network requirements of the
                                     previous workshop


                                    • Both Workshop reports are
Available at www.es.net/#research     available at es.net/#research
                     ESnet Strategic Directions
•   Developing a 5 yr. strategic plan for how to provide 
    the required capabilities identified by the workshops
    o   Between DOE Labs and their major collaborators in the 
        University community we must address
         - Scalable bandwidth
         - Reliability
         - Quality of Service
    o   Must address an appropriate set of Grid and human 
        collaboration supporting middleware services




                                                                 9
                  ESnet Connects DOE Facilities and Collaborators
 Australia                                      CA*net4                                   CA*net4                                        GEANT
 CA*net4                                        KDDI (Japan)                              CERN                                           - Germany
 Taiwan                                         France                                    MREN                                           - France
  (TANet2)                                      Switzerland                               Netherlands                                    - Italy
 Singaren                                       Taiwan                                    Russia                                         - UK
                                                 (TANet2)                                 StarTap                                        - etc
                                                                                          Taiwan                                         Sinet (Japan)
                                                                                           (ASCC)                                        Japan – Russia(BINP)
                  PN
                      W
                      G




         SEA HUB
                                 LIGO
                          PNNL
                                                         ESnet IP




                                                                                                                                                       e
                                                                                                                                                   ilen
                                                                                               Star




                                                                                                                                                 N
 Japan




                                                                                                                                                  YC
                                                                                                                                                  Ab
                                                                                                            ne
                                                                                                                                                            MIT




                                                                                                                                                    H
                                                                                                 light




                                                                                                                                                       U
                                                                                                            ile




                                                                                                                                                        B
     TW
         C                              INEE                                         Ch                                                                BNL
                                            L                                            i N




                                                                                                         Ab
                      SNLL                                                                                                    ANL-DC
                                                                                            AP
             Ab




     JGI                                                  QWEST                                               FNAL          INEEL-DC               NY-NAP
               ilen




                                                           ATM                                                              ORAU-DC                 PPPL
                              LLNL                                                AMES                                   LLNL/LANL-DC
                                                                                                                  ANL
 LBNL
                  e




                           SNV HUB                                                                CHI HUB                  4xLAB-DC            MAE-E
 NERSC
                             MAE-W                                                                                       GTN&NNSA                PAIX-E
  SLAC                                     EL
                                        HT
                             Fix-W B EC                                                  KCP
              W                                                                                                                DC HUB            JLAB
                                                           NRE




          IX-             YUCCA MT                                                                                ORNL
     PA           x                                                                                                           ORAU
              ini
                                                               L




                                                        LANL                                                 OSTI
         E uq                SDSC
                                                                                   ARM
                                                  ALB                                                        NOAA
                                                           SNLA                                                                         SRS
                                                  HUB
                                                             Allied
                                                                       PA
                                                                           NT
                                                                                                                    Abilene
42 end user sites      GA                                    Signal          EX                                          ATL HUB
   Office Of Science Sponsored (22)                        DOE-ALB
                                                                                                                              International (high speed)
   NNSA Sponsored (12)                                             H  UB                                                      OC192 (10G/s optical)
                                                               ELP
   Joint Sponsored (3)                                                                                                        OC48 (2.5 Gb/s optical)
                                                                                                                              Gigabit Ethernet (1 Gb/s)
   Other Sponsored (NSF LIGO, NOAA)                                                                                           OC12 ATM (622 Mb/s)
    Laboratory Sponsored (6)                                                                                                  OC12
                                                                                                                              OC3 (155 Mb/s)
        peering points                                                      ESnet backbone: Optical                           T3 (45 Mb/s)
SNV HUB   hubs                                                                  Ring and Hubs                                 T1-T3
                                                                                                                              T1 (1 Mb/s)                         10
          While ESnet Has One Backbone Provider, there are
           Many Local Loop Providers to Get to the Sites

                  SEA HUB
                                                                                                                                               Nevis




                                                                                                                                                               Br
                                                                                                                                                Yale




                                                                                                                                                               an
                          LIGO




                                                                                                                                                                 de
                PNNL




                                                                                                                                                                    is
                                                                                                                                                        N
                                                                                                                                                         Y
                                                                                                             ST




                                                                                                                                                     C
                                                                                                                                                      H
                                                                                                                                                                 MIT




                                                                                                              AR




                                                                                                                                                       U
                                                                                                                                                        B
                                                                                                                LI




                                                                                                                                                S
                                                                                                                  GH
                                      INEE         SNV                                        CH                                                               BNL
    SAN TW                                    L                                                    IN                                ANL-DC
               SNLL




                                                                                                                   T
           C                                                                                            AP                                                  NY-NAP
     JGI                                                                                                                FNAL       INEEL-DC
                                                             QWEST                                                                 ORAU-DC
LBNL/                                                         ATM                           AMES                                                            PPPL
                LLNL                                                                                                     ANL    LLNL/LANL-DC
CalRen2
                 SNV HUB                          SNV                                                        CHI HUB               4xLAB-DC            MAE-E
NERSC                                                                                                                                                   PAIX-E
                       Mae-W                                                          ELP                                             GTN
 SLAC                                       L
                                         TE             V
                      Fix-W             H          SN                                              Allied                        DOE-NNSA
                                 B   EC
                                                                    ELP


   PAIX-W                                                                                          Signal                              DC HUB           JLAB
                                                              NRE




                 YUCCA MT                                                                                                ORNL
                                                                                                                                     ORAU
                                                            LANL                                                       OSTI
                                                               L




          SDSC/CENIC                                                                         ARM
                          GA                        ALB                                                                NOAA
                                                    HUB
                                                               SNLA                                                                            SRS
                                                                Allied
                                                                             PA




                                                                Signal                                                               UB
                                                                              N




                                                                                                                                   LH
                                                                               TE




                                                               DOE-ALB                                                          AT
                                                                                  X




                                                                          UB
                                                                     ELP H
          Qwest Owned
          Qwest Contracted
          Touch America (bankrupt)
          MCI Contracted/Owned
          Site Contracted/Owned
          SBC(PacBell) Contracted/Owned
          FTS2000 Contracted/Owned
          SPRINT Contracted/Owned
          Level3
           ESnet Logical Infrastructure
Connects the DOE Community With its Collaborators




                                     Abilene


                      ESnet provides complete access to the 
                      Internet by managing the full complement 
                      of Global Internet routes (about 150,000) 
                      at 10 general/commercial peering points + 
                      high-speed peerings w/ Abilene and the 
                      international networks
                   ESnet Traffic




Annual growth in the past
five years has increased
from 1.7x annually to just
over 2.0x annually.




                                   13
         Who Generates Traffic, and Where Does it Go?
                      ESnet Inter-Sector Traffic Summary, Jan 2003
                          72%                          21%
                                                                                 Commercial
                DOE is a net supplier of                           14%
                data because DOE facilities      ESnet
                are used by Univ. and 
DOE sites       commercial, as well as by        ~25%              17%
                                                                                 R&E
                DOE researchers                                    10%
                             53%                                              Peering Points
                                                                   9%
                DOE collaborator traffic, inc.                                   International
                data                                               4%




ESnet Appropriate Use Policy (AUP)
All ESnet traffic must originate and/or terminate on 
an ESnet an site (no transit traffic is allowed)
                                                         ESnet Ingress Traffic = Green
E.g. a commercial site cannot exchange traffic with 
                                                         ESnet Egress Traffic = Blue
an international site across ESnet
                                                         Traffic between sites
This is effected via routing restrictions                % = of total ingress or egress traffic   14
                                ESnet Site Architecture

                                      New York (AOA)

      Chicago (CHI)
                                             Washington, DC (DC)

The Hubs have lots 
  of connections          Backbone              Atlanta (ATL)
     (42 in all)      (optical fiber ring)

Sunnyvale (SNV)
                                                               ESnet             Site 
                                                            responsibility   responsibility
                                El Paso (ELP)
          Hubs
    (backbone routers 
      and local loop                                            ESnet              Site         Site
    connection points)                                          border           gateway        LAN
                             Local loop                         router    DMZ     router
                          (Hub to local site)
                                                                                              Site




                                                                                                       15
                           SecureNet
•   SecureNet connects 10 NNSA (Defense Programs) 
    Labs
•   Essentially a VPN with special encrypters
    o   The NNSA sites exchange encrypted ATM traffic
    o   The data is unclassified when ESnet gets it because it is 
        encrypted before it leaves the NNSA sites with an NSA 
        certified encrypter

•   Runs over the ESnet core backbone as a layer 2 
    overlay – that is, the SecureNet encrypted ATM is 
    transported over ESnet’s Packet-Over-SONET 
    infrastructure by encapsulating the ATM in MPLS 
    using Juniper CCC
                                                                     16
                       SecureNet – Mid 2003


                   Backup 
                  SecureNet 
                    Path
                                                                        AOA-HUB

                                              CHI-HUB

                                                                   GTN
        SNV-HUB
 LLNL                                                                       DC-HUB
  SNLL
                                                        ORNL
                                               KCP

                              DOE-AL Pantex
                      LANL
                       SNLA                                       SRS
      Primary                                           ATL-HUB

     SecureNet 
       Path                  ELP-HUB




SecureNet encapsulates payload encrypted ATM in MPLS
using the Juniper Router Circuit Cross Connect (CCC) feature.                        17
                                    IPv6-ESnet Backbone
                                                                                                9peers
                  18 peers

                                                                                7206   StarLight         BNL
        6BONE         7peers
                                                                  7206
                             PAIX      Distributed 6TAP                                     Abilene
LBL
           7206
                                                                          Chicago
ESnet

      LBNL               Sunnyvale

                                7206     TWC                                                       New York
                                                           ANL           FNAL
SLAC
                                                                                       DC
                   Abilene
                                                  Albuquerque

           7206
                                                                   Atlanta         IPv6 only 
                                                                                   IPv4/IPv6
SLAC
                                              El Paso                              IPv4 only
• IPv6 is the next generation Internet protocol, and ESnet is working on addressing
  deployment issues
        -one big improvement is that while IPv4 has 32 bit – about 4x10 9 – addresses (which we
        are running short of), IPv6 has 132 bit – about 1040 – addresses (which we are not ever
        likely to run short of)
        -another big improvement is native support for encryption of data                                     18
    Operating Science Mission Critical Infrastructure
•   ESnet is a visible and critical pieces of DOE science 
    infrastructure
    o   if ESnet fails,10s of thousands of DOE and University users know it 
        within minutes if not seconds
•   Requires high reliability and high operational security in the 
    ESnet operational services – the systems that are integral 
    to the operation and management of the network
    o   Secure and redundant mail and Web systems are central to the 
        operation and security of ESnet
         - trouble tickets are by email
         - engineering communication by email
         - engineering database interface is via Web
    o   Secure network access to Hub equipment
    o   Backup secure telephony access to Hub equipment
    o   24x7 help desk (joint with NERSC)
    o   24x7 on-call network engineer

                                                                               19
               Disaster Recovery and Stability
•   The network operational services must be kept 
    available even if, e.g., the West coast is disabled by 
    a massive earthquake, etc.
    o   ESnet engineers in four locations across the country
    o   Full and partial engineering databases and network 
        operational service replicas in three locations
    o   Telephone modem backup access to all hub equipment
• All core network hubs are located in commercial 
  telecommunication facilities with high physical 
  security and backup power
                          Disaster Recovery and Stability
                SEA HUB     Engineers, 24x7 NOC,                   Remote Engineer
                              generator backed power               • partial duplicate 
                            • Spectrum (net mgmt                     infrastructure
                              system)
                            • DNS (name – IP address 
                              translation)                                                     DNS
                            • Eng database                                                                      BNL
                                                                     CHI HUB                                    NYC HUBS
                            • Load database                 AMES
                            • Config database                                                            PPPL
  LBNL                      • Public and private Web
                                                                                                DC HUB
              TWC           • E-mail (server and archive)
SNV HUB                     • PKI cert. repository and               Remote                      Duplicate Infrastructure
                              revocation lists                       Engineer                    (currently deploying full 
                            • collaboratory authorization                                        replication of the NOC 
Engineers                       ALB

Eng Srvr
              SDSC            service
                                HUB
                                                                                                 databases and servers 
Load Srvr                                                                            LH
                                                                                          UB     and Science Services 
                                                                                   AT
Config Srvr                                    UB                                                databases)
                                          ELP H




• ESnet backbone operated without interruption through
   • N. Calif. Power blackout of 2000
   • the 9/11 attacks
   • the Sept., 2003 NE States power blackout
    Maintaining Science Mission Critical Infrastructure
                in the Face of Cyberattack
•   A Phased Security Architecture is being implemented to 
    protect the network and the sites
•   The phased response ranges from blocking certain site traffic 
    to a complete isolation of the network which allows the sites 
    to continue communicating among themselves in the face of 
    the most virulent attacks
    o   Separates ESnet core routing functionality from external Internet 
        connections by means of a “peering” router that can have a policy 
        different from the core routers
    o   Provide a rate limited path to the external Internet that will insure site-
        to-site communication during an external denial of service attack
    o   provide “lifeline” connectivity for downloading of patches, exchange of 
        e-mail and viewing web pages (i.e.; e-mail, dns, http, https, ssh, etc.) 
        with the external Internet prior to full isolation of the network
                         Phased Response to Cyberattack
                                                                       ESnet third response – shut down the 
        ESnet first response –               ESnet second              main peering path and provide only a 
         filters to assist a site        response – filter traffic      limited bandwidth path for specific 
                                         from outside of ESnet                   “lifeline” services


                                                                                              peering
                                                                                               router
                                                     X                                   X
                                                                           router
                                                           ESnet
                                        router

 LBNL                                                                                                attack 
                                                              router                                 traffic
                                    X
                                    border
      Lab first 
response – filter 
                       gateway      router                   peering
 incoming traffic       router                                router
  at their ESnet 
                                                                                    border router
 gateway router
                           Lab
                                                                                                    gateway
Sapphire/Slammer worm infection created almost a Gb/s traffic                        Lab             router
spike on the ESnet backbone until filters were put in place (both 
into and out of sites) to damp it out.                                                                     23
        Future Directions – the 5 yr Network Strategy
•   Elements
    o   University connectivity
    o   Scalable and reliable site connectivity
    o   Provisioned circuits for hi-impact science bandwidth
    o   Close collaboration with the network R&D community
    o   Services supporting science (Grid middleware, 
        collaboration services, etc.)




                                                               24
               5 yr Strategy – Near Term Goal 1
•   Connectivity between any DOE Lab and any Major 
    University should be as good as ESnet connectivity 
    between DOE Labs and Abilene connectivity 
    between Universities
    o   Partnership with I2/Abilene
    o   Multiple high-speed peering points
    o   Routing tailored to take advantage of this
    o   Latency and bandwidth from DOE Lab to University should 
        be comparable to intra ESnet or intra Abilene
    o   Continuous monitoring infrastructure to verify




                                                               25
              5 yr Strategy – Near Term Goal 2
•   Connectivity between ESnet and R&D nets – a 
    critical issue from Roadmap
    o   UltraScienceNet and NLR for starters
    o   Reliable, high bandwidth cross-connects
        1) IWire ring between Qwest – ESnet Chicago hub and Starlight
           – This is also critical for DOE lab connectivity to the DOE funded 
             LHCNet 10 Gb/s link to CERN
           – Both LHC tier 1 sites in the US – Atlas and CMS – are at DOE Labs
        2) ESnet ring between Qwest – ESnet Sunnyvale hub and the Level 
           3 Sunnyvale hub that houses the West Coast POP for NLR and 
           UltraScienceNet




                                                                                 26
           5 yr Strategy – Near-Medium Term Goal
•   Scalable and reliable site connectivity
    o   Fiber / lambda ring based Metropolitan Area Networks
    o   Preliminary engineering study completed for San 
        Francisco Bay Area and Chicago area
         - Proposal submitted
         - At least one of these is very likely to be funded this year

•   Hi-impact science bandwidth – provisioned circuits




                                                                         27
                   ESnet Future Architecture
•   Migrate site local loops to ring structured 
    Metropolitan Area Network and regional nets in 
    some areas
    o   Goal is local rings, like the backbone, that provide multiple 
        paths
•   Dynamic provisioning of private “circuits” in the MAN 
    and through the backbone to provide “high impact 
    science” connections
    o   This should allow high bandwidth circuits to go around site 
        firewalls to connect specific systems. The circuits are 
        secure and end-to-end, so if the sites trust each other, 
        they should allow direct connections if they have
        compatible security policies. E.g. HPSS <-> HPSS
•   Partnership with DOE UltraNet, Internet 2 HOPI, and 
    National Lambda Rail                                             28
                         ESnet Future Architecture
          one optical fiber pair            site     provisioned circuits 
         DWDM providing point-                        initially via MPLS 
          to-point, unprotected                     paths, eventually via 
                 circuits                               lambda paths

 Layer 2 management 
   equipment (e.g. 
10 GigEthernet switch)


                                     Metropolitan                                       core
  site                                  Area                                            ring
                                      Networks

 Layer 3 (IP)
 management 
  equipment 
                                                        production IP
   (router)
                                                         provisioned circuits carried
                                                         over lambdas
              Optical channel (λ) 
                management                               provisioned circuits carried
                  equipment                              as tunnels through the ESnet
                                           site          IP backbone                           29
                            ESnet MAN Architecture - Example
    CERN                             StarLight                                                           Qwest
                                                    Current DMZs are
 (DOE funded                                       back-hauled to the                                     hub
     link)                                             core router
                                                   Implemented via 2                                    ESnet
    other                                            VLANs – one in                                      core
international                                        each direction
                                        Vendor
  peerings                                           around the ring             T3
                                                                                     20
                                        neutral
                                        facility

               ESnet managed
              λ / circuit services             Ethernet switch               ESnet
                                              • DMZ VLANs                management                 ESnet managed
                                              • Management              and monitoring                  λ / circuit
                       ESnet production         of provisioned            – partly to                   services
                          IP service            circuits                compensate for             tunneled through
                                                                         no site router             the IP backbone
                                                                                                       via MPLS

FNAL                                                                                              ANL
                                 monitor                     monitor




site equip.             Site gateway router                                               site equip.
                                                         Site gateway router
                             Site LAN                                     Site LAN                               30
                Future ESnet Architecture


                          circuit      ESnet             Site 
      MAN                 cross        border          gateway    Site     Specific host, 
                                                                            instrument, 
optical fiber ring       connect                DMZ
                                                        router    LAN           etc.



                                                        Site
                     New York (AOA)

                      Washington
  ESnet                Atlanta (ATL)                                       common 
 backbone                                Private “circuit” from one 
                                                                           security 
                                              Lab to another
                                                                            policy
                El Paso (ELP)




                          circuit      ESnet              Site         Site Specific host, 
      MAN                 cross        border           gateway        LAN instrument, 
                         connect                                                   etc.
optical fiber ring                              DMZ      router

                                                                   Site
                                                                                             31
                  Long-Term ESnet Connectivity Goal
         • MANs for scalable bandwidth and redundant site access to backbone
Japan
         • Connecting MANs with two backbones to ensure against hub failure
           (for example NLR is shown as the second backbone below)         Europe
                                                            CERN/Europe




 Japan




    MANs
    Local loops
    High-speed cross connects with Internet2/Abilene
                                                                          Qwest
    Major DOE Office of Science Sites                                     NLR 32
              Long-Term ESnet Bandwidth Goal
•   Harvey Newman:
    “And what about increasing the bandwidth in the 
    backbone?”
•   Answer: technology progress
    o   By 2008 (the next generation ESnet backbone) DWDM 
        technology will be 40 Gb/s per lambda
    o   And the backbone will be multiple lambdas

•   Issues
    o   End-to-End, end-to-end, and end-to-end




                                                             33
                        Science Services Strategy
•   ESnet is in a natural position to be the provider of choice for a number 
    of middleware services that support collaboration, colaboratories, Grids, 
    etc.
•   The characteristics of ESnet that make it a natural middleware provider 
    are that ESnet
     o   is the only computing related organization that serves all of the Office of 
         Science
     o   is trusted and well respected in the OSC community
     o   has the 7x24 infrastructure required to support critical services, and  is a long-
         term stable organization.
•   The characteristics of the services for which ESnet is the natural provider 
    are those that
     o   require long-term persistence of the service or the data associated with the 
         service
     o   require high availability, require a high degree of integrity on the part of the 
         provider
     o   are situated at the root of a hierarchy so that the service scales in the number 
         of people that it serves by adding nodes that are managed by local 
         organizations (so that ESnet does not have a large and constantly growing 
         direct user base).
                                                                                              34
                   Science Services Strategy
•   DOE Grids CA that provides X.509 identity 
    certificates to support Grid authentication provides 
    an example of this model
    o   the service requires a highly trusted provider, requires a 
        high degree of availability
    o   provides a centralized agent for negoiating trust 
        relationships with, e.g., European CAs
    o   it scales by adding site based or Virtual Organization 
        based Registration Agents that interact directly with the 
        users




                                                                      35
        Science Services: Public Key Infrastructure
•   Public Key Infrastructure supports cross-site, cross-
    organization, and international trust relationships that permit 
    sharing computing and data resources and other Grid 
    services
•   Digital identity certificates for people, hosts and services – 
    essential core service for Grid middleware
    o   provides formal and verified trust management – an essential service
        for widely distributed heterogeneous collaboration, e.g. in the 
        International High Energy Physics community
    o   DOE Grids CA
•   Have recently added a second CA with a policy that permits 
    bulk issuing of certificates with central private key mg’mt
    o   Important for secondary issuers
         - NERSC will auto issue certs when accounts are set up – this constitutes 
           an acceptable identity verification
         - May also be needed for security domain gateways such as
           Kerberos – X509 – e.g. KX509
                                                                                      36
        Science Services: Public Key Infrastructure
•   Policy Management Authority – negotiates and manages the 
    formal trust instrument (Certificate Policy - CP)
    o   Sets and interprets procedures that are carried out by ESnet
    o   Currently facing an important oversight situation involving potential 
        compromise of user X.509 cert private keys
         - Boys-from-Brazil style exploit => kbd sniffer on several systems that 
           housed Grid certs
         - Is there sufficient forensic information to say that the pvt keys were not 
           compromised??
             – Is any amount of forensic information sufficient to guarantee this, or should the 
               certs be revoked?
             – Policy refinement by experience 

•   Registration Agents (RAs) validate users against the CP and 
    authorize the CA to issue digital identity certs
•   This service was the basis of the first routine sharing of HEP
    computing resources between US and Europe
                                                                                               37
     Science Services: Public Key Infrastructure
•   The rapidly expanding customer base of this service will soon 
    make it ESnet’s largest collaboration service by customer 
    count




                                                                 38
         Voice, Video, and Data Collaboration Service

•   The other highly successful ESnet Science Service 
    is the audio, video, and data teleconferencing 
    service to support human collaboration
•   Seamless voice, video, and data teleconferencing is 
    important for geographically dispersed collaborators
    o   ESnet currently provides voice conferencing, 
        videoconferencing (H.320/ISDN scheduled, 
        H.323/IP ad-hoc), and data collaboration services 
        to more than a thousand DOE researchers 
        worldwide



                                                         39
    Voice, Video, and Data Collaboration Service
    o   Heavily used services, averaging around
         - 4600 port hours per month for H.320 videoconferences,
         - 2000 port hours per month for audio conferences
         - 1100 port hours per month for H.323
         - approximately 200 port hours per month for data 
           conferencing

•   Web-Based registration and scheduling for all of 
    these services
    o   authorizes users efficiently
    o   lets them schedule meetings
    Such an automated approach is essential for a scalable 
     service – ESnet staff could never handle all of the 
     reservations manually
                                                               40
                     Science Services Strategy
•   The Roadmap Workshop identified twelve high priority 
    middleware services, and several of these fit the criteria for 
    ESnet support. These include, for example
    o   long-term PKI key and proxy credential management (e.g. an 
        adaptation of the NSF’s MyProxy service)
    o   directory services that virtual organizations (VOs) can use to manage 
        organization membership, member attributes and privileges
    o   perhaps some form of authorization service
    o   in the future, some knowledge management services that have the 
        characteristics of an ESnet service are also likely to be important

•   ESnet is seeking the addition funding necessary to develop, 
    deploy, and support these types of middleware services.




                                                                                 41
                        Conclusions
•   ESnet is an infrastructure that is critical to DOE’s 
    science mission and that serves all of DOE 
•   Focused on the Office of Science Labs
•   ESnet is evolving its architecture and services 
    strategy to need the stated requirements for 
    bandwidth, reliability, QoS, and Grid and 
    collaboration supporting services




                                                            42

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:0
posted:11/24/2013
language:Unknown
pages:42