Network Services
LHC OPN Networking at BNL
Summer 2006 Internet 2 Joint Techs
John Bigrow
July 18, 2006
Brookhaven Science Associates
U.S. Department of Energy 1
Network Services
LHC Overview (very simple overview, I’m not a physicist)
• LHC / Atlas Experiments Overview (The What)
• The Physics Architecture (The Why)
• Preliminary Network and Security Architecture (The How)
Brookhaven Science Associates
U.S. Department of Energy 2
Network Services
CERN Accelerator Ring Aerial View
Brookhaven Science Associates
U.S. Department of Energy 3
Network Services
Brookhaven Science Associates
U.S. Department of Energy 4
Network Services
CERN:Outside Resource Ratio ~1:2
~PByte/sec Tier0:( Tier1):( Tier2) ~1:1:1
Online System 1 PB Disk
Tape Robot
~10 Gbits/sec
Tier 1 BNL: ~2M
IN2P3 Center RAL Center INFN Center SI2K; 2PB
Tape Robot
2.5 Gbps
Tier 2 Tier2 Tier2 Tier2
Tier2 Center Center Center Center
Tier2 Center
~2.5 Gbps
Tier 3
Tier 0: DAQ, reconstruction, archive
Institute Institute
Institute Institute
Tier 1: Reconstruction, simulation,
Physics data cache 100 - 1000
Mbits/sec
archive, mining and (large scale) analysis
Tier 2+: Analysis, simulation
Workstations
Tier 4 Tier 3+: Interactive analysis
Brookhaven Science Associates
U.S. Department of Energy 5
Network Services
The same host name for dual NIC
dCache door is resolved to different IP
addresses depending on which DNS is
inquired.
130.199.185.0
130.199.48.0
…
…
130.199.48.0
Brookhaven Science Associates
U.S. Department of Energy 6
Network Services
US ATLAS Tier 1 WAN Bandwidth Requirement Estimate
(Mbits/sec)
Year 2004 2005 2006 2007 2008 2009 2010
Remote Site(s)
Tier 0 (CERN) 52 105 349 874 1,747 1,747 3,494
Tier 1's (~2 Peer sites) 37 75 250 624 1,248 1,248 2,496
Tier 2's (5 USA satellite sites) 64 128 428 1,069 2,139 2,139 4,278
Tier 3-4 (150 Individual users) 95 190 632 1,581 3,161 3,161 6,322
Total 249 498 1,659 4,148 8,295 8,295 16,590
BNL HEP/NP WAN Bandwidth Requirement Estimate
(Mbits/sec)
Year 2004 2005 2006 2007 2008 2009 2010
US ATLAS Tier 1 Req. 249 498 1,244 4,148 8,295 9,954 16,590
RHIC Computing Facility Req. 200 500 1,023 1,286 1,847 2,422 3,381
TOTAL 449 998 2,267 5,433 10,142 12,377
λ
19,971
λ
2Xλ
2 x 3 x
BNL HEP/NP Requirement OC12 OC48 OC48 OC192
Brookhaven Science Associates
U.S. Department of Energy 7
Network Services
Brookhaven Science Associates
U.S. Department of Energy 8
GEANT, etc.
Network Services
MAN LAN
CERN (?)
ESnet
Other
NLR
BNL internal
connections
111 8th Hicksville Hauppauge
Adva
32 FSP3000
DWDM
AoA Adva Adva Diverse Route Adva Adva BNL
10GbE FSP3000 FSP2000 Protected DWDM FSP2000 FSP2000 10GbE
DWDM DWDM Core Ring DWDM DWDM
CWDM
ADVA
Diverse Route
Adva FSP2000
Diverse Route Protected DWDM or
FSP3000 DWDM
Protected DWDM Core Ring CWDM Access
DWDM CWDM
Ring
Adva
Adva Adva Diverse Route Adva FSP2000
10GbE FSP3000 FSP2000 Protected DWDM FSP2000 DWDM 10GbE
DWDM DWDM Core Ring DWDM CWDM
Adva
FSP3000
DWD
Garden City Brentwood
60 Hudson
Other
connections
Brookhaven Science Associates
U.S. Department of Energy 9
Network Services
BNL LHC OPN Conceptual Block Diagram
CIDR Restricted
Distribute List
ES Net Only
LHC OPN Private ACL
ES Net / General
Core Intranetwork
ACL Internet / Tier 2
BNL Internet /
LHC OPN T0-T1 Tier 2 Lambda
Lambda Layer 2
Tunnel ACL
20 Gb/Sec ACL NYSERNET /
Broadwing
Other Tier 1 Sites
BNL Border Router
Optional Dedicated
BNL Campus Network LHC OPN FWSMs
20 Gb/Sec
BNL LHC OPN
Primary Distribution
Switchs
ES Net Provisioned
CIDR IP Space
Future 10 Gb/Sec
Upgrades 1 Gb/Sec 1 Gb/Sec 1 Gb/Sec
d i g i t a l d i g i t a l d i g i t a l d i g i t a l
BNL LHC OPN Disk Cache / Storage / Analysis Facilities
Multi-homed
Brookhaven Science Associates
U.S. Department of Energy 10
Network Services
Network Security Limitations
• Current firewall Architecture
– 6 virtual 1 Gb/Sec EtherChannel to Catalyst backplane
– Rated total throughput of 5 Gb/Sec
– EtherChannel Overhead Loss
– Single 1 Gb/Sec flow / interface
• New Cisco ACE blade might address these limitations
Brookhaven Science Associates
U.S. Department of Energy 11
Network Services
Network Security Limitations (Continued)
• Current Router Architecture
– Single Access Control List (ACL) / interface
- 1 inbound and 1 outbound per interface
- Default behavior Implicit deny
- Policy route map for traffic flow
– A single ACL can become unwieldy in a complex WAN environment (what
are the network prefixes, DHCP, NAT)
– Manual changes to the route map for additional access
Brookhaven Science Associates
U.S. Department of Energy 12
Network Services
BNL LHC Overview cont.
• Networking resources
– IP Address space allocations / access
– 10Gig interfaces / 20Gig Etherchannels
– Performance Monitoring
Brookhaven Science Associates
U.S. Department of Energy 13
Network Services
IP Address Allocation Tier 0 to Tier 1 (BNL - CERN)
• Requires routable IP Address space
• Direct dedicated access with CERN to / from BNL
• Limited route advertisements between T0 and T1
– For the LHC OPN Circuit BNL will use 192.12.15.0/24
– No direct T1 to T1 access through CERN at this time
Brookhaven Science Associates
U.S. Department of Energy 14
Network Services
BNL OPN to Tier 2 and others
• Tier 2 and other traffic dependant on Internet connectivity
– Path to BNL via all service providers (ES Net now, NYSERNET,
Broadwing in the future ?)
– Dedicated paths to other institutions welcome (you buy)
Brookhaven Science Associates
U.S. Department of Energy 15
Network Services
Preliminary
BNL 10 /20 Gig-E LHC OPN
Initial Architecture 1 x 10G
1 x 10G
3 Peerings
Internet Peer with ES Net Direct Layer 2 Interface to CERN
Gateway ACL T0 - T1
Amon Mutt
SW9
Core
Shu Tefnut
BNL LHC OPN
Anubis Isis
Nephthys Osiris SW7
Brookhaven Science Associates
U.S. Department of Energy 16
Network Services
Future BNL LHC OPN Enhancements
• Dedicated Cisco Firewall Service Modules (ACE) when
available
– Eliminate router ACL Functionality / Maintenance
– Connection Logging
– Each FWSM circuit will not impede the 10 Gb/Sec.
– Stateful FWSM redundancy
• IDS / IPS when available
Brookhaven Science Associates
U.S. Department of Energy 17
Network Services
BNL Campus Network
Including Near-Term Upgrades
1 x 10G 1 x 10G
Internet Peer with ES Net Direct Layer 2 Interface to CERN
T0 - T1
FE
Stateful NYSERNET
Link Broadwing
Amon Mutt
SW9
Core
Shu Tefnut
BNL LHC OPN
DL2
Building Access
Layer Switch (Typical
DL1
Deployment) Failover Anubis Failover Isis
FWSM
Nephthys Osiris SW7
Brookhaven Science Associates
U.S. Department of Energy 18
Network Services
Mon
• browser-based IP service monitor
• Internet-centric WAN based monitor application
• Interrogates essential BNL network services
Brookhaven Science Associates
U.S. Department of Energy 19
Brookhaven Science Associates
U.S. Department of Energy 20
Network Services
MonaLisa
• Java based SNMP monitoring tool
External WAN based monitor
Tracks BNL 10G/Sec. Interfaces
Firewall Service Module
20 Gb/Sec. Uplinks to the BNL core
Brookhaven Science Associates
U.S. Department of Energy 21
Network Services
Brookhaven Science Associates
U.S. Department of Energy 22
Network Services
Brookhaven Science Associates
U.S. Department of Energy 23
Network Services
Cacti
• SNMP monitoring tool
• Replacement for MRTG
• Tracks most BNL core network interfaces
• Firewall Service Module EtherChannel interfaces also
Brookhaven Science Associates
U.S. Department of Energy 24
Network Services
Brookhaven Science Associates
U.S. Department of Energy 25
Network Services
Brookhaven Science Associates
U.S. Department of Energy 26
Network Services
Brookhaven Science Associates
U.S. Department of Energy 27
Network Services
Brookhaven Science Associates
U.S. Department of Energy 28
Network Services
Brookhaven Science Associates
U.S. Department of Energy 29
Network Services
Thanks (a few kind words to so many)
• Thanks to the many individuals and groups who have
donated their time, code, and talents to make the Internet
what it is today. Without their efforts, this infrastructure we
take for granted would not exist. We owe many our
gratitude.
Brookhaven Science Associates
U.S. Department of Energy 30
Network Services
Questions/Comments
???
Brookhaven Science Associates
U.S. Department of Energy 31
Network Services
BNL Points of Contact
Scott Bradley, Manager of Network Services
• 631.344.5745, bradley@bnl.gov
John Bigrow, Senior Network Architect
• 631.344.2648, big@bnl.gov
Brookhaven Science Associates
U.S. Department of Energy 32