Get documents about "Recommending a Strategy
Document Sample
The Network
Affandi
Singaren, Singapore
Doug Pearson
Indiana University
Internet2 Commons Site Coordinator Training
December 3, 2003
National University of Singapore
A Sample University
Network
LAN
Building network
Campus backbone
WAN intranet
Internet
Internet2
2
Sample Network
3
Wiring and Station
Cables
Ethernet switch horizontal wiring datajack station cable device
iMac
3rd floor
Common practice is unshielded twisted-pair
(UTP) according to the specs:
• CAT 3 [old] supports 10 Mbps Ethernet (10base-T)
• CAT 5 [modern] supports 10base-T, 100 Mbps
(100base-TX) and 1000 Mbps (1000base-T) Ethernet
• CAT 6 [new] supports CAT 5 applications +
4
Wiring and Station
Cables
Actual wire used and quality of
installation may vary widely – know
your wiring!
Important to consider the station cables
• Don’t use sub-CAT 5 station cables for 100 Mbps
connections.
5
Ethernet LAN
Ethernet switch horizontal wiring datajack station cable device
iMac
3rd floor
2nd floor
1st floor
router
6
Ethernet LAN
10 / 100 / 1000 Mbps
Full- and Half-Duplex
• Half-duplex: send or receive, one at a time.
• Full-duplex: send and receive simultaneously.
• 10 Mbps Ethernet supports half-duplex; full-duplex
is not consistently implemented.
• 100 Mbps supports half- and full-duplex.
7
LAN: Switches vs.
Repeaters
Repeaters (hubs) are old technology.
A repeater sends (repeats) packets
that are incoming on one port, out all
other ports (I know you’re out there
somewhere!).
Can only operate in half-duplex mode.
Bandwidth and jitter provided to any
single device is highly dependent on
the LAN traffic.
8
LAN: Switches vs.
Repeaters
A switch learns the MAC addresses of the
devices connected to it, and sends packets
directly and only to the target end-point.
Provides much more consistent bandwidth
and latency (low jitter).
A well-designed switched LAN is important
for videoconferencing. Repeater-based
LANs should be upgraded to switched for
videoconferencing!
9
LAN: Ethernet Duplex
Mismatch
“One of the most common causes of
performance issues on 10/100Mb
Ethernet links is when one port on the
link is operating at half-duplex while
the other port is operating at full-
duplex.”
• http://www.cisco.com/warp/public/473/3.html
10
LAN: Ethernet Duplex
Mismatch
“There is a silent performance-killer out
there, one so inconspicuous that it is hardly
ever looked for or even suspected. You
could suffer from it and never know it, as it
robs a site of performance but not
connectivity. This performance-killer has a
name: Ethernet duplex mismatch.”
• http://www.hostingtech.com/nm/01_01_mismatch.html
11
LAN: Ethernet Duplex
Mismatch
If one end of a connection (device or
Ethernet switch) is set for auto-
negotiation, and fails to see auto-
negotiation at the other end, the former
sets itself to the default, half-duplex.
Auto-negotiation can sometimes fail,
even when both sides are set to auto
(although this isn’t as prevalent as in
the past).
12
LAN: Duplex Mismatch
– Detection
Microsoft Windows doesn’t display
the auto-negotiated duplex setting.
Some routers re-negotiate Auto-
duplex, which introduces jitter.
13
LAN: Ethernet Duplex
Mismatch
SETTINGS RESULTS
switch device switch device
auto auto auto auto
half half half half
full full full full
auto full BAD! half full
full auto BAD! full half
auto half half half
half auto half half
14
LAN: Duplex Mismatch
– Detection
Show port statistics on the Ethernet switch. When
mismatched, the full-duplex end will report a high
level of CRC or alignment errors; the half-duplex
end will report a high number of late collisions.
Port Align FCS Xmit Rcv UnderSize
Err Err Err Err
2/11 - 0 0 3077 0
Port Single- Multi- Late- Excess- CarriSen Runts Giants
Coll Coll Coll Coll
2/11 3233 0 2588 0 0 2489 0 15
LAN: Duplex Mismatch-
Prevention
Always configure switches and devices
according to your local policy. An
example policy is:
• If building wiring is sub-CAT 5, then set switch
ports to 10/half
• If building wiring is CAT 5 or better, then set
switch ports and devices to Auto.
Monitor switch port stats and logs
16
Router
Ethernet switch
Segments LANs into
distinct networks and
3rd floor subnetworks, e.g., the
distinct red, green, and
2nd floor
blue LANs with distinct
1st floor
network numbers.
router Segments LANs into
broadcast domains
17
Router
Provides interface to
the WAN.
• Intranet, commercial
Internet, and Internet2
connections.
• Typically, every
networked device at an
Internet2-connected
institution has
connectivity to Internet2.
18
VLAN
A single, physical LAN can be logically
segmented into multiple logical LANs;
and,
Physically separate LANs can be made
to behave and appear as a single LAN.
19
VLAN
Packets are tagged according to LAN
membership, e.g., green LAN, red
LAN, and blue LAN.
Ethernet switches establish broadcast
domains according to the defined
VLAN boundaries.
Routers establish multiple VLANs on a
single interface.
20
VLAN
21
VLAN
Modern campus network architectures
are tending to move away from
traditional router-for-a-building design,
to VLAN designs.
22
Old Design Included a
lot of Routers
routers
23
New Design Includes
VLAN Router
router
24
WAN Segments
Service Speed Sample Uses
T1 1.5 Mbps remote building; extension center
DS3 45 Mbps inter-campus; Internet (I1) connection
OC3 155 Mbps inter-campus; I1 & Internet2 connection
OC12 622 Mbps I1 backbones; Internet2 connection
OC48 2.4 Gbps I1 and Internet2 backbones
Gigabit 1 Gbps advanced inter-campus connections
Ether when have access to dark fiber
net 25
Indiana University Abilene
NOC Weathermap
26
High Performance Research and
Education Networks
Internet2 / Abilene
• http://www.internet2.edu
• http://www.abilene.iu.edu
STARTAP and International Networks
• http://www.startap.net
US Government-Sponsored Networks
• http://www.startap.net/NETWORKS
27
Traffic on the Network
Typical university today:
• IP
– TCP
– UDP
• IPX [diminishing]
• Appletalk [diminishing]
28
Traffic on the LAN
Unicast : one-to-one
Multicast: one-to-many
Broadcast: one-to-every
29
Unicast
Most common traffic
Common applications: mail,
Web browsing, file transfer,
etc.
30
IP Multicast
A one-to-many mode of transmission
Network numbers 224.0.0.0 through
239.255.255.255 are reserved for
multicast.
Examples of multicast applications:
• Vic/rat videoconferencing
• Centralized PC software administration tools such
as Symantec Ghost
31
IP Multicast – Leak
Problems
Beware: high rates of unpruned
multicast can adversely affect
videoconference performance.
Use a network traffic and protocol
analyzer to identify this problem.
32
Broadcast
A one-to-every mode of transmission
Used by network protocols including ARP
and IPX, NetBIOS system discovery, and
name resolution.
All devices on the network must process
every broadcast packet; high broadcast rates
can divert processing capacity.
If the broadcast domain is too large or
unusually active, the activity required at the
end-point to deal with the broadcasts could
diminish performance.
33
Broadcast
A healthy network should have less
than 100 broadcast packets per
second.
Check using a network traffic and
protocol analyzer tool.
34
Firewalls
A firewall is a network node that acts to
enforce an access control policy between
two networks, e.g., between a university
intranet and the commercial Internet.
Used to secure IT resources against external
attacks and break-ins.
Network-layer firewalls typically make their
decisions based upon port numbers and
source/destination addresses.
Application-layer firewalls act as proxies.
35
Firewalls
H.323 uses the IP ports:
• Statically-assigned TCP ports 1718 – 1720 and
1731 for call setup and control.
• Dynamically-assigned UDP ports in the range of
1024 – 65535 for video and audio data streams.
Firewalls don’t allow unrestricted ports.
Typical modern firewalls and H.323
don’t get along so well.
36
Firewalls – Solutions for
H.323
[bad; non-scaleable] Allow unrestricted
ports for specific, known, external IP-
addresses.
[better, but still not so good] Use feature of
some videoconferencing clients to confine
dynamic ports to a specific, narrow range.
[OK, but extra admin work and cost] Use an
H.323 application proxy.
[best] Use a firewall that snoops on the
H.323 call set-up channels (static ports) and
opens ports for the audio/video (dynamic
ports) as needed. 37
NATs
Allows multiple computers behind the NAT to
share one external network address.
Uses:
• Alleviate shortage of IP addresses
• Security – obscures view of the network from
outside
• Flexible network administration
Not commonly used at universities on the
campus level. Used somewhat in
corporations. Common in small offices and
at home – behind DSL, cable modem, or
ISDN network service.
38
NATs
Difficult to use H.323 behind NATs.
Some videoconferencing terminals provide
features to work with NAT – refer to
videoconferencing terminal documentation.
39
Latency
Latency is the time required for a packet to
traverse a network from source to
destination.
Components of latency include:
• Propagation delay: the time it takes to traverse
the distance of the transmission line; controlled by
the speed of light in the media; rule-of-thumb:
20ms San Francisco to New York.
40
Latency
Transmission delay: the time it takes for the
source to put a packet on the network. Rule-
of-thumb: < 1ms.
Store-and-forward delay: the cumulative
length of time it takes the internetworking
devices along the path to receive, process,
and resend the packets. Rule-of-thumb:
variable, and depends upon network load.
41
Latency
Rule of thumb:
• A one-way delay of:
• 0 – 150 ms provides excellent interactivity
• 150 – 300 ms is OK
• 300 – 400 ms is bad
• 400+ ms is unacceptable
42
Jitter
Jitter is variation in latency over time.
If the endpoints are on switched LANs,
then the primary source of jitter is
variation in the store-and-forward time,
resulting from network load.
H.323, particularly audio, is adversely
affected by high levels of jitter.
What is high? Rule of thumb?
43
Packet Loss
Packet loss is typically due to
congested links and routers.
• 1% is noticeable
• 5% becomes intolerable
44
QoS
Not currently feasible on commercial
Internet and Internet2 networks for
production, regular use. Internet2 is
working on QoS plans, but the current
over-provisioned Internet2 network
doesn’t dictate need.
Is useful on over-utilized intranet WAN
links.
45
QoS
How:
• Some videoconferencing terminals can set
the IP precedence bits. Use that for
marking and priority queuing on the WAN.
Or:
• Use a H.323 Proxy for consolidation of
traffic to a single address, router access list
for marking, and priority queuing on the
WAN.
46
QoS
Caution!
• The wrong implementation could result in
unwanted tradeoffs, e.g., packet loss
improves but jitter gets worse.
47
The End-to-End
Performance Problem
Scenario
• Users on two different campuses of a university
are experiencing poor video and audio in a
conference.
• Each user is supported by a different group of
videoconferencing engineers.
• Each campus is supported by a different group of
network engineers.
• The wide-area network is supported by a third
group of network engineers.
48
The End-to-End
Performance Problem
Problem
• How do the users get timely, useful assistance?
• How is network problem resolution coordinated?
49
The End-to-End
Performance Problem
Obstacles
• Different groups, schedules, and priorities.
• No one engineer has a complete understanding of
the entire network path.
• No one engineer can gain access to all the
network nodes (routers, switches) along the path
to inspect for trouble.
• Communications are inconsistent from engineer to
engineer.
50
The End-to-End
Performance Problem
Solutions
• Articulate the E-2-E problem to network
management and engineers on all campuses.
• Establish reliable communication tools, and insist
that engineers utilize the tools.
• Hold regular meetings; bring all engineers
together in one place and time to share
information.
• Have good network documentation for all
networks.
51
H.323 is Network
Sensitive!
The big problems are:
• Half/Full-duplex mismatches
• Packet loss
• Jitter
• Substandard horizontal wiring or station cables
• Multicast leaks
• High broadcast rates
52
Tools
Ping – availability, loss, roundtrip time
Traceroute – path discovery
Pingplot – graphical traceroute/ping
MRTG – graph link/port utilization & errors
Iperf – bandwidth, loss and jitter
Gnuplotping – visualize jitter
Sniffer – inspect traffic on the LAN
VideNet Scout – bandwidth, loss and jitter
Internet2 Detective – detect I2 connection
H.323. Beacon –protocol-specific tests
53
Tools: Ping
Test for availability, loss, and roundtrip
time
ICMP Echo Request
• Plus optional dummy payload – only in the
direction of the ping, i.e., source destination
54
Tools: Sample Ping from
Windows
C:\WINDOWS>ping 10.1.1.1
Pinging 10.1.1.1 with 32 bytes of data:
Reply from 10.1.1.1: bytes=32 time=88ms TTL=112
Reply from 10.1.1.1: bytes=32 time=72ms TTL=112
Reply from 10.1.1.1: bytes=32 time=69ms TTL=112
Reply from 10.1.1.1: bytes=32 time=69ms TTL=112
Ping statistics for 10.1.1.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0%
loss),
Approximate round trip times in milli-seconds:
Minimum = 69ms, Maximum = 88ms, Average =
74ms
C:\WINDOWS> 55
Tools: Sample Ping from
Windows
C:\WINDOWS>ping -l 40000 10.1.1.1
Pinging 10.1.1.1 with 40000 bytes of data:
Reply from 10.1.1.1: bytes=40000 time=2412ms TTL=112
Reply from 10.1.1.1: bytes=40000 time=2721ms TTL=112
Reply from 10.1.1.1: bytes=40000 time=2761ms TTL=112
Reply from 10.1.1.1: bytes=40000 time=2714ms TTL=112
Ping statistics for 10.1.1.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0%
loss),
Approximate round trip times in milli-seconds:
Minimum = 2412ms, Maximum = 2761ms, Average =
2652ms
C:\WINDOWS>
56
Tools: Ping Big and Fast
on Linux
ping -c2000 -i.03 -s1470 -q
[destination]
• Count of 2000 packets
• Interval of .03 seconds between packet starts
• Packet size of 1470 bytes
• (2000)(.03) = 60 second long test
• (1/(.03 sec/packet))(1470 bytes/packet)(8
bits/byte) = 392 Kbps
57
Tools: Ping Big and Fast
on Linux
$ ping -c2000 -i.03 -s1470 -q
[hostname]
PING [hostname] ([hostaddr]) from
[hostaddr2] : 1470(1498) bytes of data.
--- [hostname] ping statistics ---
2000 packets transmitted, 2000 packets
received, 0% packet loss
round-trip min/avg/max = 4.8/5.1/13.2
ms
58
Tools: Traceroute
Used to discover the layer-3 network path
(routers) between the two endpoints
Doesn’t identify layer-2 devices (switches)
Must run from one of the discovery
endpoints – it can’t act as a third party.
Take baselines – know what your path
should be in advance of trouble
59
Tools: Traceroute
Microsoft tracert uses ICMP packets,
whereas Unix traceroute uses UDP;
may be of importance in networks
where routers are configured to not
respond to ICMP; or if ICMP is blocked.
60
Tools: Traceroute;
Sample Output
[dodpears@huck dodpears]$ traceroute www.internet2.edu
traceroute to www.internet2.edu (209.211.239.208), 30 hops max, 38 byte
packets
1 wcc-sub5-hp1 (129.79.5.253) 11.726 ms 0.627 ms 0.571 ms
2 iub-gw (129.79.8.10) 3.133 ms 0.717 ms 0.651 ms
3 156.56.249.22 (156.56.249.22) 2.544 ms 3.138 ms 2.538 ms
4 abilene-iupui.abilene.ucaid.edu (198.32.11.13) 5.245 ms 3.402 ms
3.493 ms
5 clev-ipls.abilene.ucaid.edu (198.32.8.26) 9.381 ms 9.586 ms 9.244 ms
6 nycm-clev.abilene.ucaid.edu (198.32.8.30) 23.198 ms 21.963 ms
21.775 ms
7 border-abilene-oc3.advanced.org (209.211.237.97) 23.448 ms 23.268
ms 23.052 ms
61
8 www.internet2.edu (209.211.239.208) 23.559 ms 23.478 ms 23.234 ms
Tools: Traceroute
8 www.internet2.edu (209.211.239.208) 23.559 ms 23.478 ms 23.234
ms
Hop
Router/host name
Router/host address
Round-trip times of each of three probes
62
Tools: Traceroute –
Example Uses
Identify the path, and then perform
pings along the path segments to
isolate troublesome segments.
Insure that Internet2 is being used for a
connection, rather than commercial
Internet.
63
Tools: Ping Plotter
Shareware tool; $15
• http://www.pingplotter.com
Performs a visual traceroute and ping
tests along the entire path
Permits identification of bottlenecks
along a path
64
Tools: Ping Plotter
65
Tools: MRTG
Multi Router Traffic Grapher
Collect and graph scalar, time-based
data, e.g., router and link performance
data.
66
Tools: MRTG
67
Tools: MRTG
68
Tools: Iperf
http://dast.nlanr.net/Projects/Iperf/
Client/server application that
• Measures maximum TCP bandwidth
• Facilitates tuning of TCP and UDP parameters
• Reports bandwidth, jitter, and packet loss
69
Tools: Iperf
Example on Intercampus DS3
At server, invoke:
iperf -fk -i30 -u -s
(f)ormat reports in kbps
(i)nterval for reporting = 30 seconds
(u)dp
(s)erver mode
70
Tools: Iperf
Example on Intercampus DS3
At client, invoke:
iperf -u -b800k -t3600 -c [hostname-
server]
(u)dp
(b)andwidth = 800kbps
(t)ime of run = 3600 seconds
(c)lient mode
[hostname-server] = server to target
71
Tools: Iperf
Example on Intercampus DS3
[dodpears@vc-iperf iperf]$ iperf -fk -i30 -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 149.166.197.80 port 5001 connected with 129.79.92.230 port 1031
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Datagrams
[ 3] 0.0-30.0 sec 3000 KBytes 819 Kbits/sec 0.300 ms 0/ 2090 (0%)
[ 3] 30.0-60.0 sec 3000 KBytes 819 Kbits/sec 0.242 ms 0/ 2090 (0%)
[ 3] 60.0-90.0 sec 3000 KBytes 819 Kbits/sec 0.338 ms 0/ 2090 (0%)
[...]
[ 3] 0.0-90.0 sec 9000 KBytes 819 Kbits/sec 0.263 ms 72
0/ 6393 (0%)
Iperf- Example on
Intercampus DS3
MRTG utilization graph
showed bandwidth peaking at
capacity ~ 10:00a – 2:00p
As utilization peaked on the
DS3, jitter measured by Iperf
rose to unacceptable level
Iperf also reported periodic
high packet loss, with no
apparent correlation to the
low-resolution MRTG
utilization reports
73
Iperf- Example on
Intercampus DS3
Second day, utilization as
reported by MRTG is
staying reasonable.
Jitter measured by Iperf is
staying low.
The periodic high packet
loss remains, until noon
when network engineer
adjusted the QoS settings.
74
Iperf- Example on Intercampus
DS3
Third day, utilization
peaking at 3:00p.
As utilization peaked
jitter measured by Iperf
also rising.
Still no packet loss.
QoS fixed the packet loss
problem, but still not
certain about the jitter –
more analysis needed.
75
Tools: Gnuplotping
Pings multiple hosts in parallel with
graphical display (gnuplot) of the delay
distribution.
Runs on Unix/X-Windows
76
Tools: gnuplotping
77
Tools: Network Traffic
Analyzer
Reveals the traffic on a LAN
Protocol analysis
Reports such as utilization, protocols,
conversations, nodes, etc.
Network General Sniffer
WildPackets EtherPeek
78
Tools: ViDeNet Scout
Scout is a web-based, distributed
network performance analysis tool
developed at the University of North
Carolina at Chapel Hill.
Scout makes use of the Chariot
performance testing engine developed
by NetIQ.
• http://scout.video.unc.edu/
79
Tools: Internet2
Detective
I2 Detective is a small application.
Detects Internet2 connection.
Measures connection bandwidth (using
Iperf).
Detects multicast connection.
• http://detective.internet2.edu/
80
Tools: H.323 Beacon
Used to measure, monitor and qualify the
performance of an H.323 Videoconference
session.
Provides H.323-protocol specific evidence
and other information necessary to
troubleshoot H.323 application performance
problems in the network and at the host
(end-to-end)
• http://www.itecohio.org/beacon/
81
Recommendations
Develop a close relationship with the
network engineers and NOC. Make sure
they understand what’s being done with
videoconferencing and the network
sensitivity of IP-based video.
Articulate the End-to-End Performance
Problem to network engineering and
operations management. Champion ways to
reduce the problem.
Be sure to open trouble tickets with your
NOC so that a problem history is
maintained. 82
Recommendations
Use switched Ethernet.
Watch out for duplex mismatches.
Keep an eye on utilization of WAN
links, packet loss, and jitter.
Make sure you don’t have broadcast or
multicast leaking problems.
Make sure wiring is up to the task.
83
Recommendations
Have engineers in the
videoconferencing support group
trained to understand networking
issues and tools.
84
