multi by shimeiyan


									How to Multi-Home

     Avi Freedman
    VP Engineering
AboveNet Communications
      What is Multi-Homing?
• Multi-homing is the process of selecting,
  provisioning, and installing a redundant
  connection to the Internet.
• Could be the same provider, or a different
            Why Multi-Home?
• Slow is 1,000,000% better than dead.
• You may be out of bandwidth.
• And
  –   Telco circuits die.
  –   Routers die.
  –   Providers‟ networks fail.
  –   Different networks have better performance to
      different sites.
   A Multi-Homed Architecture
• Ideally, take advantage of the opportunity to
  multi-home to remove all single points of
  failure in your network.
• Use -
  – Multiple providers, unless your current
    provider will let you have cheap backup
  – Multiple routers
  – Multiple telco vendors
    Multi-Homed Architecture
• Two routers, each with a different WAN
  connection from a different telco vendor.
• Use HSRP or VRRP internally to make both
  routers look like one “virtual” router.
• Eventually, multiple providers.
• Upcoming Boardwatch article with configs.
      How the Internet Works
• Well, it breaks more than it works but when
  it does work -
• The Internet is a network of networks.
• Each network (called Autonomous System)
  on the Internet announces “routes”, which
  are lists of the IP addresses of the boxes on
  their network.
• You need to be able to send packets *to*,
  and get packets *from*, everywhere.
      Inbound Traffic - Routes
• Routes are announced via BGP4 (the
  Border Gateway Protocol)
• Routers are announced to BGP peers.
• Each “BGP peer” can be a “network peer”
  or a “transit peer”.
• Network peers exchange just lists of
  customer routes.
• Each route is tagged by the ASNs it passes
     Inbound Traffic - Routes
• So when AboveNet and UUNET peer, only
  AboveNet and UUNET routes are
  exchanged. No Sprint, PSI, etc...
• Transit peers -
  – Announce to their customers all of the routes
    on the „net (AboveNet, UUNET, Sprint, PSI,
    and the 60,000+ routes on the „net).
  – Announce to their peers all routes heard via
      Inbound Traffic - Routes
• So if you advertise to
  AboveNet, -
  – If you‟re a network peer, they only re-announce to customers (and use it
  – If you‟re a transit peer/customer, they announce to all of their network peers.
• That‟s how you get global *inbound*
        Address Space Issues
• Noone wants to hear a route for you unless -
  – You are multi-homed (even then, some people
    don‟t want to hear routers), or
  – You have your own direct IP space allocation
    from ARIN, RIPE, or APNIC.
• So, when you‟re single-homed without your
  own space, your IPs are reachable because
  they‟re part of your provider‟s “aggregate”
        Address Space Issues
• For example, your provider has
• You have from them.
• You‟re single-homed.
• The only route on the „net for you is the route, “originated” by your
  provider‟s ASN (and you don‟t have to do
  anything special).
        Address Space Issues
• If you have your own CIDR block and are
  single-homed, your provider will originate
• So, if you have, it‟ll be
  visible as an announcement by your
  provider, originated into the BGP mesh with
  your provider‟s ASN as the “origin”.
        Address Space Issues
• If you have your own IP space and want to
  multi-home, addressing issues are simple.
• Your other provider will start also
  originating your IP blocks.
• Or you‟ll start speaking BGP, originate your
  IP blocks, and your providers will re-
  advertise them to the world.
        Address Space Issues
• If you don‟t have your own IP space, it‟s a
  bit more complicated.
• So, normally your ISP will only be
  advertising if you have
• If you‟re multi-homed, your other provider
  will have to advertise
• But *so will your first provider*.
• Why?
        Address Space Issues
• Routes are chosen first by specificity.
• That is, to how many IP addresses they
• The route “covering” the fewest IP is the
  most specific, and wins.
• (Otherwise default would always win and
  nothing would work.)
        Address Space Issues
• So, if ISP 1 advertises only
  and ISP 2 advertises only,
  all inbound traffic from the „net will come
  in on ISP2.
• So, ISP 1 needs to “blow a hole in their
  filters” to “leak” the more specific route.
      Address Space: Filtering
• Some ISPs do or did filter on routes smaller
  than (more specific than) /19s in >
• But it doesn‟t matter as long as your two
  upstreams have good connectivity.
• Why?
      Address Space: Filtering
• If Sprint doesn‟t see from
  ISP1 or ISP2, they‟ll still see your
  provider‟s route.
• So if your connectivity to ISP1 (the owner
  of goes down, all will be
  well as long as ISP1 still sees from ISP2.
• Sprint -> ISP1 -> ISP2
• This is why people don‟t let you take IPs...
     Load-Balancing Outbound
• You can use static default routes to control
  outbound packets.
  – ip route serial0/0
  – ip route serial1/0
• If they‟re equal-cost (no metric at the end),
  it‟ll load-balance based on *destination*, by
     Load-Balancing Outbound
• Why load-balance based on destination?
• For internal networking, sometimes per-
  packet-load balancing makes sense.
• But if you‟re trying to talk to England and
  one provider has a 60ms path and the other
  has a 150ms path, packets will arrive out of
  order and TCP and UDP apps get unhappy
  and slow.
  How it works, Single-Homed
• Outbound (easy):
  – Use a default route to your provider.
• Inbound:
  – Your provider originates a large (aggregate)
    BGP route, and gives you some space from
    inside it; and/or
  – Your provider originates BGP routes for your
    ARIN/RIPE/APNIC CIDR blocks as well.
 How it Works, Multi-Homed, Static
• Outbound (easy):
  – Load-balance default routes to deal with
    outbound packets.
• Inbound:
  – Your providers both originate BGP routes for
    just the address space you‟re using, even if it‟s
    out of one provider‟s space; and/or
  – Your providers both originate BGP routes for
    your ARIN/RIPE/APNIC CIDR blocks as well.
 How it Works, Multi-Homed, Static
• Special note:
  – When providers configure BGP for single-
    homed customers, they will generally “nail up”
    your routes (even your directly-issued) CIDR
    blocks, so that if your connection goes down
    and up and down and ..., they don‟t have to flap
    that route out to the whole Internet. This is a
    good thing.
 How it Works, Multi-Homed, Static
• Special note (ctd):
  – But you NEED to make sure, when you‟re
    multi-homed, that the providers are NOT
    nailing your routes up.
  – Why?
  – Because if they do, when one T1 goes down,
    that provider will still advertise you to the
    world, thus “blackholing” you.
 How it Works, Multi-Homed, BGP

• Topic of next talk.
• You either load-balance outbound with
  statics, or take full routes from your
  providers (if you can).
• You originate advertisements under your
  ASN for your directly-issued CIDR blocks,
  AND for the parts of your providers‟ space
  that you‟re using (with their permission).
  The Transition: Static Routing
• To transition:
  –   Turn up the other T1/T3/Ethernet.
  –   Put IPs on the interface.
  –   Run tests end-end.
  –   Start load-balancing default to the new T1.
  –   Then, in the middle of the night, have the new
      provider start advertising your IP space. Make
      sure you have reachability to every other ISP
      you can think of afterwards.
  The Transition: Static Routing
• To transition (ctd):
  – After testing it live, turn off your other transit
    pipes and make sure that, after a few minutes,
    you still have connectivity.
   The Transition: BGP Routing
• To transition:
  – Turn up the other T1/T3/Ethernet.
  – Put IPs on the interface.
  – Run tests end-end.
  – Start load-balancing default to the new T1.
  – Then, undo that and bring up a BGP session
    that permits no routes either way.
  – Then start taking routes, and watch outbound
   The Transition: BGP Routing
• To transition (ctd):
  – Then, start announcing your routes.
  – Then, in the middle of the night, have your ISP
    take out the static route and BGP
    announcement they were making.
  – Make sure your route is propagating.
  – Test reachability.
  – Turn off your other pipes.
  – Test reachability.
               BGP or no?
• Advantages of doing static -
  – Cheaper/smaller routers (less true nowadays)
  – Simpler to configure
• Advantages of doing BGP -
  – More control of your destiny (have providers
    stop announcing you)
  – Faster/more intelligent selection of where to
    send outbound packets.
  – Better debugging of net problems (you can see
    the Internet topology now)
    Same Provider or Multiple?
• If your provider is reliable and fast, and
  affordably, and offers good tech-support,
  you may want to multi-home initially to
  them via Frame, SMDS, or some backup
  path (slow is 1,000,000% better than dead).
• Eventually you‟ll want t multi-home to
  different providers, to avoid failure modes
  due to one provider‟s architecture decisions.


• inet-access mailing list
• Nailing routes

To top