networks by gjjur4356

VIEWS: 4 PAGES: 76

									                          EE898.02
               Architecture of Digital Systems
                           Lecture 4
             Interconnection Networks and Clusters



                        Prof. Seok-Bum Ko




                                                      EE898
11/12/2004
                                                     Lec 4.1
                                    Networks

             • Goal: Communication between computers
             • Eventual Goal: treat collection of computers as
               if one big computer, distributed resource
               sharing
             • Theme: Different computers must agree on
               many things
                – Overriding importance of standards and protocols
                – Error tolerance critical as well
             • Warning: Terminology-rich environment




                                                                      EE898
11/12/2004
                                                                     Lec 4.2
                                       Networks
             • Facets people talk a lot about:
                –   direct (point-to-point) vs. indirect (multi-hop)
                –   topology (e.g., bus, ring, DAG)
                –   routing algorithms
                –   switching (aka multiplexing)
                –   wiring (e.g., choice of media, copper, coax, fiber)
             • What really matters:
                –   latency
                –   bandwidth
                –   cost
                –   reliability




                                                                           EE898
11/12/2004
                                                                          Lec 4.3
                 Interconnections (Networks)
   • Examples (Figure 8.1, page 788):
       – Wide Area Network (ATM): 100-1000s nodes; ~ 5,000 kilometers
       – Local Area Networks (Ethernet): 10-1000 nodes; ~ 1-2 kilometers
       – System/Storage Area Networks (FC-AL): 10-100s nodes;
                                  ~ 0.025 to 0.1 kilometers per link
 a.k.a.
 end systems,
 hosts




 a.k.a.
                           Interconnection Network
 network,
 communication
 subnet
                                                                      EE898
11/12/2004
                                                                     Lec 4.4
                    SAN: Storage vs. System
             • Storage Area Network (SAN): A block I/O
               oriented network between application
               servers and storage
                – Fibre Channel is an example
             • Usually high bandwidth requirements, and
               less concerned about latency
                – in 2001: 1 Gbit bandwidth and millisecond latency OK
             • Commonly a dedicated network
               (that is, not connected to another network)
             • May need to work gracefully when saturated
             • Given larger block size, may have higher bit
               error rate (BER) requirement than LAN
                                                                          EE898
11/12/2004
                                                                         Lec 4.5
                    SAN: Storage vs. System
             • System Area Network (SAN): A network
               aimed at connecting computers
                – Myrinet is an example
             • Aimed at High Bandwidth AND Low Latency.
                – in 2001: > 1 Gbit bandwidth and ~ 10 microsecond
             • May offer in order delivery of packets
             • Given larger block size, may have higher bit
               error rate (BER) requirement than LAN




                                                                      EE898
11/12/2004
                                                                     Lec 4.6
                    More Network Background

             • Connection of 2 or more networks:
               Internetworking
             • 3 cultures for 3 classes of networks
                – WAN: telecommunications, Internet
                – LAN: PC, workstations, servers cost
                – SAN: Clusters, RAID boxes: latency (System A.N.) or
                  bandwidth (Storage A.N.)
             • Try for single terminology
             • Motivate the interconnection complexity
               incrementally


                                                                         EE898
11/12/2004
                                                                        Lec 4.7
                              ABCs of Networks

             • Starting Point: Send bits between 2 computers




             •   Queue (FIFO) on each end
             •   Information sent called a “message”
             •   Can send both ways (“Full Duplex”)
             •   Rules for communication? “protocol”
                  – Inside a computer:
                      » Loads/Stores: Request (Address) & Response (Data)
                      » Need Request & Response signaling


                                                                             EE898
11/12/2004
                                                                            Lec 4.8
                           A Simple Example

             • What is the format of message?
                – Fixed? Number bytes?
                 Request/
                                         Address/Data
                 Response



                   1 bit                 32 bits
             0: Please send data from Address
             1: Packet contains data corresponding to request
             • Header/Trailer: information to deliver a message
             • Payload: data in message (1 word above)

                                                              EE898
11/12/2004
                                                             Lec 4.9
                   Questions About Simple Example

    • What if more than 2 computers want to
      communicate?
             – Need computer “address field” (destination) in packet
    • What if packet is garbled in transit?
             – Add “error detection field” in packet (e.g., Cyclic Redundancy Chk)
    • What if packet is lost?
             – More “elaborate protocols” to detect loss
               (e.g., NAK, ARQ, time outs)
    • What if multiple processes/machine?
             – Queue per process to provide protection
    • Simple questions such as these lead to more complex
      protocols and packet formats => complexity
                                                                              EE898
11/12/2004
                                                                             Lec 4.10
                   A Simple Example Revised

             • What is the format of packet?
               – Fixed? Number bytes?
                 Request/
                                        Address/Data   CRC
                 Response



                  2 bits                  32 bits      4 bits

   00: Request—Please send data from Address
   01: Reply—Packet contains data corresponding to request
   10: Acknowledge request
   11: Acknowledge reply

                                                                 EE898
11/12/2004
                                                                Lec 4.11
                 Software to Send and Receive

             • SW Send steps
               1: Application copies data to OS buffer
               2: OS calculates checksum, starts timer
               3: OS sends data to network interface HW and says start
             • SW Receive steps
               3: OS copies data from network interface HW to OS buffer
               2: OS calculates checksum, if matches send ACK; if not,
                 deletes message (sender resends when timer expires)
               1: If OK, OS copies data to user address space and signals
                 application to continue
             • Sequence of steps for SW: protocol
               – Example similar to UDP/IP protocol in UNIX

                                                                          EE898
11/12/2004
                                                                         Lec 4.12
             Network Performance Measures




    • Overhead: latency of interface vs. Latency: network
                                                       EE898
11/12/2004
                                                      Lec 4.13
               Universal Performance Metrics

               Sender        Transmission time
   Sender     Overhead       (size ÷ bandwidth)


              (processor
                 busy)
                           Time of   Transmission time    Receiver
                            Flight   (size ÷ bandwidth)   Overhead
  Receiver
                                                          (processor
                                Transport Latency
                                                             busy)

                                     Total Latency

  Total Latency = Sender Overhead + Time of Flight +
                  Message Size ÷ BW + Receiver Overhead
   Includes header/trailer in BW calculation?
                                                                        EE898
11/12/2004
                                                                       Lec 4.14
                Total Latency Example
   • 1000 Mbit/sec., sending overhead of 80 µsec &
     receiving overhead of 100 µsec.
   • a 10000 byte message (including the header), allows
     10000 bytes in a single message
   • 2 situations: distance 100 m vs. 1000 km
   • Speed of light ~ 300,000 km/sec
   • Latency0.01km = 80 + 0.01km / (50% x 300,000)
                     + 10000 x 8 / 1000 + 100 = 260 µsec
   • Latency0.5km = 80 + 0.5km / (50% x 300,000)
                     + 10000 x 8 / 1000 + 100 = 263 µsec
   • Latency1000km = 80 + 1000 km / (50% x 300,000)
                     + 10000 x 8 / 1000 + 100 = 6931
   • Long time of flight => complex WAN protocol
                                                     EE898
11/12/2004
                                                    Lec 4.15
                         Universal Metrics

             • Apply recursively to all levels of system
             • inside a chip, between chips on a board,
               between computers in a cluster, …
             • Look at WAN v. LAN v. SAN




                                                            EE898
11/12/2004
                                                           Lec 4.16
                  Simplified Latency Model

         • Total Latency = Overhead + Message Size / BW

         • Overhead = Sender Overhead + Time of Flight +
                         Receiver Overhead

         • Effective BW = Message Size / Total Latency




                                                          EE898
11/12/2004
                                                         Lec 4.17
                                                             Overhead, BW, Size
                       Delivered BW
                                              1,000
                                                                                 o1,                     o25,                 o500,
                                                                             bw1000                   bw1000                bw1000
                                               100
             Effective Bandwidth (Mbit/sec)




                                                                                   o25,
                                                                                                          o500,
                                                                o1,
                                                             bw100               bw100                    bw100
                                                 10
                                                       o1,
                                                      bw10            o25,
                                                                      bw10                    o500,
                                                  1                                           bw10


                                                  0


                                                  0                                                                                                Msg Size
                                                      16


                                                             64


                                                                      256


                                                                              1024


                                                                                       4096


                                                                                                  16384


                                                                                                           65536


                                                                                                                   262144


                                                                                                                               1048576


                                                                                                                                         4194304
         •How big are                                                        Message Size (bytes)
         real messages?
                                                                                                                                                        EE898
11/12/2004
                                                                                                                                                       Lec 4.18
                                     Measurement:
                               Sizes of Message for NFS
                    100%
                     90%
                     80%               Msgs
    Cummulative %




                     70%
                     60%               Bytes
                     50%
                     40%
                     30%
                     20%
                     10%
                      0%
                           0   1024   2048 3072 4096   5120 6144 7168 8192

                                      Packet size
                    • 95% Msgs, 30% bytes for packets ~ 200 bytes
                    • > 50% data transferred in packets = 8KB
                                                                          EE898
11/12/2004
                                                                         Lec 4.19
                      Interconnect Issues

             • Performance Measures
             • Network Media




                                             EE898
11/12/2004
                                            Lec 4.20
                            Network Media
     Twisted Pair:
                                             Copper, 1mm think, twisted to avoid
                                             attenna effect (telephone)
    Coaxial Cable:                           "Cat 5" is 4 twisted pairs in bundle
                       Plastic Covering

                              Insulator            Used by cable companies:
                                 Copper core
                                                   high BW, good noise
                        Braided outer conductor immunity
                                      Buffer                        Light: 3 parts
                                         Cladding                   are cable, light
    Fiber Optics                           Total internal           source, light
                                             reflection             detector.
      Transmitter                                     Receiver
       – L.E.D                                         – Photodiode Note fiber is
       – Laser Diode
                                                                    unidirectional;
             light                                                  need 2 for full
             source                                Silica core      duplex
                                             Cladding

                                                                              EE898
11/12/2004
                                          Buffer                             Lec 4.21
                                          Fiber
    • Multimode fiber: ~ 62.5 micron diameter vs. the 1.3
      micron wavelength of infrared light. Since wider it has
      more dispersion problems, limiting its length at 1000
      Mbits/s for 0.1 km, and 1-3 km at 100 Mbits/s. Uses
      LED as light
    • Single-mode fiber: "single wavelength" fiber (8-9
      microns) uses laser diodes, 1-5 Gbits/s for 100s kms
             – Less reliable and more expensive, and restrictions on bending
             – Cost, bandwidth, and distance of single-mode fiber affected by
               power of the light source, the sensitivity of the light detector, and
               the attenuation rate (loss of optical signal strength as light passes
               through the fiber) per kilometer of the fiber cable.
             – Typically glass fiber, since has better characteristics than the less
               expensive plastic fiber

                                                                                EE898
11/12/2004
                                                                               Lec 4.22
                Wave Division Multiplexing Fiber

             • Send N independent streams on single fiber!
             • Just use different wavelengths to send and
               demultiplex at receiver
             • WDM in 2000: 40 Gbit/s using 8 wavelengths
             • Plan to go to 80 wavelengths => 400 Gbit/s!




                                                              EE898
11/12/2004
                                                             Lec 4.23
                      Compare Media
      Assume 40 2.5" disks, each 25 GB, Move 1 km
      Compare Cat 5 (100 Mbit/s), Multimode fiber (1000
        Mbit/s), single mode (2500 Mbit/s), and car
      • Cat 5: (1000 x 1024 x 8 Mb) / 100 Mb/s = 23 hrs
      • MM: (1000 x 1024 x 8 Mb) / 1000 Mb/s = 2.3 hrs
      • SM:    (1000 x 1024 x 8 Mb) / 2500 Mb/s = 0.9 hrs
      • Car: 5 min + 1 km / 50 kph + 10 min       = 0.3 hrs
      • Car of disks = high BW media




                                                       EE898
11/12/2004
                                                      Lec 4.24
                       Interconnect Issues

             • Performance Measures
             • Network Media
             • Connecting Multiple Computers




                                                EE898
11/12/2004
                                               Lec 4.25
                 Connecting Multiple Computers
 • Shared Media vs. Switched:
   pairs communicate at same time:
   “point-to-point” connections
 • Aggregate BW in switched
   network is many times shared
       – point-to-point faster since no
         arbitration, simpler interface
 • Arbitration in Shared network?
       – Central arbiter for LAN?
       – Listen to check if being used (“Carrier
         Sensing”)
       – Listen to check if collision
         (“Collision Detection”)                   (A. K. A. data switching
       – Random resend to avoid repeated           interchanges, multistage
         collisions; not fair arbitration;         interconnection networks,
       – OK if low utilization                     interface message processors)
                                                                         EE898
11/12/2004
                                                                        Lec 4.26
             Connection-Based vs. Connectionless

       • Telephone: operator sets up connection between
         the caller and the receiver
             – Once the connection is established, conversation can continue for
               hours
       • Share transmission lines over long distances by
         using switches to multiplex several conversations on
         the same lines
             – “Time division multiplexing” divide B/W transmission line into a
               fixed number of slots, with each slot assigned to a conversation
       • Problem: lines busy based on number of
         conversations, not amount of information sent
       • Advantage: reserved bandwidth



                                                                             EE898
11/12/2004
                                                                            Lec 4.27
              Connection-Based vs. Connectionless

             • Connectionless: every package of
               information must have an address =>
               packets
               – Each package is routed to its destination by looking at
                 its address
               – Analogy, the postal system (sending a letter)
               – also called “Statistical multiplexing”
               – Note: “Split phase buses” are sending packets




                                                                            EE898
11/12/2004
                                                                           Lec 4.28
                            Routing Messages
             • Shared Media
               – Broadcast to everyone
             • Switched Media needs real routing. Options:
               – Source-based routing: message specifies path to the
                 destination (changes of direction)
               – Virtual Circuit: circuit established from source to
                 destination, message picks the circuit to follow
               – Destination-based routing: message specifies
                 destination, switch must pick the path
                   » deterministic: always follow same path
                   » adaptive: pick different paths to avoid congestion,
                     failures
                   » Randomized routing: pick between several good
                     paths to balance network load



                                                                            EE898
11/12/2004
                                                                           Lec 4.29
                    Deterministic Routing Examples
      • mesh: dimension-order routing
             – (x1, y1) -> (x2, y2)
             – first x = x2 - x1,
             – then y = y2 - y1,
      • hypercube: edge-cube routing
             – X = xox1x2 . . .xn -> Y = yoy1y2 . . .yn
             – R = X xor Y                                 010          110
             – Traverse dimensions of differing
               address in order
                                                                                111
      • tree: common ancestor                                   011
                                                                          100

                                                          000

                                                                  001         101

                                                                                 EE898
11/12/2004
                                                                                Lec 4.30
               Store and Forward vs. Cut-Through
    • Store-and-forward policy: each switch waits for
      the full packet to arrive in switch before sending to
      the next switch (good for WAN)
    • Cut-through routing or worm hole routing: switch
      examines the header, decides where to send the
      message, and then starts forwarding it immediately
             – In worm hole routing, when head of message is blocked, message
               stays strung out over the network, potentially blocking other
               messages (needs only buffer the piece of the packet that is sent
               between switches).
             – Cut through routing lets the tail continue when head is blocked,
               compressing the strung-out message into a single switch. (Requires
               a buffer large enough to hold the largest packet).



                                                                              EE898
11/12/2004
                                                                             Lec 4.31
             Cut-Through vs. Store and Forward
       • Advantage
          – Latency reduces from a function of:

              # of intermediate switches ×
             by the size of the packet to the time for 1st
              part of the packet to negotiate the switches +
             transmission time (=the packet size ÷
              interconnect BW)




                                                           EE898
11/12/2004
                                                          Lec 4.32
                            Congestion Control
   • Packet switched networks do not reserve bandwidth; this
     leads to contention (connection based limits input)
   • Solution: prevent packets from entering until contention
     is reduced
     (e.g., freeway on-ramp metering lights)
   • Options:
         – Packet discarding: If packet arrives at switch and no room in buffer,
           packet is discarded (e.g., UDP)
         – Flow control: between pairs of receivers and senders;
           use feedback to tell sender when allowed to send next packet
             » Back-pressure: separate wires to tell to stop
             » Window: give original sender right to send N packets before getting
               permission to send more; overlaps latency of interconnection with
               overhead to send & receive packet (e.g., TCP), adjustable window
         – Choke packets: aka “rate-based”; Each packet received by busy switch in
           warning state sent back to the source via choke packet. Source reduces
           traffic to that destination by a fixed % (e.g., ATM)


                                                                             EE898
11/12/2004
                                                                            Lec 4.33
                    Protocols: HW/SW Interface

     • Internetworking: allows computers on independent
       and incompatible networks to communicate reliably
       and efficiently;
             – Enabling technologies: SW standards that allow reliable
               communications without reliable networks
             – Hierarchy of SW layers, giving each layer responsibility for
               portion of overall communications task, called
               protocol families or protocol suites
     • Transmission Control Protocol/Internet Protocol
       (TCP/IP)
             – This protocol family is the basis of the Internet
             – IP makes best effort to deliver; TCP guarantees delivery
             – TCP/IP used even when communicating locally: NFS uses IP even
               though communicating across homogeneous LAN


                                                                               EE898
11/12/2004
                                                                              Lec 4.34
                  Protocol Family Concept
             Message      Logical       Message
                 Actual                     Actual
                          Logical
        H Message T                   H Message T
                 Actual                     Actual
  H H Message T T                   H H Message T T


                          Physical



                                                       EE898
11/12/2004
                                                      Lec 4.35
             Protocol Family Concept
   • Key to protocol families is that communication occurs
     logically at the same level of the protocol, called
     peer-to-peer,
   • but is implemented via services at the next lower level
   • Encapsulation: carry higher level information within
     lower level “envelope”
   • Fragmentation: break packet into multiple smaller
     packets and reassemble
   • Danger is each level increases latency if implemented
     as hierarchy (e.g., multiple check sums)




                                                         EE898
11/12/2004
                                                        Lec 4.36
     TCP/IP packet, Ethernet packet, protocols
       • Application sends message
                                          Ethernet Hdr
       • TCP breaks into 64KB segments,    IP Header
         adds 20B header
                                          TCP Header
       • IP adds 20B header, sends to     EHIP Data
         network
                                            TCP data
       • If Ethernet, broken into 1500B     Message
         packets with headers, trailers   Ethernet Hdr
         (24B)

      • All Headers, trailers have
        length field, destination, ...

                                                       EE898
11/12/2004
                                                      Lec 4.37
                            Example Networks

             • Ethernet: shared media 10 Mbit/s proposed
               in 1978, carrier sensing with exponential
               backoff on collision detection
             • 15 years with no improvement; higher BW?
             • Multiple Ethernets with devices to allow
               Ehternets to operate in parallel!
             • 10 Mbit Ethernet successors?
               –   FDDI: shared media (too late)
               –   ATM (too late?)
               –   Switched Ethernet
               –   100 Mbit Ethernet (Fast Ethernet)
               –   Gigabit Ethernet


                                                            EE898
11/12/2004
                                                           Lec 4.38
                           Connecting Networks
             • Bridges: connect LANs together, passing traffic
               from one side to another depending on the addresses
               in the packet.
                – operate at the Ethernet protocol level
                – usually simpler and cheaper than routers
             • Routers or Gateways: these devices connect LANs to
               WANs or WANs to WANs and resolve incompatible
               addressing.
                – Generally slower than bridges, they operate at the
                  internetworking protocol (IP) level
                – Routers divide the interconnect into separate smaller subnets,
                  which simplifies manageability and improves security
             • Cisco is major supplier;
               basically special purpose computers



                                                                                    EE898
11/12/2004
                                                                                   Lec 4.39
                           Comparing Networks
                        SAN                     LAN              WAN
                   FC-AL  Infini-   10 Mb      100 Mb 1000 Mb ATM
                          band      Ethernet   Ethernet Ethernet
        Length     30/1000 17/100   500/2500 200       100
        (meters)
        Data       2        1, 4, 12 1         1       4/1     1
        lines
        Clock      1000     2500    10         100     1000    155/
        (MHz)                                                  622
        Switch?    Opt.     Yes     Optional Opt.      Yes     Yes
        Nodes      <=127    ~1000   <=254      <=254   <=254   ~10000
        Material Copper     Copper Copper      Copper Copper Copper
                 / fiber    /fiber                    /fiber /fiber


                                                                       EE898
11/12/2004
                                                                      Lec 4.40
                           Comparing Networks
                        SAN                     LAN               WAN
                    FC-AL   Infini-    10 Mb    100 Mb 1000 Mb ATM
                            band       Ethernet Ethernet Ethernet
        Switch?     Opt.     Yes       Optional Opt.        Yes      Yes
        Bisection 800        (2000 -   10        100        1000 x   155 x
        BW        shared     24000)    shared    shared     switch   switch
        (Mbits    or 800 x   x         or 10 x   or 100 x   ports    ports
        /sec)     switch     switch    switch    switch
                  ports      ports     ports     ports
        Peak link 800        2000,     10        100        1000     155/
        BW(Mbits             8000,                                   622
        /sec)                24000
        Topology Ring or     Star      Line or   Line or    Star     Star
                  Star                 Star      Star


                                                                            EE898
11/12/2004
                                                                           Lec 4.41
                          Comparing Networks
                         SAN                  LAN              WAN
                    FC-AL  Infiniband 10 Mb    100 Mb 1000 Mb ATM
                                      Ethernet Ethernet Ethernet
        Connec-     Yes       Yes        Yes       Yes       Yes       No
        tionless?
        Store &     No        No         No        No        No        Yes
        forward?
        Conges-     Credit-   Back-      Carrier   Carrier   Carrier   Credit
        tion        based     pressure   sense     sense     sense     based
        control
        Standard    ANSI      Infiniband IEEE      IEEE      IEEE    ATM
                    Task      Trade      802.3     802.3     802.3   Forum
                    Group     Associa-                       ab-1999
                    X3T11     tion



                                                                         EE898
11/12/2004
                                                                        Lec 4.42
                         Packet Formats

             • See Fig 8.20 on page 826




                                           EE898
11/12/2004
                                          Lec 4.43
                           Wireless Networks
         • Media can be air as well as glass or copper
         • Radio wave is electromagnetic wave propagated by
           an antenna
         • Radio waves are modulated: sound signal
           superimposed on stronger radio wave which carries
           sound signal, called carrier signal
         • Radio waves have a wavelength or frequency:
           measure either length of wave
           or number of waves per second (MHz):
           long waves => low frequencies,
           short waves => high frequencies
         • Tuning to different frequencies => radio receiver
           pick up a signal.
             – FM radio stations transmit on band of 88 MHz to 108 MHz
               using frequency modulations (FM) to record the sound signal
                                                                              EE898
11/12/2004
                                                                             Lec 4.44
                           Issues in Wireless

             • Wireless often => mobile => network must
               rearrange itself dynamically
             • Subject to jamming and eavesdropping
                – No physical tape
                – Cannot detect interception
             • Power
                – devices tend to be battery powered
                – antennas radiate power to communicate and little of it
                  reaches the receiver
             • As a result, raw bit error rates are
               typically a thousand to a million times higher
               than copper wire
                                                                            EE898
11/12/2004
                                                                           Lec 4.45
               Reliability of Wires Transmission
             • bit error rate (BER) of wireless link
               determined by received signal power, noise
               due to interference caused by the receiver
               hardware, interference from other sources,
               and characteristics of the channel
               – Path loss: power to overcome interference
               – Shadow fading: blocked by objects (walls, buildings)
               – Multipath fading: interference between multiple version
                 of signals arriving different times
               – Interference: reuse of frequency or from adjacent
                 channels




                                                                            EE898
11/12/2004
                                                                           Lec 4.46
                      2 Wireless Architectures

             • Base-station architectures
                – Connected by land lines for longer distance
                  communication, and the mobile units communicate only
                  with a single local base station
                – More reliable since 1-hop from land lines
                – Example: cell phones
             • Peer-to-peer architectures
                – Allow mobile units to communicate with each other, and
                  messages hop from one unit to the next until delivered
                  to the desired unit
                – More reconfigurable




                                                                            EE898
11/12/2004
                                                                           Lec 4.47
                         Cellular Telephony
             • Exploit exponential path loss to reuse same
               frequency at spatially separated locations,
               thereby greatly increasing customers served
             • Divide region into nonoverlaping hexagonal cells
               (2-10 mi. diameter) which use different
               frequencies if nearby, reusing a frequency when
               cells far apart so that mutual interference OK
             • Intersection of three hexagonal cells is a base
               station with transmitters and antennas
             • Handset selects a cell based on signal strength
               and then picks an unused radio channel
             • To properly bill for cellular calls, each cellular
               phone handset has an electronic serial number
                                                                     EE898
11/12/2004
                                                                    Lec 4.48
                       Cellular Telephony II
             • Original analog design frequencies set for each
               direction: pair called a channel
                – 869.04 to 893.97 MHz, called the forward path
                – 824.04 MHz to 848.97 MHz, called the reverse path
                – Cells might have had between 4 and 80 channels
             • Several digital successors:
                – Code division multiple access (CDMA) uses a wider radio
                  frequency band
                – time division multiple access (TDMA)
                – global system for mobile communication (GSM)
                – International Mobile Telephony 2000 (IMT-2000) which is
                  based primarily on two competing versions of CDMA and one
                  TDMA, called Third Generation (3G)


                                                                         EE898
11/12/2004
                                                                        Lec 4.49
             Practical Issues for Interconnection
                           Networks

             • Connectivity: max number of machines
               affects complexity of network and protocols
               since protocols must target largest size
             • Connection Network Interface to computer
               – Where in bus hierarchy? Memory bus? Fast I/O bus?
                 Slow I/O bus? (Ethernet to Fast I/O bus, Infiniband
                 to Memory bus since it is the Fast I/O bus)
               – SW Interface: does software need to flush caches for
                 consistency of sends or receives?
               – Programmed I/O vs. DMA? Is NIC in uncacheable
                 address space?




                                                                         EE898
11/12/2004
                                                                        Lec 4.50
             Practical Issues for Interconnection
                           Networks

             • Standardization advantages:
                – low cost (components used repeatedly)
                – stability (many suppliers to chose from)
             • Standardization disadvantages:
                – Time for committees to agree
                – When to standardize?
                    » Before anything built? => Committee does design?
                    » Too early suppresses innovation
             • Reliability (vs. availability) of interconnect



                                                                          EE898
11/12/2004
                                                                         Lec 4.51
                       Practical Issues
    Interconnection    SAN          LAN        WAN
    Example            Infiniband   Ethernet   ATM
    Standard           Yes          Yes        Yes
    Fault Tolerance?   Yes          Yes        Yes
    Hot Insert?        Yes          Yes        Yes

    • Standards: required for WAN, LAN, and likely SAN!
    • Fault Tolerance: Can nodes fail and still deliver
      messages to other nodes?
    • Hot Insert: If the interconnection can survive a
      failure, can it also continue operation while a new
      node is added to the interconnection?
                                                      EE898
11/12/2004
                                                     Lec 4.52
             Cross-Cutting Issues for Networking

    • Efficient Interface to Memory Hierarchy vs. to
      Network
       – SPEC ratings => fast to memory hierarchy
       – Writes go via write buffer, reads via L1 and
         L2 caches
    • Example: 40 MHz SPARCStation(SS)-2 vs 50
      MHz SS-20, no L2$ vs 50 MHz SS-20 with L2$
      I/O bus latency; different generations
    • SS-2: combined memory, I/O bus => 200 ns
    • SS-20, no L2$: 2 busses +300ns => 500ns
    • SS-20, w L2$: cache miss+500ns => 1000ns

                                                         EE898
11/12/2004
                                                        Lec 4.53
                Crosscutting: Smart Switch vs.
                Smart Network Interface Card

                       Less Intelligent   More Intelligent
                       Small Ethernet     Large Ethernet
             Switch       Myrinet
                         Infiniband
                           Ethernet           Myrinet
              NIC     Infiniband Target   Infiniband Host
                       Channel Adapter    Channel Adapter


   •Inexpensive NIC => Ethernet standard in all computers
   •Inexpensive switch => Ethernet used in home networks
                                                              EE898
11/12/2004
                                                             Lec 4.54
                                      Cluster

         • LAN switches => high network bandwidth and scaling
           was available from off the shelf components
         • 2001 Cluster = collection of independent computers
           using switched network to provide a common service
         • Many mainframe applications run more "loosely
           coupled" machines than shared memory machines
           (next lecture)
             – databases, file servers, Web servers, simulations, and
               multiprogramming/batch processing
             – Often need to be highly available, requiring error tolerance and
               repairability
             – Often need to scale

                                                                               EE898
11/12/2004
                                                                              Lec 4.55
                           Cluster Drawbacks
         • Cost of administering a cluster of N machines
           ~ administering N independent machines
           vs. cost of administering a shared address space N
           processors multiprocessor
           ~ administering 1 big machine
         • Clusters usually connected using I/O bus, whereas
           multiprocessors usually connected on memory bus
         • Cluster of N machines has N independent memories
           and N copies of OS, but a shared address multi-
           processor allows 1 program to use almost all memory
             – DRAM prices has made memory costs so low that this
               multiprocessor advantage is much less important in 2001




                                                                          EE898
11/12/2004
                                                                         Lec 4.56
                       Cluster Advantages
         • Error isolation: separate address space limits
           contamination of error
         • Repair: Easier to replace a machine without bringing
           down the system than in an shared memory
           multiprocessor
         • Scale: easier to expand the system without bringing
           down the application that runs on top of the cluster
         • Cost: Large scale machine has low volume => fewer
           machines to spread development costs vs. leverage
           high volume off-the-shelf switches and computers
         • Amazon, AOL, Google, Hotmail, Inktomi, WebTV, and
           Yahoo rely on clusters of PCs to provide services used
           by millions of people every day
                                                              EE898
11/12/2004
                                                             Lec 4.57
                 Addressing Cluster Weaknesses

             • Network performance: SAN, especially
               Infiniband, may tie cluster closer to memory
             • Maintenance: separate of long term storage
               and computation
             • Computation maintenance:
                – Clones of identical PCs
                – 3 steps: reboot, reinstall OS, recycle
                – At $1000/PC, cheaper to discard than to figure out
                  what is wrong and repair it?
             • Storage maintenance:
                – If separate storage servers or file servers, cluster is
                  no worse?

                                                                             EE898
11/12/2004
                                                                            Lec 4.58
                 Putting it all together: Google
             • Google: search engine that scales at growth
               Internet growth rates
             • Search engines: 24x7 availability
             • Google 12/2000: 70M queries per day, or
               AVERAGE of 800 queries/sec all day
             • Response time goal: < 1/2 sec for search
             • Google crawls WWW and puts up new index
               every 4 weeks
             • Stores local copy of text of pages of WWW
               (snippet as well as cached copy of page)
             • 3 collocation sites (2 CA + 1 Virginia)
             • 6000 Processors and 12000 disks
                                                              EE898
11/12/2004
                                                             Lec 4.59
               Hardware Infrastructure
             • VME rack 19 in. wide, 6
               feet tall, 30 inches deep
             • Per side: 40 1 Rack Unit
               (RU) PCs +1 HP Ethernet
               switch (4 RU): Each blade
               can contain 8 100-Mbit/s
               EN or a single 1-Gbit
               Ethernet interface
             • Front+back => 80 PCs +
               2 EN switches/rack
             • Each rack connects to 2
               128 1-Gbit/s EN switches
             • Dec 2000: 40 racks at
11/12/2004
               most recent site             EE898
                                           Lec 4.60
                                  Google PCs
        • 2 IDE drives, 256 MB of SDRAM, modest Intel
          microprocessor, a PC mother-board, 1 power
          supply and a few fans.
        • Each PC runs the Linux operating system
        • Buy over time, so upgrade components:
          populated between March and November 2000
             – microprocessors: 533 MHz Celeron to an 800 MHz Pentium III,
             – disks: capacity between 40 and 80 GB, speed 5400 to 7200
               RPM
             – bus speed is either 100 or 133 MH
             – Cost: ~ $1300 to $1700 per PC

        • PC operates at about 55 Watts
        • Rack => 4500 Watts , 60 amps
                                                                        EE898
11/12/2004
                                                                       Lec 4.61
                                  Reliability
             • For 6000 PCs, 12000s, 200 EN switches
             • ~ 20 PCs will need to be rebooted/day
             • ~ 2 PCs/day hardware failure, or 2%-3% / year
                – 5% due to problems with motherboard, power supply, and
                  connectors
                – 30% DRAM: bits change + errors in transmission (100 MHz)
                – 30% Disks fail
                – 30% Disks go very slow (10%-3% expected BW)
             • 200 EN switches, 2-3 fail in 2 years
             • 6 Foundry switches: none failed, but 2-3 of 96
               blades of switches have failed (16 blades/switch)
             • Collocation site reliability:
                – 1 power failure,1 network outage per year per site
11/12/2004
                – Bathtub for occupancy                                   EE898
                                                                         Lec 4.62
                  Google Performance: Serving


             • How big is a page returned by Google?
               ~16KB
             • Average bandwidth to serve searches
                70,000,000/day x 16,750 B x 8 bits/B
                             24 x 60 x 60
                    =9,378,880 Mbits/86,400 secs
                              = 108 Mbit/s



                                                        EE898
11/12/2004
                                                       Lec 4.63
                 Google Performance: Crawling

             • How big is a text of a WWW page? ~4000B
             • 1 Billion pages searched
             • Assume 7 days to crawl
             • Average bandwidth to crawl
             1,000,000,000/pages x 4000 B x 8 bits/B
                             24 x 60 x 60 x 7
                    =32,000,000 Mbits/604,800 secs
                               = 59 Mbit/s



                                                          EE898
11/12/2004
                                                         Lec 4.64
              Google Performance: Replicating Index

             • How big is Google index? ~5 TB
             • Assume 7 days to replicate to 2 sites,
               implies BW to send + BW to receive
             • Average bandwidth to replicate new index
                    2 x 2 x 5,000,000 MB x 8 bits/B
                            24 x 60 x 60 x 7
                   =160,000,000 Mbits/604,800 secs
                              = 260 Mbit/s



                                                           EE898
11/12/2004
                                                          Lec 4.65
                             Colocation Sites
         • Allow scalable space, power, cooling and network
           bandwidth plus provide physical security
         • charge about $500 to $750 per Mbit/sec/month
             – if your continuous use measures 1- 2 Gbits/second
         to $1500 to $2000 per Mbit/sec/month
             – if your continuous use measures 1-10 Mbits/second
         • Rack space: costs $800 -$1200/month, and
           drops by 20% if > 75 to 100 racks (1 20 amp
           circuit)
             – Each additional 20 amp circuit per rack costs another $200
               to $400 per month
         • PG&E: 12 megawatts of power, 100,000 sq.
           ft./building, 10 sq. ft./rack => 1000 watts/rack
                                                                             EE898
11/12/2004
                                                                            Lec 4.66
                      Google Performance: Total

             • Serving pages: 108 Mbit/sec/month
             • Crawling: 59 Mbit/sec/week, 15 Mbit/s/month
             • Replicating: 260 Mbit/sec/week, 65 Mb/s/month
             • Total: roughly 200 Mbit/sec/month
             • Google’s Collocation sites have OC48
               (2488 Mbit/sec) link to Internet
             • Bandwidth cost per month?
               ~$150,000 to $200,000
             • 1/2 BW grows at 20%/month


                                                           EE898
11/12/2004
                                                          Lec 4.67
                           Google Costs
             • Collocation costs: 40 racks @ $1000 per
               month + $500 per month for extra circuits
             = ~$60,000 per site, * 3 sites
             ~$180,000 for space
             • Machine costs:
             • Rack = $2k + 80 * $1500/pc + 2 * $1500/EN
               = ~$125k
             • 40 racks + 2 Foundry switches @$100,000
               = ~$5M
             • 3 sites = $15M
             • Cost today is $10,000 to $15,000 per TB

                                                        EE898
11/12/2004
                                                       Lec 4.68
               Comparing Storage Costs: 1/2001

             • Google site, including 3200 processors and
               0.8 TB of DRAM, 500 TB (40 racks)
               $10k - $15k/ TB
             • Compaq Cluster with 192 processors,
               0.2 TB of DRAM, 45 TB of SCSI Disks
               (17+ racks) $115k/TB (TPC-C)
             • HP 9000 Superdome: 48 processors,
               0.25 TB DRAM, 19 TB of SCSI disk =
               (23+ racks) $360k/TB (TPC-C)



                                                             EE898
11/12/2004
                                                            Lec 4.69
              Putting It All Together: Cell Phones


             • 1999 280M handsets
               sold; 2001 500M
             • Radio steps/components:
               Receive/transmit
               –   Antenna
               –   Amplifier
               –   Mixer
               –   Filter
               –   Demodulator
               –   Decoder




                                                      EE898
11/12/2004
                                                     Lec 4.70
             Putting It All Together: Cell Phones




             • about 10 chips in 2000, which should shrink,
               but likely separate MPU and DSP
             • Emphasis on energy efficiency
                                                               EE898
11/12/2004
                                                              Lec 4.71
                      Cell phone steps (protocol)

             1. Find a cell
                •   Scans full BW to find stronger signal every 7 secs
             2. Local switching office registers call
                •   records phone number, cell phone serial number,
                    assigns channel
                •   sends special tone to phone, which cell acks if correct
                •   Cell times out after 5 sec if doesn't get supervisory
                    tone
             3. Communicate at 9600 b/s digitally (modem)
                •   Old style: message repeated 5 times
                •   AMPS had 2 power levels depending on distance (0.6W
                    and 3W)


                                                                               EE898
11/12/2004
                                                                              Lec 4.72
             Frequency Division Multiple Access
                          (FDMA)

      • FDMA separates the
        spectrum into distinct
        voice channels by
        splitting it into uniform
        chunks of bandwidth
      • 1st generation analog




                                                   EE898
11/12/2004
                                                  Lec 4.73
                Time Division Multiple Access
                           (TDMA)
      • a narrow band that is 30 kHz
        wide and 6.7 ms long is split
        time-wise into 3 time slots.
      • Each conversation gets the
        radio for 1/3 of time.
      • Possible because voice data
        converted to digital
        information is compressed so
      • Therefore, TDMA has 3
        times capacity of analog
      • GSM implements TDMA in a
        somewhat different and
        incompatible way from US
        (IS-136); also encrypts the
        call

                                                 EE898
11/12/2004
                                                Lec 4.74
               Code Division Multiple Access
                          (CDMA)
  • CDMA, after digitizing data,
    spreads it out over the entire
    bandwidth it has available.
  • Multiple calls are overlaid over
    each other on the channel, with
    each assigned a unique sequence
    code.
  • CDMA is a form of spread
    spectrum; All the users transmit
    in the same wide-band chunk of
    spectrum.
  • Each user's signal is spread over
    the entire bandwidth by a unique
    spreading code. same unique code
    is used to recover the signal.
  • GPS for time stamp
  . Between 8 and 10 separate calls
    space as 1 analog call
                                                EE898
11/12/2004
                                               Lec 4.75
         Cell Phone Towers




                              EE898
11/12/2004
                             Lec 4.76

								
To top