Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

extreme

VIEWS: 4 PAGES: 52

									Emergence of Extreme Networked
           Devices

           David Culler
     Computer Science Division
          U.C. Berkeley
    www.cs.berkeley.edu/~culler


             USC, Feb 28, 2001
The Expanding Computing Spectrum


• Internet Services

 • Servers

 • Workstations

 • Personal Computers

 • PDAs / HPCs/ smartphones




2/28/2001               Emerging Extremes   2
Convergence at the middle
• Common platform
      – powerful microproc (choice of 3), dram (3), disk (2)
      – deep I/O hierarchy, OS layering
• Common system abstraction
      – collection of threads sharing a large virtual address space
      – GUI orientation
      – blocking interfaces
• Concurrency as threads
• Services as local call / remote thread
      – RPC, rmi, dCOM, http
• Ample resources easily abstracted
      – open loop
      – transparent allocation and usage


2/28/2001                   Emerging Extremes                         3
Emerging Extremes
 • Planetary Services
 • Open Internet Services
 • Internet Services

 • Servers
                                            svr
 • Workstations

 • Personal Computers

 • PDAs / HPCs/ smartphones

 • Microscopic sensor networks


2/28/2001               Emerging Extremes     4
Convergence at the Extremes
• Concurrency intensive
      – data streams and real-time events, not command-response
• Communications-centric
• Limited resources (relative to load)
• Huge variation in load
      – population usage & physical stimuli
      – robustness
• Hands-off (no UI)
• Dynamic configuration, discovery
      – Self-organized and reactive control
• Similar execution model
      – event driven,
      – components
• Complimentary roles
      – tiny semi-autonomous devices empowered by infrastructure
      – infrastructure services connected to the real world

2/28/2001                     Emerging Extremes                    5
Outline
• Emerging Extremes
• Robust Framework for Open Scalable Internet
  Services
      – the garden path: threads to non-block I/O and RPC
      – structured event-driven alternatives
      – controllers within a graph of stages
• Tiny OS for Wireless Embedded Sensor
  Networks




2/28/2001                  Emerging Extremes                6
Ninja: Open Infrastructure Services

                       Open
 Clients            Infrastructure
 Clients               Services
  Clients                                            Servers
  Clients
 Clients
 Clients
                                                         Servers


                                                     Servers



 systematic framework for building robust, composable
  services
 focus here on execution model

2/28/2001              Emerging Extremes                       7
 Variation in Load – slashdot effect




USGS Web Server Traffic
October 16, 1999 Hector Mine Earthquake


http://pasadena.wr.usgs.gov/stans/slashdot.html
 2/28/2001                     Emerging Extremes   8
Inherent Variation: Gnutella Router
Traffic




2/28/2001      Emerging Extremes   Matt Welsh   9
Toward Robust Behavior Under Load
• Traditional Capacity Planning
      – over-provision by factor over typical (increasing 4 -> 10-15)
      – cluster-based replication is, at least, cost-effective
      – peaks occur when it matters most
• Content-distribution
      – potential replication proportional to use
• Still want graceful degradation when instance is
  overloaded                response-time (s)


                                           thru-put (op/s)




                        load
2/28/2001                      Emerging Extremes                        10
Threads as THE building block

Remote Services       Masking I/O Latency




 • Freely compose these two primitives
 • But,... threads a limited resource

2/28/2001          Emerging Extremes        11
    Service “test problem”

                                               • A: popularity
                   task arrivals
                   rate: A tasks / sec         • L: I/O, network, or service
                                                 composition depth

 closed loop
                  Threaded                  dispatch( ) or create( )
implies S = A
                   server



                                                latency: L sec



                                                    # concurrent tasks in
                task completions                    server: T = A x L
                 rate: S tasks / sec
    2/28/2001                          Emerging Extremes                    12
     Threads are a “limited resource”
          • Fix L = 10 ms, for each T measure max A = S
                         2500
                                    1-w ay Java
m ax server throughput




                         2000       4-w ay Java
     (S tasks/sec)




                         1500

                         1000

                         500

                           0
                                1     10                 100                   1000            10000
                                           # threads executing in server (T)
          • Cluster parallelism just raises the threshold
                                                           ultra 170 and E450, Solaris 7.2, jdk 1.2.2
     2/28/2001                               Emerging Extremes                                    13
 Alternative: queues, events, typed msgs
                • single-threaded server
                • queues absorb load and decouple
                  operations
                   – svr chooses when to assign resources to
                     request event
                • bounded resources at request
                  interface
                   – impose load-conditioning or admission
                     control
  Explicit      • provide non-blocking interface
request queue
                • client retains control of its thread
                   – chooses when to block
                   – permits negotiation protocol
                   – key to service composition



  2/28/2001      Emerging Extremes                     14
Event-per-task saturates gracefully
                           6000
   max server throughput




                           5000       1-way Java
       (S tasks/sec)




                           4000       4-way Java

                           3000

                           2000

                           1000

                             0
                                  1        10                 100                   1000   10000
                                                # tasks in client-server pipeline
• Better and more robust performance
             – Use cluster parallelism to match desired thruput
• Can decompose task into multiple events
             – circulate or pipeline
• but ...
2/28/2001                                          Emerging Extremes                               15
Down-side of monolithic event approach
 • Lose familiar programming model
       – thread steps through each stage in the task
       – need a handler per stage
 • Difficult software engineering
       – composing and scheduling



 • Does not naturally exploit SMP parallelism
       – must pipeline multiple event handler blocks
 • Whenever the thread blocks, the whole structure
   stalls
       – throughput ~ 1/L


 2/28/2001                   Emerging Extremes         16
State-of-practice: bounded thread pool
            task arrivals
        rate: A tasks / sec
                                          • Only allow K
                                            threads to “accept”
                                            connections
                                                  – some OS’s have fixed
       Threaded                                     hard limit
        server
                                          • Additional requests
                                            time-out
                                          • choose K < Tmax xput
                                          • choose K large
                                            enough to hide L
    task completions
      rate: S tasks / sec


2/28/2001                     Emerging Extremes                            17
A “third road”
• Building block
      – bounded internal thread pool
      – queue-based interface
      – subset of task stages
          » request event processing in familiar style
      – can chunk request stream for efficiency
• Compose Service as a graph of “stages”
      – modularity
      – stages can be replicated across nodes
                                  read
                   read             cache
                                  header
        read         cache          miss
                   header                        read
          read
        header       check                         write
                                                 header
          header
                                                   resp
                   read
                     exec
                   header

• Stage “control loop” manages threads
2/28/2001                    Emerging Extremes             18
Well-conditioned Service Architecture
• Abstract System I/F as non-blocking stages
      – careful engineering at the system interface
• Describe stages as modular state machines
• Associate thread manager with stages
• Build Service as composition of stages
      – can be dynamic




                                                      Matt Welsh
2/28/2001                   Emerging Extremes                19
example: http throughput


                                  1100
                                  1000
                                   900
            Requests per second




                                   800
                                   700                                           SEDA
                                   600                                           Apache
                                   500
                                   400
                                   300
                                   200
                                   100
                                     0
                                         0   200     400    600     800   1000
                                                        # Clients



     SPECweb99 static workload, 4 classes

2/28/2001                                          Emerging Extremes                      20
Response Time

                                 7000


                                 6000


                                 5000
            Response Time (ms)




                                                                               SEDA mean
                                 4000
                                                                               Apache mean
                                                                               SEDA std dev
                                 3000
                                                                               Apache stdev

                                 2000


                                 1000


                                   0
                                        0   200   400       600   800   1000
                                                        Clients




2/28/2001                                         Emerging Extremes                           21
Reactive Stage Thread Pool Sizing
                     Two Packet Types
                           ping – fast
                           query – 20 ms delay




                   3455       6442
            50
            45
            40                                              clients
            35
                                                   29.36       100
            30
  Latency




                                                               200    Thread Governor
            25
                                                               400
            20
                                                               1000
                                                                       - observes queue length
            15                          10.934
            10
                                                                       - over threshold => add threads
            5
            0
                 ST-ping    ST-query   TG-ping   TG-query


2/28/2001                                        Emerging Extremes                                22
Scalable Persistent Data Structures
   Clustered Service
                                                      distributed hashtable
            Service Service   Service                     “RPC” skeletons


            DDS lib DDS lib   DDS lib
                                                    single-node
                                                        HT
               System Area Network

Storage Storage Storage StorageStorage Storage
                                                       buffer

“brick” “brick” “brick” “brick” “brick” “brick”
                                                       cache




                                                      I/O core           I/O core
                                                        disk             network




Steve Gribble                                           operating system

                                                                  DDS Brick
2/28/2001                       Emerging Extremes                              23
                         Scalable Throughput

                         100000
                                                                            (128,61432)
max throughput (ops/s)




                                                                            (128,13582)
                          10000
                                         reads


                           1000
                                                  writes


                             100
                                     1            10                  100                  1000
                                                    # of DDS bricks

                         2/28/2001               Emerging Extremes                        24
                                  Robust under load
hash table throughput (reads/s)




                                  20000
                                                            2 bricks
                                                            8 bricks
                                  16000
                                                            16 bricks

                                  12000                     32 bricks


                                   8000


                                   4000

                                        0
                                              0         5         10         15      20        25       30
                                                                    # client process
                                  2/28/2001                       Emerging
                                                  (100 parallel requests Extremes per client process)
                                                                           issued                       25
Outline
• Emerging Extremes
• Robust Framework for Open Scalable
  Internet Services
      – modular generalized state machines
      – constrained use of threads
      – thread-manager as controller
• Tiny OS for Wireless Embedded Sensor
  Networks
      –     characteristics of the other extreme
      –     current platforms
      –     events and primitive threads in a graph of components
      –     exploring open problems
2/28/2001                    Emerging Extremes                      26
 Emerging Microscopic Devices
• CMOS trend is not just Moore’s law
• Micro Electical Mechanical Systems (MEMS)
   – rich array of sensors are becoming cheap and tiny




• Low-power Wireless Communication
                                                          I SDQ SD

• Imagine, all sorts of chips                                   baseband
                                                         PLL      filters
that are connected to the                                      mixer
physical world and to                                          LNA
cyberspace!
 2/28/2001                 Emerging Extremes                      27
Characteristics of Network Sensors
• Small physical size and low power consumption
• Concurrency-intensive operation
      – flow-thru, not wait-command-respond
• Limited Physical Parallelism and Controller
  Hierarchy
      – primitive direct-to-device interface        actuators   sensors
• Diversity in Design and Usage                                       storage
      – application specific, not general purpose
      – huge device variation                           network
      => efficient modularity
      => migration across HW/SW boundary
• Robust Operation
      – numerous, unattended, critical
      => narrow interfaces

2/28/2001                    Emerging Extremes                        28
Current Example
• 1” x 1.5” motherboard
    –   ATMEL 4Mhz, 8bit MCU, 512 bytes RAM, 8K pgm flash
    –   900Mhz Radio (RF Monolithics) 10-100 ft. range
    –   ATMEL network pgming assist
    –   Radio Signal strength control and sensing
    –   I2C EPROM (logging)
    –   Base-station ready
    –   stackable expansion connector
          » all ports, i2c, pwr, clock…
• Several sensor boards
    –   basic protoboard
    –   tiny weather station (temp,light,hum,press)
    –   vibrations (acc, temp, ...)
    –   accelerometers
    –   magnetometers




2/28/2001                       Emerging Extremes           29
 Basic Power Breakdown…
                    Active         Idle             Sleep

      CPU           5 mA           2 mA             5 μA
      Radio         7 mA (TX)      4.5 mA (RX)      5 μA
      EE-Prom       3 mA           0                0
      LED’s         4 mA           0                0        Panasonic
                                                              CR2354
      Photo Diode   200 μA         0                0
                                                              560 mAh
      Temperature   200 μA         0                0

• But what does this mean?
   – Lithium Battery runs for 35 hours at peak load and years at
     minimum load!
       » three orders of magnitude difference!
   – A one byte transmission uses the same energy as approx
     11000 cycles of computation.
   – Idleness is not enough, sleep!
 2/28/2001                      Emerging Extremes                  30
A Operating System for Tiny Devices?
• Traditional approaches
      – command processing loop (wait request, act, respond)
      – monolithic event processing
      – bring full thread/socket posix regime to platform
• Alternative
      –     provide framework for concurrency and modularity
      –     never poll, never block
      –     interleaving flows, events, energy management
      –     allow appropriate abstractions to emerge




2/28/2001                    Emerging Extremes                 31
Tiny OS Concepts
• Scheduler + Graph of
  Components                                         Commands                                      Events




                                                                                                        msg_rec(type, data)
    – constrained two-level scheduling




                                                                   power(mode)
      model: threads + events




                                                                                   type, data)
                                                                                   send_msg
• Component:




                                                                                                        msg_sen
                                                                                   (addr,




                                                                                                        d_done)
                                                   init
    –   Commands,
    –   Event Handlers
    –   Frame (storage)                                   Messaging Component
                                                                                                   Internal
    –   Tasks (concurrency)




                                                  TX_packet(buf)
                                                                                 internal thread     State




                                                  Power(mode)
• Constrained Storage Model
    – frame per component, shared stack, no




                                                  init




                                                                                                   RX_pack
                                                                                                   TX_pack

                                                                                                   (success
      heap




                                                                                                   et_done

                                                                                                   et_done
                                                                                                   (buffer)
• Very lean multithreading




                                                                                                   )
• Efficient Layering

2/28/2001                     Emerging Extremes                                                               32
               Application = Component Graph
 application




                  Route map         router        sensor appln




                           Active Messages
packet




                Radio Packet     Serial Packet   Temp     photo
                                                                      SW


                                                                      HW
byte




                 Radio byte       UART
                                                    ADC
                                                                           Example: ad hoc, multi-hop
                                                                           routing of photo sensor
                                                                  clocks
                                                                           readings
bit




                 RFM




               2/28/2001                          Emerging Extremes                            33
 TOS Execution Model
• commands request action
                                                application comp          data processing
    – ack/nack at every boundary
    – call cmd or post task                                     message-event driven

• events notify occurrence                      active message

    –   HW intrpt at lowest level                                event-driven packet-pump

    –




                                       packet
        may signal events                        Radio Packet       crc
    –   call cmds
    –   post tasks                                                event-driven byte-pump

• Tasks provide logical                          Radio byte

                                       byte
                                                                 encode/decode
  concurrency                                                   event-driven bit-pump
    – preempted by events
                                                 RFM
                                       bit




• Migration of HW/SW
      boundary

 2/28/2001                     Emerging Extremes                                  34
    Dynamics of Events and Threads
                     bit event =>
                        end of byte =>
                                                      thread posted to start
bit event filtered        end of packet =>
at byte layer               end of msg send           send next message




                                          radio takes clock events to detect recv
    2/28/2001                     Emerging Extremes                             35
Storage Breakdown (C Code)

  4000


                            Multihop Router
  3500
                            AM light
                            AM Temp
                                               3450 B code
  3000
                            AM                  226 B data
                            Packet
  2500                      Radio Byte
                            RFM
  2000                      Photo
                            Temp
  1500                      UART Packet
                            UART
                            i2c
  1000
                            Init
                            TinyOS Scheduler
   500
                            C Runtime

     0



2/28/2001    Emerging Extremes                           36
Empirical Breakdown of Effort
                                  Packet reception Percent CPU
            Components            work breakdown Utilization      Energy (nj/Bit)
            AM                           0.05%           0.20%             0.33
            Packet                       1.12%           0.51%             7.58
            Radio handler               26.87%          12.16%           182.38
            Radio decode thread          5.48%           2.48%             37.2
            RFM                         66.48%          30.08%           451.17
            Radio Reception                   -               -            1350
            Idle                              -         54.75%                -
            Total                      100.00%         100.00%          2028.66



• can take apart time, power, space, …
• 50 cycle thread overhead, 10 cycle event overhead


2/28/2001                             Emerging Extremes                             37
Working Across Levels
• Encoding
      – DC-balanced SECDED
• Proximity detection
      – signal strength or error rates
•   Low power listening
•   Fair and efficient network access
•   Security
•   Tiny virtual machines
•   Larger challenges




2/28/2001                    Emerging Extremes   38
Low-Power Listening
• Costs about as much to listen as to xmit, even when
  nothing is received
• Only way to save power is to turn radio off when there is
  nothing to hear.
• Can turn radio on/of in about 1 bit
      – Can detect transmission at cost of ~2 bit times
 Small sub-msg recv sampling (10x)
                   sleep
 Xmit:                                       preamble     message
               b
 Recv:



 Application-level synchronization rendezvous to
 determine when to sample (10X)
                                                              Jason Hill
2/28/2001                      Emerging Extremes                           39
Managing local contention
• Highly correlated traffic, no collision detection
      – sensor events and beacons
• Randomize initial listen period, simple backoff
                       Channel Utilization and % Throughput per Mote at 4 packet/s
                                                Duty Cycle
                                                            Channel Utilization ~70%
                   1
                 0.9
                 0.8
                 0.7
                 0.6
             %




                 0.5
                 0.4
                 0.3
                 0.2
                 0.1
                   0
                          1      2     3      4      5      6       7   8     9      10
                                                  Number of Motes
                                                                            Throughput
2/28/2001                            Emerging Extremes
                                                                            per node is
                                                                                      40
            Alec Woo
                                                                            fair
Managing aggregate contention




• Hidden nodes between each pair of “levels”
      – CSMA is not enough
• RTS/CTS acks too costly (power & BW)
• P[msg-to-base] drops rapidly with hops
      – Investment in packet increases with distance
 Local rate control to approx. fairness
       Priority to forwarding, adjust own data rate
       Additive increase, multiplicative decrease
 Listen for retransmission as ack
~ ½ of packets get through 4 levels out

2/28/2001                      Emerging Extremes       41
 Authentication / Security

• RC-5 shared key
  crypto in 1.7 kb
                                 Energy costs of a secure channel
• Modified Tesla                     encryption
                                                 freshness
                                                              freshness
  protocol for                     transmission
                                                computation
                                                     0%     transmission
                                        0%                        7%
  confidential &         encryption
                        computation
                                                                        MAC
                                                                    computation
  authenticated base        0%                                           2%
  broadcast                                                                MAC
                                                                       transmission
• Easy to compromise a                                                     20%

  node, but hard to get
                                  data
  most of them                transmission
                                            71%




 2/28/2001                        Emerging Extremes                               42
What’s in a program?
• HW + collection of components supports space
  of applications
• Application-Specific Virtual Machine
      –     code-density, not portability
      –     small byte-code interpreter component
      –     accepts clock & message event capsules
      –     Hides split-phase operations below interpreter
• Capsules define specific query / logic
      – filter criteria
      – diffusion primitives
      – ...




2/28/2001                       Emerging Extremes            43
Thoughts about robust Algorithms
• Active Dynamic Route Determination
      – When hear a new route beacon, record “parent”, retransmit
        from SELF, ignore additional messages for epoch
•   Radio cell structure very unpredictable
•   Builds and maintains good breadth-first forest
•   Each node maintains O(1) state
•   Fundamental operation is pruning
    retransmission
      – Monotonic variables
      – Message signature caches
• Takes energy to retain structure


2/28/2001                  Emerging Extremes                        44
Larger Challenges
• Programming support for systems of generalized
  state machines
      – language, debugging, verification
• Programming the unstructured aggregate
• Resilient Aggregators
• Understanding how an extreme system is
  behaving and what is its envelope
      – adversarial simulation




2/28/2001                   Emerging Extremes   45
Tides of Innovation

                                                          Innovation

                                                          Integration
                           Personal Computer
                           Workstation
                           Server
Log R

                   Minicomputer



            Mainframe




                             Time                2/2001

2/28/2001                    Emerging Extremes                    46
Summary
• The extremes of the computing spectrum
  present tremendous opportunities for innovation
• Systems challenges
      – variation in load, unpredictability, hands-off embedded
        operation
      – limited resources, concurrency intensive, power constrained
      – self-organizing and adaptive
• More in common with each other than with the
  “average” devices
• New kinds of software system structures
      – modular event-driven structures
      – intrinsic feedback and control



2/28/2001                  Emerging Extremes                          47
2/28/2001   Emerging Extremes   48
Historical Perspective
• New eras of computing start when the previous
  era is so strong it is hard to imagine that things
  could be different
      – mainframe -> mini
      – mini -> workstation -> PC
      – PC -> ???
• It is often smaller than what came before.
      – Most think of the new technology as “just a toy”
• The new dominant use was almost completely
  absent.

• it is likely to come from the extremes

2/28/2001                   Emerging Extremes              49
Mean Response Time(A)

                                    10000
                                                      L = 10ms
     end-to-end task latency (ms)



                                                      L = 50ms
                                     1000             L = 200ms


                                      100


                                       10


                                        1
                                            1        10                100        1000
                                                server throughput (S tasks/sec)


• closed system, but limited bandwidth
2/28/2001                                          Emerging Extremes                     50
Threaded non-blocking disk-read
service




2/28/2001     Emerging Extremes   51
Example: Disk-read “Stage”




2/28/2001     Emerging Extremes   52

								
To top