(The Case for) Methodology Research -- part 2 by juanagui

VIEWS: 18 PAGES: 45

									     (The Case for)
Methodology Research
      -- part 2
      Indranil Gupta
       Oct 28, 2004
    CS598IG Fall 2004
          (recall) Definition


Protocol Design Methodology =
 An organized, documented set of building
  blocks, rules and/or guidelines for design
  of a class of protocols, possibly amenable
  to automated code generation.


                              [adapted from FOLDOC]
   Methodologies Seen So Far
• Differential Equations  Equivalent Probabilistic
  Distributed Protocols
  – Endemic replication, majority voting, epidemics
  – DiffGen for automatic code generation
  – innovative, formal, progressive
• Routing Design [Maltz et al]
  – composable, formal, retroactive
• Survivable Storage Architectures
  – composable, informal, progressive

• Today: More Composable Methodologies.
  – Building blocks + composition guidelines/rules
     OSI 7-Layer Protocol Stack

                       Message Sent                                  Message Received
OSI Layered Protocol




                        Application

                        Presentation

                        Session
       Model




                        Transport

                        Network

                        Data link

                        Physical
                                             Communication Channel

                                      [formal, composable, progressive]
  OSPF Monitoring [Shaikh et al]
OSPF uses links state advertisements (LSAs)
Monitoring architecture has three components
1. LSAReflector: captures LSAs from network
2. LSA aGgregator: receives a stream of LSAs
    from >=1 LSARs, and performs real-time
    analysis. Maintains a picture of network
    topology.
3. OSPFScan: offline analysis of LSA archives
Information flow: 1  (stream) 2  (storage) 3
     Balanced Binary Trees
              for
ID Management & Load Balance
   in DHTs [G.S.Manku et al]



    (slides adapted from Gurmeet’s PODC presentation)
DHT Design (I): Flat Address
         Space

                                                        Chord,Pastry,Kademlia,..
                                                        (Kelips not included)




          0.00      0.010   0.011   0.100   0.101   0.110   0.111


         0                                                          1


Each host acquires an ID in [0, 1) and “manages” a “partition”.
DHT Design (II): Structured
        Overlay
  Problem: insert/lookup/delete hash entries?

  Issue:     no global knowledge of who manages what!

  Solution: an overlay of TCP/UDP “links”
             insert/lookup/delete hash entries: By “routing”




     0.00       0.010   0.011   0.100   0.101   0.110   0.111
 Overlay = “Circle” + “Routing
          Network”

0.00       0.010   0.011   0.100   0.101   0.110   0.111




                                                                    0.00
TWO types of overlay links
                                                    0.111
        “Circle”
           for a strong core
           for “correctness”                                                        0.010
                                                   0.110
        “Routing Network”
          for short routes                                                  0.011
                                                            0.101
          for “efficiency”
                                                                    0.100
  A Modular Framework
                      Choice of Topology
            Circle      Emulation Engine
               ID Assignment


Why “modular”?
 „Cleaner‟ system-design
 Different researchers to focus on different sub-problems
 Enables meaningful „comparisons‟ among alternatives
 Allows us to pick the „best‟ algorithm for each sub-problem

Existing designs are „monolithic‟ (topology-centric)

“ID Assignment” and “Choice of Topology” are decoupled

composable, informal, retroactive/progressive
Probabilistic I/O Automata and
         Composition
           [Wu et al]
Example Scenario




        Peer to peer file sharing app over a DHT
        Multiple nodes accessing given file
        When black node makes update, it asks
                  other nodes to commit changes
     I/O Automaton for Red Node

                       do update




                                                 Prov.
      Init                Exec                                       Commit
                                                Commit
             receive               reply to              commit
             update                black node            update
             req.                                        to local copy

                                     Red Node: Resulting algorithm/protocol
                                     can be represented as a state machine
[2 phase commit Protocol]
                                      or I/O automata

                                     Easy to analyze and prove properties about
                                      (compared to code specification)
                                                         [Lynch and Tuttle]




                 do update




                                           Prov.
Init                Exec                                       Commit
                                          Commit
       receive               reply to              commit
       update                black node            update
       req.                                        to local copy
                          do update

FILE 1
                                        [1 phase commit Protocol]

         Init                Exec                                        Commit
                receive               reply to
                update                black node,
                req.                  commit
                          do update
                                           [COMPOSE]

FILE 2

                                                     Prov.
         Init                Exec                                        Commit
                                                    Commit
                receive               reply to               commit
                update                black node             update
                req.                                         to local copy
[Wu et al]
  Prob. I/O Automata Composition
• Composition done through delay functions
  (memoryless)
  – Compose states as in I/O automata
  – For an input event, probability of transitioning to a
    given next (composed) state is product of probabilities
    of component transitions
  – For finding which next local event takes place, fastest
    component automaton (shortest delay) determines
    this
  – The corresponding local action from that automaton is
    executed
(Automaton with shorter delay has higher value of
delta)
  Different Notions of Composition
• Theorem results in paper show conditions for
  equivalence of different automata

• These are only limited notions of composition
• The authors do not justify the practicality of defining
  these, but they are used in consensus /probabilistic
  consensus algorithms.
• The prob. I/O automata are similar to the probabilistic
  protocol state machines generated by differential
  equations
   – Input, output (e.g., sampling,tokenizing), local (e.g., flipping)
     actions
   – However, the notion of composition is different
• Next: composition for probabilistic protocols (e.g.,
  epidemics, endemics, …)
Composition of Probabilistic
       Protocols
  Distributed Protocols Research
            Group, UIUC
New Peer to peer Systems
Kelips: a new peer to peer system




                  •Fully Decentralized
                  •Uses replication and structure to
                  ensure constant lookup times
                  •100,000 nodes and 10 Million files
                  2 MB and 2 Kbps per node!
                  Epidemic Protocols used to
                  replicate information
                     Composition techniques:
                   “Constraining”, “Fashioning”



                            Uniform
                           Epidemics
      Kelips –
   P2P resource              Weak
Location & discovery                      (affinity groups)
                            Overlay
      [IPTPS 03]
                            Topology
                                           (round-trip times)
                           Awareness
                Composition techniques:
              “Constraining”, “Fashioning”



                       Uniform
                      Epidemics
    Kelips for
sensor networks:        Weak
  replace one                        (affinity groups)
                       Overlay
   component
                       Topology
                                      (hop counts)
                      Awareness
                        Repair/
Building Blocks                                  Deterministic
                       Recovery
                                                    Layer


        Topology
                      Uniform
       Awareness
                     Epidemics
                                   K-committee     Probabilistic
                                                      Layer
            Random
             ping         Weak
                         Overlay
                                      Tree Diss.
 (Recap) Epidemic Dissemination

[based on previous literature – Demers 87]
A node periodically gossips to a few other nodes
  picked uniformly at random

Advantages
  –   Overhead [ fanout b ] and [log N gossip rounds]
  –   High fault-tolerance – non-receipt prob. [ o( 1 2b ) ]
                                                     N
  –   [ O(log N ) ] dissemination latency
  –   Eventual Dissemination of updates
 Uniform    (uniform target choices)
Epidemics
                  Composition technique:
                “Fashioning augmentation”




                        Uniform       (uniform target choices)
gossip target          Epidemics
  choices
weighted by             Topology
                                      (round-trip times)
                       Awareness
                   Composition technique:
                 “Constraining augmentation”




                          Uniform      (uniform target choices)
   gossip                Epidemics
 constrained
within overlay             Weak        (virtual graph
                          Overlay      among nodes)
                Composition technique:
                  “Template/Modular”
       (merge main functions of two periodic building blocks)




                          Random
                                              (uniform ping choices)
 SWIM                      ping
[DSN 2002]
                          Uniform
                         Epidemics            (uniform target choices)
                     Composition techniques:
                   “Constraining”, “Fashioning”



                            Uniform
                           Epidemics
      Kelips –
   P2P resource              Weak
Location & discovery                      (affinity groups)
                            Overlay
      [IPTPS 03]
                            Topology
                                           (round-trip times)
                           Awareness
   Preservation of Properties
      under Composition
• Correctness: Liveness inherited under all
  compositions
• Performance: Latency, Reliability, Member
  overhead
  – inherited under template composition
  – Constraining and fashioning can be applied to
    “approximately” preserve these properties while
    improving other metrics
• Automatic Code Generation: use existing C
  source code of blocks AND a new language
  called PPCL [Thompson and Gupta 04]
                Repair/                  Deterministic
               Recovery
                                            Layer


 Topology
              Uniform
Awareness
             Epidemics
                           K-committee     Probabilistic
                                              Layer
    Random
     ping         Weak
                 Overlay
                              Tree Diss.
Group Membership               Deterministic
                                  Layer




                   Random
   SWIM             ping        Probabilistic
                                   Layer
  [DSN 2002]
                    Uniform
                   Epidemics
Peer-to-peer Resource Location    Deterministic
                                     Layer


                       Uniform
                      Epidemics
  Kelips                           Probabilistic
                         Weak
                                      Layer
 [IPTPS 2003]           Overlay

                       Topology
                      Awareness
                                   Deterministic
                                      Layer


                     Uniform
                    Epidemics
   Uniform
  Epidemics          Topology
                    Awareness        Probabilistic
 K-committee                            Layer
                       Weak
                      Overlay
Leader Election

   [DISC 2000]    Data Aggregation

                      [DSN 2001]
                   Virtually Synchronous     Repair/
                           Mulicast
                          [QREI 2002]       Recovery

                       Uniform              Uniform
                      Epidemics            Epidemics

                       Topology             Topology
  Uniform             Awareness            Awareness
 Epidemics               Weak                 Weak
    Bimodal             Overlay              Overlay
   Multicast
[previous work]        Tree Diss.           Tree Diss.

                  Adaptive Topological
                        Mulicast           K-committee
                       [SRDS 2002]
                    Summary

• Probabilistic protocols to implement important services
  for large-scale distributed applications
   – Membership, Resource Location, Multicast,
      Aggregation, Election
• Protocols constructed by composing building blocks
• Have low overhead and probabilistic reliability
• Scale robustly
   – Simulations: up to 64,000 nodes
   – PC clusters: 60 to 100 nodes

• composable, informal  formal (PPCL),
  retroactive
Kelips-based                                      PAST,Squirrel, CFS,…
Web caching

  A New Formal Composable Methodology for P2P DHTs and DHT-based apps
                                                                 \end{minipage}
         /* Blocks to be called by App-1. */                     \begin{minipage}{2.9in}
         component {                                             {\tiny
             file = dht.c;                                       \begin{verbatim}
             function = char *get(void *);                           function = void run();
         } lookup;                                               } membership;
         component {                                             component {
             file = dht.c;                                           file = diss.c;
             function = char *put(void *, char *);                   function = void run();
         } insert;                                               } dissemmination;

         /* Blocks with call-in functions. */                    /* Composition -- Merging membership and dissemination. */
         component {                                             ...
             file = overlay.c;                                   template {
             function = get_neighbor(int dest);                       component = membership;
         } overlay;                                                   component = dissemination;
         ...                                                          match (int send_member_update(),
                                                                           int send(),
         /* Compositions -- Augmentation. */                               int send(),
         augment {                                                         piggyback);
             target = lookup;                                         match (int process(char *recv_msg),
             replace(int forward_search(void *data), overlay);             int process(char *recv_msg),
         } lookup-overlay;                                                 int process_join(char *piggy_backed, char *recv_msg),
              // replace each call to forward_search in lookup             unpiggyback);
              // with call to main function of overlay                match (recvmsg, in_buff, recvmsg);
         ...                                                     } membership-dissem;
                                                                 // Merges the main functions of membership.c and dissemination.c.
         /* Periodically Executed Blocks. */                     // For each match statement, a call to the first function
         component {                                             // (in membership) and the second function (in dissemination)
             file = swim.c;                                      // are both merged (top-down in the code) and replaced with
         \end{verbatim}                                          // a call to the third function. The fourth (optional) function
         }                                                       // is user-defined and merges arguments from the first
                                                                 // two calls for the third function call.




Automatic Code Generation from original source code and PPCL specification
                       [Thompson and Gupta, 04]
              Back to the Big Picture
    •    We set out to create new resources for the protocol
         designer
         – beyond research literature and experience
         – help in systematic design of protocols

    Our approach:
    1. Methodologies: E.g., [composable] to reuse protocol
       design. [innovative] To translate differential equation
       systems into equivalent protocols.
    2. Automation: E.g., [composable] PPCL. [innovative]
       DiffGen toolkit (PODC 2004 poster). Spews out
       ready-to-deploy code.

[Distributed Protocols Research Group, UIUC]   http://www-faculty.cs.uiuc.edu/~indy/rsrch.htm
               More to be Done
• Discovery of Methodologies
  – Retroactive: underlying the Web?
  – Progressive: for fresh classes of novel protocols?
  – Auxiliary: to improve existing and emerging systems?
• Many different kinds
  – Formal
  – Informal
  – Hybrid

• Think: one level of abstraction up
                Implementation + Experimentation



  Sensor Networks                        Internet-based distributed
                                                  systems




                    The Act of Protocol Design




Design Methodologies                        Theoretical Work

            Distributed Protocols Research Group

								
To top