The essence of P P A reference architecture for overlay networks by benbenzhou


More Info
									              The essence of P2P: A reference architecture for overlay networks∗

                   ,                ,           ,                      ,            ,
        Karl Aberer† Luc Onana Alima‡ Ali Ghodsi§ Sarunas Girdzijauskas† Seif Haridi§ Manfred Hauswirth†
                         e e
    Ecole Polytechnique F´ d´ rale de
    †                                                   ‡
                                                            Universit´ de Mons-Hainaut             §
                                                                                                       Swedish Institute of Computer
           Lausanne (EPFL)                                            (UMH)                                  Science (KTH)
    CH-1015 Lausanne, Switzerland                            B-7000 Mons, Belgique                       S-164 29 Kista, Sweden

                             Abstract                                       tifiers. On top of this resource location service different ap-
                                                                            plication services can be realized, such as data management
    The success of the P2P idea has created a huge diversity                (search, insert, update, etc.). In principle, distributed appli-
of approaches, among which overlay networks, for example,                   cation services could also use directly the physical network-
Gnutella, Kazaa, Chord, Pastry, Tapestry, P-Grid, or DKS,                   ing layer for managing their resources, but using an over-
have received specific attention from both developers and                    lay network has the advantage of supporting application-
researchers. A wide variety of algorithms, data structures,                 specific identifiers and semantic routing, and offers the pos-
and architectures have been proposed. The terminologies                     sibility to provide additional, generic services for support-
and abstractions used, however, have become quite incon-                    ing network maintenance, authentication, trust, etc., all of
sistent since the P2P paradigm has attracted people from                    which would be very hard to integrate into and support at
many different communities, e.g., networking, databases,                    the networking layer. The introduction of overlay networks
distributed systems, graph theory, complexity theory, biol-                 and self-management at the service-level are probably the
ogy, etc. In this paper we propose a reference model for                    essential innovations of P2P systems.
overlay networks which is capable of modeling different ap-                    A wide range of algorithms, structures, and architec-
proaches in this domain in a generic manner. It is intended                 tures for overlay networks have been proposed already, inte-
to allow researchers and users to assess the properties of                  grating knowledge from many different communities, such
concrete systems, to establish a common vocabulary for sci-                 as networking, distributed systems, databases, graph the-
entific discussion, to facilitate the qualitative comparison of              ory, agent systems, complex sytems, etc. The terminolo-
the systems, and to serve as the basis for defining a stan-                  gies and abstractions used, however, are quite inconsistent,
dardized API to make overlay networks interoperable.                        which makes it very hard to assess and compare differ-
                                                                            ent approaches. Only a few relevant attempts to remedy
                                                                            this situation exist so far. For example, JXTA [10] de-
1       Introduction                                                        fines a 3-layer architecture (kernel, services, application),
                                                                            XML-based communication protocols, and basic abstrac-
   P2P is not a new paradigm and in fact has already been                   tions, such as peer groups, pipes, and advertisements. JXTA
applied in the original Internet’s design, for example, in ba-              intends to provide a uniform programming platform for P2P
sic Internet routing or in applications such as Usenet News.                applications and facilitate interoperability. It provides well-
What is new, however, is its broad application to all system                structured APIs and a clear separation of concerns in its ar-
layers and to new application domains. Most prominently,                    chitecture but does not mean to describe the structural and
the P2P approach has been applied for resource location                     functional properties of overlay networks as we do in this
by building so-called overlay networks, such as Gnutella,                   paper. Our work and JXTA are thus complementary.
Freenet, Pastry, P-Grid, or DKS, on top of a physical net-                     Dabek et al. [7] propose a common API for structured
work. Basically all these overlay networks provide a re-                    overlays, basically for CAN [16], Chord [18], Pastry [17],
source location service supporting application specific iden-                and Tapestry[20]. The API only takes into account struc-
    ∗ The work presented in this paper was supported (in part) by the Na-   tured overlays and the used abstraction are at a very low
tional Competence Center in Research on Mobile Information and Com-         level (C programming interface level), so that using it as a
munication Systems (NCCR-MICS), a center supported by the Swiss             general architecture for modeling overlay networks is not
National Science Foundation under grant number 5005-67322 and was           possible.
(partly) carried out in the framework of the EPFL Center for Global Com-
puting and supported by the Swiss National Funding Agency OFES as part         In this paper we thus propose a reference model for over-
of the European project Evergrow No 001935.                                 lay networks which is capable of modeling all existing ap-
proaches in this domain. We focus on decentralized over-                 1.   choice of an identifier space
lay networks such as Gnutella [6], Freenet [5], CAN [16],                2.   mapping of resources and peers to the identifier space
Chord [18], P-Grid [1], DKS [4], etc., as this class is the              3.   management of the identifier space by the peers
most relevant one. From a modeling point of view, central-               4.   graph embedding (structure of the logical network)
ized P2P systems, such as Napster, are simply client-server              5.   routing strategy
architectures where the participants can directly communi-               6.   maintenance strategy
cate after a discovery phase (similar to a DNS name lookup                 In taking these design decisions the following key re-
and then contacting a web server, for example). Hierar-                quirements for overlay networks are addressed:
chical P2P systems such as Kazaa, basically consist of a                   Efficiency: Routing should incur a minimum number of
decentralized overlay network of super-peers for locating              overlay hops (with minimum “physical” distance) and the
resources that are used by the normal peers. Thus these                bandwidth (number and size of messages) for constructing
system can be modeled by our proposed model with an ad-                and maintaining the overlay should be kept minimal.
ditional client-server step when contacting a super-peer.
                                                                           Scalability: The concept of scalability includes many
    Our model is intended to support the assessment of sys-            aspects. We focus on numerical scalability, i.e., very large
tem properties, establishes a common vocabulary, facilitates           numbers of participating peers without significant perfor-
the qualitative comparison of the systems, and can serve               mance degradation.
as the basis for defining a standardized API to make over-
                                                                           Self-organization: The lack of centralized control and
lay networks interoperable. The major contributions of our
                                                                       frequent changes in the set of participating peers requires a
model are (1) a conceptual model capturing the concept of
                                                                       certain degree of self-organization, i.e., in the presence of
embedding a graph into a virtual identifier space, which
                                                                       churn the overlay network should self-reconfigure itself to-
is fundamental for all overlay networks and (2) a well-
                                                                       wards stable configurations. This is a stabilization require-
defined peer architecture comprising user-level interfaces
                                                                       ment as external intervention typically is not possible.
for applications wanting to use the overlay, interfaces for
                                                                           Fault-tolerance: Participating nodes and network links
intra-network communication among homogeneous peers,
                                                                       can fail at any time. Still all resources should be accessible
and interfaces for cooperation among heterogeneous over-
                                                                       from all peers. This is typically achieved by some form of
lay networks.
                                                                       redundancy. This is also a stabilization requirement for the
                                                                       same reason as above. Fault-tolerance implies that the par-
2   Conceptual Model for Overlay Networks                              tial failure property of distributed systems [19] is satisfied,
                                                                       i.e., even if parts of the overlay network cease operation, the
    In any overlay network a group of peers P provides ac-
                                                                       overlay network should still provides an acceptable service.
cess to a set of resources R by mapping P and R to an
(application-specific) identifier space I using two functions                Cooperation: Overlay networks depend on the coopera-
FP : P → I and FR : R → I. These mappings estab-                       tion of the participants, i.e., they have to trust that the peers
lish an association of resources to peers using a closeness            they interact with behave properly in respect to routing, ex-
metric on the identifier space. To enable access from any               change of index information, quality of service, etc.
peer to any resource a logical network is built, i.e., a graph             In the following we will provide detailed formal specifi-
is embedded into the identifier space. These basic concepts             cations for these key design concepts of overlay networks
of overlay networks are depicted in Figure 1.                          and discuss the issues related to the requirements listed

                                                                       2.1     Choice of Identifier Space

                            Identifier space                              A central decision in designing an overlay network is the
                                                                       selection of the virtual identifier space I which has to pos-
                                      Structuring strategy
                                                                       sess some closeness metric d : I × I → R, where R de-
                                                Universe of overlays   notes the set of real numbers. d must satisfy properties 1–3
                                                                       below and if possible should satisfy properties 4–5.
                                                                              ∀x, y ∈ I : d(x, y) ≥ 0                            (1)
                                                                                 ∀x ∈ I : d(x, x) = 0                            (2)
                                                                              ∀x, y ∈ I : d(x, y) = 0 ⇒ x = y                    (3)
    Figure 1. Overlay network design decisions                                ∀x, y ∈ I : d(x, y) = d(y, x)                      (4)
                                                                            ∀x, y, z ∈ I : d(x, z) ≤ d(x, y) + d(y, z)           (5)
   Each specific overlay network is characterized by the de-               If d satisfies all the five properties then (I, d) is a metric
cisions made on the following six key design aspects:                  space. However, in many cases only the first three proper-
ties will be satisfied. In this case we call (I, d) a pseudo-                                    • Completeness: FP may be complete or partial. When
metric space.                                                                                      FP is partial, peers might (temporarily) not be associ-
    The choice of the virtual identifier space is important for                                     ated with an identifier.
several reasons:                                                                                • Morphism: If no replication (for fault-tolerance) is re-
   • Addressing: The identifier space plays the role of an                                          quired, FP will be one-to-one (injective), i.e., ∀p, q ∈
     address space used for identifying resources in the                                           P : p = q ⇒ FP (p) = FP (q). However, the more
     overlay network. Each peer and resource in an over-                                           typical case is that the system uses replication and the
     lay network receives a virtual identifier taken from I                                         mapping is not injective.
     (explicitly or implicitly).                                                                • Dynamicity: FP can be either statically defined, e.g.,
   • Scalability: To support very large systems, I has to                                          by its physical address or other unique attributes, or
     be very large. Through a mapping FP each peer with                                            dynamically change over time. In order to simplify our
     a physical address in P is assigned a virtual identifier                                       notations, in the following we will focus on the struc-
     from I. This is an application of the well-known prin-                                        tural aspects and will not explicitly represent time-
     ciple of indirection for achieving numerical scalability.                                     dependency in our notations.
   • Location-independence: The virtual identifier space                                          Additionally, FP may satisfy certain distributional prop-
     allows peers to communicate which each other irre-                                      erties, for example, that the range of values of FP follows a
     spective of their actual physical location. This ad-                                    certain distribution in space I, e.g., uniform. Such proper-
     dresses physical address changes and enables mobility.                                  ties may then be exploited, for example, for load balancing.
   • Clustering of resources with peers: The closeness met-                                  The properties FP satisfies will be denoted as CFP in the
     ric d enables the clustering of resources with peers                                    following.
     based on proximity. This is discussed in detail in Sec-                                     The mapping FR : R → I associates resources with
     tion 2.3.                                                                               identifiers from I. The choice of this mapping can be crit-
   • Message routing: Virtual identifiers and the closeness                                   ical for the application using the resources. Typically “se-
     metric d are essential for realizing efficient routing.                                  mantic closeness” of resources, e.g., resources frequently
   • Preservation of application semantics: As virtual                                       requested jointly, can be translated into closeness of identi-
     identifiers can be defined in an application-specific                                      fiers. Thus the possibility of using application-specific iden-
     way, application semantics, for example, “proximity”                                    tifiers is taking advantage of this. If the resources should be
     of resources (clustering), can be preserved.                                            identified uniquely, FR has to be injective. The distribution
    Examples: CAN [16] uses a Euclidean space with vir-                                      of identifiers generated by FR has an important impact on
tual identifiers being coordinates in this space. The dis-                                    the load-balancing properties of the overlay network em-
tance function d is the Euclidean distance. P-Grid [1] uses                                  bedded into the space I.
a prefix-preserving hash function on strings, i.e.,                                               Examples: A standard example for FP and FR is a uni-
                                                                                             form hashing function as, e.g., used by Chord [18]. This
            ∀s1 , s2 : s1 < s2 ⇒ h(s1 ) < h(s2 )                                             will generate a uniform distribution of peers on the identifier
                                                                                             space and implicitly provides load-balancing as also the re-
(< denotes lexicographic order). Identifiers in P-Grid are                                    source identifiers are uniformly distributed. However, clus-
bit strings and d is defined as (for a k-bit identifier a and an                               tering of information will not be possible and thus higher-
l-bit identifier b):                                                                          level search predicates such as range queries will be expen-
                                                                                             sive to process. P-Grid’s mapping functions on the other
                  k                 l                     k                 l                hand supports clustering but thus requires an explicit load-
d(a, b) = min(|         ai 2−i −         bi 2−i | , 1−|         ai 2−i −         bi 2−i |)   balancing strategy.
                  i=1              i=1                    i=1              i=1
                                                                                             2.3   Management of the Identifier Space
   while in Chord [18] and DKS [4] the identifier space is a
subset of the natural numbers of size N and                                                     At any point in time, I is managed by the set of current
                                                                                             peers P. The responsibility for peers for specific identifiers
                   d(x, y) = (y − x) mod N.                                                  is captured by a function M : I → 2P , which associates
                                                                                             with each identifier of a resource r, i = FR (r) ∈ I, the
2.2   Mapping to the Identifier Space                                                         set of peers that are managing r. Through M, each peer
                                                                                             p is assigned responsibility for the set M−1 (p) of identi-
   The mapping FP : P → I associates peers with a                                            fiers. Locating a resource r corresponds to finding a peer in
unique virtual identifier from I. Different approaches can                                    M(FR (r)). The lookup operation of overlay networks typ-
be distinguished by the properties of the chosen functions                                   ically provides an implementation of M through routing.
FP :                                                                                         We may identify various basic properties for M:
  • Completeness: M may be complete or partial. When                  • Uniqueness: For deterministic systems, e.g., Chord,
    M is incomplete, identifiers might (temporarily) not                  DKS, for a given set P and mapping FP only one valid
    be associated with a peer. Typically the mapping will                network N exists. In randomized systems such as P-
    be complete, such that each point of the identifier space             Grid and randomized Chord, multiple valid N are pos-
    is under the responsibility of some peers, i.e., ∀i ∈ I :            sible.
    ∃p ∈ P : p ∈ M(i)                                                 • Graph diameter: A small diameter provides lower
  • Cardinality: To provide fault-tolerance, M typically                 bounds on the latency of routing in the network.
    contains more than one element, i.e., a set of peers is           • Connectivity: Some overlay network approaches may
    responsible for managing each identifier.                             require that the overlay graph is connected at any time.
  • M induced by proximity: A standard way to specify                 • Distributional properties: These are typically distribu-
    M is that identifiers are associated with their closest               tional properties of node degrees. A frequently occur-
    peers, i.e.,                                                         ring class of graphs are power-law graphs [14]. Other
                                                                         distributional properties relate to the clustering coeffi-
       p ∈ M(i) ⇒ d(FP (p), i) = minq∈P d(FP (q), i).                    cient of the graph.
  • Dynamicity: M typically changes dynamically as the             Typical local properties of the graph include:
     set of peers and their mapping to the identifier space            • Minimal out-degree: This property is beneficial to en-
     changes.                                                            sure fault-tolerance, when many neighbors fail.
  • Uniformity of replication: The cardinality of M                   • Maximal out-degree: This property is relevant for en-
     (which corresponds to the degree of replication) may                suring bounded maintenance cost for connections to
     be constant or uniformly distributed to ensure com-                 other peers.
     parable availability of resources. Non-uniform dis-              • Distributional properties of in-degree: These are rele-
     tributions can be used to adapt the availability of re-             vant for load balancing in the message forwarding.
     sources to application requirements, e.g., popularity of         More complex properties refer to relationships of the
     resources.                                                    graph structure to the distance function. These relationships
   In the following, CM denotes the properties M satisfies.         are tightly intertwined with the strategy for efficient routing
Examples: In Chord a peer with virtual identifier a is re-          in an overlay network. Typical examples of such constraints
sponsible for the interval (predecessor(a), a], i.e.,              are:
          min{FP (a)       FP (b)|b ∈ P} ⊕ FP (b).                    • Local connectivity: This property ensures that peers
                                                                         are connected to some specific subset of their immedi-
In P-Grid a peer with a k-bit path a is responsible for all              ate neighbors. An example of such a requirement for a
identifiers in the interval                                               given peer p would be
                k                     k
                     ai 2−i , 2−k +         ai 2−i   .                    ∀q ∈ P : d(FP (p), FP (q)) < dmin ⇒ q ∈ N (p).
               i=1                    i=1
                                                                     • Long-range connectivity: Many overlay network de-
2.4   Graph Embedding                                                  signs are structurally similar to small-world graphs as
                                                                       introduced by Kleinberg [12]. These graphs are con-
    An overlay network can be modeled as a directed graph,
                                                                       structed such that long range connections satisfy the
G = (P, E), where P denotes the set of vertices (i.e., peers)
and E denotes the set of edges. Due to the dynamics in
overlay networks, G is time-dependent, but as before we                                                     1
                                                                                 P [q ∈ N (p)] ∝                       ,
will not explicitly denote this. By virtue of this graph we                                        d(FP (p), FP (q))−d
define a neighborhood relationship N : P → 2P , such that
for a given peer p, N (p) is the set of peers with which peer            where d is the dimensionality of the identifier space.
p maintains a connection, i.e., there is a directed edge (p, q)          Many overlay networks satisfy more strict variations
in E for q ∈ N (p).                                                      of this condition.
    The properties of the overlay network relate to proper-           The properties N satisfies are denoted by CN in the fol-
ties of the directed graph generated by N and to the proper-       lowing. At this point we are able to completely characterize
ties of the embedding of the graph into the (pseudo-) metric       the structural aspects of overlay networks by the following
space (I, d). Purely structural properties of the graph can        definition:
be further distinguished into local and global properties, i.e.,      Definition. The structure of an overlay network O ∈ O
whether they relate to local characteristics of graph nodes or     for a set of peers P is given by
to global characteristics of the graph. Typical global prop-
erties of the graph are the following:                                        O = (I, d, FP , CFP , M, CM , N , CN ).
2.5   Routing Strategy                                              The practical usability of an overlay network critically
                                                                 depends on the efficiency of the maintenance strategy. The
    The basic service an overlay network provides is to route    goal is to maintain a “sufficient” level of consistency while
a request for an identifier i to a peer pr responsible for it,    minimizing effort. Since a dynamically evolving overlay
i.e., pr ∈ M(i). Routing is a distributed process using the      network on top of a dynamically changing physical network
overlay network. We model it by asynchronous message             is a complex dynamical system, the goal is to arrive at a
passing: route(p, i, m) forwards a message m to a peer p         stable dynamic equilibrium for a variety of conditions while
responsible for i. A routing strategy can be described by a      guaranteeing successful routing.
potentially non-deterministic function R : P × I → 2P ,
which selects at a given peer p with neighborhood N (p) for      2.7    Other Properties
a target identifier i the (set of) next peers R(p, i) ∈ N (p),
                                                                     There are a number of further properties which we can
to which the message is forwarded. In structured overlay
                                                                 only briefly mention here due to space limitations.
networks routing typically is greedy, i.e.,
                                                                     Constraints, such as those introduced in the previous sec-
                 d(i, FP (q)) < d(i, FP (p))                     tions, can be guaranteed at different levels of strictness: If
                                                                 the constraints are valid all the time, they are invariants of
for q ∈ R(p, i). Some systems satisfy weaker conditions,         the system; if the constraints hold eventually, they may be
e.g., in Pastry,                                                 satisfied after self-stabilization of the system induced by
                                                                 changes to the systems state; if the constraints hold prob-
             d(i, FP (R(p, i))) ≤ d(i, FP (p)).
                                                                 abilistically, they are satisfied with a specific probability ei-
In unstructured overlay networks the set R(p, i) may con-        ther all the time or eventually.
tain several peers.                                                  By taking into account the physical characteristics of
   Properties of routing algorithms are characterized by         peers, such as their network location, their storage capac-
their associated cost measures, such as latency, number of       ity, etc., additional properties can be specified which are in
hops, and probability of successful routing. Given a rout-       particular useful to obtain insights and control over the per-
ing algorithm together with an overlay network structure         formance characteristics of the overlay network, for exam-
the properties regarding the expected usage of the peers’        ple, efficiency of routing and reliability of the network. An
resources can be analyzed.                                       important example of such a property is locality of routing.
                                                                 A possible formulation of such a property is a constraint on
2.6   Maintenance Strategy                                       the stretch introduced by the overlay network, i.e., that the
                                                                 physical distance of the path traversed to reach a node does
   Participation of peers in an overlay network dynamically      not exceed the distance of the shortest physical path by a
changes over time. Each peer can freely decide to join or        given stretch factor.
leave an overlay network at any time. These changes, re-
ferred to as churn in the literature, can happen quite fre-      3     Reference architecture
quently. To maintain the structural integrity of an overlay
network a maintenance strategy is required, which compen-           From an application-oriented perspective, any middle-
sates for changes to the network structure due to peers going    ware technology—and we see P2P systems and specifically
offline or failure of network connections.                        overlay networks as a form of middleware—should provide
   In all overlay networks, joining the network is done ex-      powerful and easy to use abstractions that hide implementa-
plicitly by a join operation, whereas leaving typically is im-   tion details as much as possible from the user/implementer
plicit as peers may simply go offline or crash or their net-      while offering enough control and access options to actu-
work connection may drop. Regardless whether peers leave         ally meet application requirements. Additionally, the ab-
gracefully or not, changes in the participation in an overlay    stractions should be defined in a way that the concrete in-
network typically require the application of a maintenance       frastructure implementing the middleware functionality can
strategy. Aside from access control aspects, i.e., who is al-    be replaced without requiring to rewrite code.
lowed to participate, this basically requires to repair rout-       Given these goals, we see P2P systems based on over-
ing tables which have been invalidated due to churn, i.e.,       lay networks as layered systems as depicted in in Figure 2
to maintain the connectivity of the underlying graph [8].        (for a single node). From a user’s perspective a P2P sys-
Maintenance strategies can be classified [2] into proactive       tem facilitates to realize a specific application by sharing
correction (PC) using periodic probing or heartbeats to re-      resources with other users and using services provided by
pair inconsistencies, and reactive mechanisms, with the sub-     the P2P layer. One particularly important example of such
categories correction on use (CoU), e.g., P-Grid and DKS,        a service is P2P data storage, which allows to insert, search,
correction on failure (CoF), e.g., P-Grid, and correction on     and access data items. This service as well as the applica-
change (CoC), e.g., Chord.                                       tions take advantage of the basic resource location service
provided by the P2P basic layer that implements the overlay                                                   symbolic name). As this information changes over time,
network.                                                                                                      each peer has to apply a maintenance strategy as discussed
                                                           Application                                        in Section 2.6 to have a consistent view (depending on the
                                                       P2P storage                                            specific overlay network). The number of neighbors a peer
                                                            P2P basic                                         maintains and the strategy how neighbors are selected is de-
                                                        Network (TCP/IP)                                      fined by the Constraints of the overlay network which de-
                                                                                                              pend on properties of the identifier-resource and identifier-
                 Figure 2. Layered architecture view                                                          peer association strategies, the graph embedded in the iden-
   This simple layered architecture supports separation of                                                    tifier space, and its constraints, etc.
concern between the application layer, the generic services                                                      As shown in in Figure 2, we distinguish two layers
of a P2P system and the basic overlay network of a P2P sys-                                                   of functionality. The basic layer (P2P Basic Interface)
tem. It facilitates to replace a specific implementation of a                                                  provides the low-level operations which the overlay needs
P2P system, or selected services and layers, that an appli-                                                   to be able to function. Its main functionalities, besides
cation is using by alternative implementations. In order to                                                   the mandatory join and optional leave operations, are the
support this form of modularity it is important to provide a                                                  lookup and route operations. The lookup function allows
standardized specification of the interfaces among the lay-                                                    an application to find a peer by its identifier to be able
ers. In Figure 3 we provide a class diagram that provides                                                     to directly communicate with it (point-to-point), for exam-
the core of such an interface specification. It is based on the                                                ple, for transferring data items. The route operation, which
conceptual model we have introduced in Section 2.                                                             lookup typically builds on, allows the user to send a mes-
                      $                    #
                                                                                                              sage to any peer responsible for a given identifier. A mes-
                                                       % &   '           !            (                       sage can contain any data specified by the application, for
         $           *                             +
                                                                                                              example, the data to be stored by the peer or a synchroniza-
                           &'! (, -
                          &'! '(/ -
                              &'! (/ -         '/
                                                                                                              tion request among replicas. routeToReplicas propagates a
                          &'! 3(4 &'! (% & ! 3(
                          &'! (/ & ! '(                                                       )
                                                                                                              given message to the set of peers responsible for the same
         #                         #                                                                          identifier. getLocalPeer returns the administrative informa-
                                                                                                  $           tion about the local peer and getNeighbors provides the list
                                                                                                              of neighbors of a peer, i.e., its routing table information.
                                                                                                          !      The storage layer (P2P Storage Interface) builds on
                                                                                                              these functionalities and provides the typical data manage-

             6             7                                                                                  ment functionalities of inserting, updating, deleting, and
                                                        6        7
                                                                                                              querying data, that made the P2P paradigm popular. The
                                                                                          6           7
                                                                                                              resources affected by the functions are specified via the
                      6                7                                                                      DataItem abstraction that includes the resource’s data and
                                                                                                              the application-specific key(s) to be used by the storage
     #       (
                     ) *       )                       &
                                                                             !                                layer to generate a corresponding identifier, i.e., map the
                                                                             "   #$               &
                                                                                                              data item to its position in the identifier space. This can then
             Figure 3. Conceptual decomposition                                                               be used by the basic layer to find the responsible peer(s) and
    Overlay networks are based on the embedding of a graph                                                    perform the requested operation. The DataItem set returned
into an Identifier Space which provides a closeness metric.                                                    by search includes both the application-specific keys and
Each Peer is mapped into this space, i.e., it is assigned an                                                  the identifiers of the found resources.
Identifier from the virtual identifier space, which defines                                                         We would like to emphasize that Figure 3 provides a min-
its current position in this space and (indirectly) the sub-                                                  imal model, i.e., it provides what we identified as the mini-
set of identifiers the peer is responsible for as described                                                    mal common denominator for different overlay network ap-
in Sections 2.2 and 2.3. Note that a peer’s position can                                                      proaches. All parts of the architecture can be (and in fact
change over time. How the partitioning of the identifier                                                       are) extended by concrete systems. For example, each sys-
space is done, i.e., how a node is assigned a coordinate and                                                  tem will typically have more structured message types. For
responsibility, is subject to the specific overlay approach.                                                   example, in Gnutella, as one of the simplest systems, a join
A Peer is uniquely identified by an immutable name (im-                                                        operation would mean the issuing of a Ping message which
mutableName) and maintains a neighborhood (neighbors),                                                        has a simple structure holding a descriptor ID (to prevent
i.e., references to other peers (PeerReference) for forward-                                                  loops in the routing), a payload descriptor, a time-to-live
ing. Each PeerReference includes the referenced peer’s im-                                                    counter, a hop counter and a field defining the length of the
mutable name, its position in the identifier space, i.e., re-                                                  payload. Yet, extensions of our model are intuitive and sim-
sponsibility, and physical network address (IP address or                                                     ple: A concrete system can basically “subclass” and “ex-
tend” any of the components in Figure 3.                         tical overlays systems, such as Gnutella. Multiple versions
                                                                 of overlay protocols can work together, and different peers
4   Interoperability                                             may use different policies, e.g., with respect to network con-
    Up to now we have introduced a conceptual model and             For characterizing the possibilities of interoperability
abstract interfaces to capture the specific properties of a       among peers participating in different overlay networks O1
given overlay network approach. In practice, multiple over-      and O2 , we can systematically compare the specifications
lay networks will co-exist simultaneously in a physical net-     of the networks. We assume that at the level of protocols,
work, which raises issues of managing multiple overlay net-      O1 and O2 are compatible by following the API defined in
works and interoperability.                                      Section 3 and using compatible protocol messages. This is
    We consider an overlay network as a group of peers P         a purely syntactic agreement. The classification of interop-
that share the same specification of their specific overlay        erability follows the concepts described in Section 2 and we
network mechanisms. The sharing of this specification is a        can distinguish the following levels of structural interoper-
problem of group management and can be done either ex-           ability:
plicitly or implicitly.
                                                                    • Compatible Identifiers: The identifier spaces I1 and I2
    With an explicit management explicit group identifiers G
                                                                       are the same or can be related to each other by applying
(e.g., URIs) are used to identify an overlay network and are
                                                                       a transformation. Then for identifiers in i ∈ I1 ∩ I2 ,
bound to a specific type of overlay network by a mapping
                                                                       peers from both O1 and O2 can route messages to the
T : G → O, which associates the identifier with a specifi-
                                                                       resources identified by i. Routing would be processed
cation of an overlay network. We consider this as providing
                                                                       independently in O1 and O2 . Thus peers can play the
the overlay network with a type (or schema in database par-
                                                                       role of gateways among different overlay networks.
lance). Thus every peer joining a group g ∈ G obtains the
                                                                    • Compatible Identifier Spaces: If additionally the dis-
associated type information and adheres to the specification.
                                                                       tance functions (possibly after applying a transforma-
The issue of non-complying peers is related to security and
                                                                       tion) are compatible, peers from O1 may use peers
trust which we cannot elaborate further here. As a conse-
                                                                       from O2 (and vice versa) and their knowledge on
quence, joining an overlay network would only be possible
                                                                       neighbors to integrate them into their own routing ta-
if the joining peer uses the same group identifier as the peers
of the network.
                                                                    • Compatible Structures: If additionally the structural
    With implicit management a group of peers is considered
                                                                       constraints of two overlay networks are in a subsump-
as participating in the same overlay network if they use the
                                                                       tion relationship, i.e., one of the overlay networks is
same overlay network specification. Thus there is no global
                                                                       more constrained but compatible with the more general
knowledge on the existence of a specific overlay network,
                                                                       overlay network, peers of the more constrained net-
but the network results from the cooperation of peers us-
                                                                       work may participate as peers in the less constrained
ing the same specification. Thus when joining, a peer ob-
                                                                       network by adopting the routing and maintenance al-
tains/shares the specification with the peer to which it joins.
                                                                       gorithms of the less constrained network.
    Another interesting aspect of group management in an
overlay network is the degree of coupling. In tightly cou-          An important open issue, when exploiting these forms
pled overlay network the overlay graph is at any time con-       of structural interoperability, are the effects on the perfor-
nected. This implies that such an overlay network has to         mance of the routing and maintenance mechanisms and the
be initiated by a single peer (that could, for example, de-      impact on certain structural properties of the overlay net-
termine the identifier and specification of the network prop-      works, such as distributional properties. These questions
erties, when explicit group management is used). Chord is        are closely related to the study of overlay networks built by
an example of a tightly coupled overlay network. In loosely      peers with highly heterogeneous resources, a topic which
coupled overlay networks different overlay graphs based on       has been studied only to a very limited degree so far.
the same specification (e.g., using implicit group manage-
ment) can evolve, merge, or split. Gnutella and P-Grid are
examples of loosely coupled overlay networks.
                                                                 5   Validation of the reference architecture
    The approach to implicitly manage groups of peers par-
ticipating in the same overlay network suggests a more gen-         In this section we will briefly describe key aspects
eral view of how groups of peers constructing overlay net-       of a representative set of overlay networks—Chord [18],
works may work together. In order to interact, it is in fact     DKS [4], LAND [3], P-Grid [1], Pastry [17], Sym-
not necessary that the type of overlay network is exactly        phony [13], Freenet [5], and Gnutella [6]—in terms of our
the same, but it may be sufficient that the specifications are     architecture to demonstrate its validity. Additionally, we
compatible. This approach can be observed for some prac-         provide a brief qualitative comparison of the systems.
5.1    Identifier Space                                                             that nodes in the same organizational domain are logically
                                                                                   close in the identifier space.
    The identifier spaces are very similar for all logarithmic-                         Most of the logarithmic-style overlay approaches like
style overlay networks (LAND, P-Grid, Chord, Pastry,                               Chord, LAND, Pastry, Symphony, or P-Grid have a one-
Symphony, etc.). In these approaches identifiers are chosen                         dimensional identifier space. In Chord, LAND, and Pas-
from an alphabet with radix b, e.g., b = 2 for P-Grid, Chord,                      try identifiers are assigned by hashing the node’s IP ad-
and Symphony, b = 16 for Pastry. Some of them limit the                            dress using SHA-1. In contrast, in P-Grid each peer ini-
identifier length, e.g., Pastry uses 128-bit, Chord and DKS                         tially is responsible for the whole identifier space and has
use 160-bit length identifiers, whereas in P-Grid identifiers                        an empty identifier which grows bit by bit in the lifetime of
can be of arbitrary length. A similar distance function is                         the peer depending on which other peers it encounters and
shared by all of these overlays, though there are some sub-                        what type of data they and the peer itself store. Similarly
tle differences. In P-Grid, LAND, Pastry, and Symphony                             in Freenet, each node assigns itself an identifier vector of
the distance d(u, v) of two identifiers u (of length k) and v                       size r, consisting of r 160-bit elements representing the r
(of length l) is                                                                   identifiers of data items the peer stores. Additionally, the
                                                                                   identifier of a Freenet peer changes during its lifetime de-
        k                l                     k                 l
                                                                                   pending on the queries it handles. Thus the identifiers in
min          ui b−i −         vi b−i , 1 −           ui b−i −         vi b−i   .
                                                                                   P-Grid and Freenet dynamically change, whereas in Chord,
       i=1              i=1                    i=1              i=1
                                                                                   LAND, Pastry, and Symphony they are static.
   Note that in Pastry’s case k = l, and for these systems                             Mapping of resources: Mapping of resources (data
d(u, v) is symmetric as d(u, v) = d(v, u). For example, in                         items) is done similarly to mapping peers. Usually it
P-Grid,                                                                            is done by hashing a data key, e.g., the filename, with
                                                                                   SHA-1 (Chord, LAND, Pastry, Freenet). While this im-
        d(“0000”, “10”) = d(“10”, “0000”) = 0.5.                                   plicitly distributes the assigned identifiers uniformly in the
                                                                                   identifier space and thus provides a simple load-balancing
   The identifier space in Chord is not symmetric, i.e.,                            mechanism, it destroys the semantics of keys, e.g., their
d(u, v) = d(v, u). d(u, v) can be defined as                                        application-specific clustering, which can be exploited to
                                                                                   provide efficient data access. To prevent this, P-Grid, for
               k                 k
                                                                                   example, uses a prefix-preserving hash function, i.e., u <
                    vi 2−i −          ui 2−i       +1     mod 1.                   v ⇒ h(u) < h(v). This has advantages in query processing
              i=1               i=1
                                                                                   but requires an additional and more complex load-balancing
Thus                                                                               strategy. It is crucial that this mapping of resources is deter-
                    d(“001”, “111”) = 0.75,                                        ministic, static, and globally known.

but                                                                                5.3   Management of Identifier Space
                    d(“111”, “001”) = 0.25.
                                                                                      In P-Grid each peer is responsible for resource identi-
   In Freenet the situation is slightly different. Due to the
                                                                                   fiers that share the largest common prefix with the peer’s
way Freenet identifies nodes, it uses an r-dimensional 160-
                                                                                   identifier, i.e., a resource identifier is managed by the peer
bit identifier space. r depends on the data items a peer
                                                                                   with the closest identifier in terms of P-Grid’s distance func-
stores, but usually r = 50. d(u, v) between two Freenet
                                                                                   tion. For example, peer “0011” is responsible for resource
peers is the Euclidean distance in this multidimensional
                                                                                   identifier “001110101”, if no peer with a longer common
                                                                                   prefix exists. The situation in Freenet is very similar: Each
5.2    Mapping to the Identifier Space                                              peer is responsible for the resource identifiers which are nu-
                                                                                   merically closest to one of the peer’s elements in its vector
   Mapping of peers: The key difference among the over-                            identifier. Also in LAND, Pastry, and Symphony a simi-
lays with respect to this mapping is whether the virtual iden-                     lar condition applies. Data items are managed by the peer
tifier is assigned to a peer randomly or the peer adopts the                        with the closest identifier. For example, identifier “2A83”
identifier depending on environment conditions, e.g., de-                           will be managed by peer “2A84” if no peers “2A82” and
pending on the data a peer and its neighboring peers store.                        “2A83” exist. As Chord’s and DKS’s identifier spaces are
In Chord, LAND, Pastry, and Symphony the virtual identi-                           asymmetric, the situation is slightly different. A peer is re-
fier is generated using some random function and assigned                           sponsible for all identifiers in the interval between its own
to a peer upon joining the overlay and remains stable. In                          identifier and the identifier of its predecessor on the ring.
DKS identifiers can be mapped order-preservingly based on                           In all these approaches the responsibility of a peer may dy-
their domain name, e.g., lexicographic ordering, to ensure                         namically change due to arrivals or departures of peers in
the overlay.                                                     networks have been characterized by using algebraic meth-
                                                                 ods (Cayley graphs) [15]. This work is complementary as
5.4    Graph Embedding                                           it could be used to formulate more specific constraints on
                                                                 the structure of overlay networks within our architectural
   It has already been shown that peers cooperating in           framework. In [11] classifications for structured overlay
Freenet evolve the graph into a small-world graph. For           networks, e.g., deterministic and randomized networks, are
logarithmic-style overlay approaches, [9] shows that these       introduced. These classifications correspond to different
approaches form graphs according to Kleinberg’s small-           constraints that we can capture in our conceptual model.
world principles [12]. It is proven that such graphs belong      To best of our knowledge, no other proposals for an overlay
to the special class of “routing efficient” small-world net-      network reference architecture exist.
works where decentralized, greedy search algorithms pro-
vide the best performance. Therefore, conceptually all of        7   Future Work
these approaches build similar small-world graphs with cer-
tain constraints for each case. E.g., Symphony by its na-           The reference architecture as presented so far focuses on
ture of construction forms a small-world graph. In the other     the most relevant functional and non-functional properties
logarithmic-style overlay cases each peer u views the iden-      of overlay networks. However, there are some design as-
tifier space as partitioned in log (N ) partitions where each     pects of overlay networks which we could not discuss in
partition is b times bigger than the previous one (b is the      detail due to space constraints.
radix of the identifier alphabet). The routing table of u in         As already mentioned in Section 2, overlay networks
such systems contains logb (N ) links to some nodes from         highly depend on the proper cooperation of their partici-
each partition. In Chord’s case the chosen node will be          pants, i.e., participants have to be trusted to perform non-
the one with the smallest identifier of the given partition,      maliciously, route correctly, return unaltered data, etc. This
while Pastry and P-Grid use any random node in the parti-        is a key assumption underlying all overlay networks. In
tion, which is a more relaxed constraint.                        each approach a minimum percentage of the peer popula-
                                                                 tion has to be trustworthy for the system to function prop-
5.5    Gnutella                                                  erly and there are significant differences in the way how
                                                                 overlays handle trust, i.e., they are vulnerable to different
   Gnutella’s underlying paradigm has major conceptual           extents. A number of approaches to secure routing and as-
differences compared to the structured overlay systems de-       sessing the reputation of peers exist already but have only
scribed above. Despite its simplicity, the description of        been integrated into current overlay networks to a limited
Gnutella is not trivial. On the surface, it may seem that        extent. As part of our future work we will detail our archi-
Gnutella has no identifier space and no mappings are done.        tecture in this respect to come up with a systematic model
But then it would be impossible to describe Gnutella as a        to describe and classify trust and reputation approaches in
graph embedded in an identifier space. Nevertheless, we           current overlay networks.
can assume that peers in Gnutella exist in an Euclidean             Additionally, there are some areas which require further
identifier space R and each peer assigns itself a random          research to extend the applicability of our reference archi-
identifier i ∈ R. Then we assume that each peer is responsi-      tecture. At the moment we have validated our reference ar-
ble for the whole identifier space and therefore it is not nec-   chitecture against “classical” overlay approaches and some
essary to map resources to the identifier space. Addition-        recent systems. It will be part of our future work to de-
ally, each peer chooses four random neighbors. Given such        scribe a larger number of systems in terms of our architec-
conditions and Gnutella’s routing and maintenance strate-        ture and include also more of the most recent systems to
gies, the peers form a small-world graph in the identifier        further validate it. This will also include qualitative com-
space. As this graph has a very low diameter of approxi-         parisons of existing approaches to describe in which as-
mately log (N ), constrained flooding for search works ef-        pects the systems differ and which settings they cover best.
ficiently in terms of latency. This shows that also Gnutella      We are confident that the architecture will only require mi-
fits our conceptual model of overlay networks.                    nor adjustments as most recent system are either variants
                                                                 of “classical” approaches or focus on one specific property,
6     Related Work                                               for example, BitTorrent which focuses mainly on efficient
                                                                 distribution while putting other P2P aspects more into the
    Although a large number of overlay networks have been        background.
devised, only very few works on unifying architectures ex-          So far our model is primarily targeted at understanding
ist. The closest ones, JXTA [10] and Dabek et al. [7]            and analyzing overlay networks. An interesting related as-
have already been discussed in the introduction. In a re-        pect, however, is to investigate whether and how our model
cent work the structural properties of a subclass of overlay     can be applied to detect design weaknesses and therefore
might be used to improve specific systems or may even en-          [3] I. Abraham, D. Malkhi, and O. Dobzinski. LAND: Stretch
able us to provide general suggestions for improvements.              (1+ε) Locality Aware Networks for DHTs. In ACM-SIAM
   Another line of future efforts will deal with providing a          Symposium on Discrete Algorithms (SODA 2004), 2004.
                                                                  [4] L. O. Alima, S. El-Ansary, P. Brand, and S. Haridi.
reference implementation of our architecture. Besides be-
                                                                      DKS(N,k,f): A Family of Low Communication, Scalable and
ing applicable as an introduction to the P2P paradigm and
                                                                      Fault-Tolerant Infrastructures for P2P Applications. In 3rd
being usable for classification purposes, this implementa-             IEEE/ACM International Symposium on Cluster Computing
tion and an architecture simulator would then be a platform           and the Grid (CCGRID), 2003.
which researchers could use to plug in new approaches or          [5] I. Clarke, O. Sandberg, B. Wiley, and T. W. Hong. Freenet:
test variants of existing approaches or would be able to un-          A Distributed Anonymous Information Storage and Retrieval
derstand the effects of certain system and environmental              System. In Designing Privacy Enhancing Technologies: In-
parameters in experiments and validate the performance of             ternational Workshop on Design Issues in Anonymity and
their changes against the reference architecture. As a first           Unobservability, number 2009 in LNCS, 2001.
                                                                  [6] The Gnutella Protocol Specification v0.4 (Document Re-
step in this direction we have already changed the imple-             vision 1.2), June 15 2001.
mentation of some overlay approaches to match our refer-              developer/gnutella protocol 0.4.pdf.
ence architecture and have defined a reference API to enable       [7] F. Dabek, B. Zhao, P. Druschel, J. Kubiatowicz, and I. Stoica.
interoperability among different overlay networks.                    Towards a Common API for Structured Peer-to-Peer Over-
                                                                      lays. In IPTPS, 2003.
8   Conclusions                                                   [8] A. Ghodsi, L. O. Alima, and S. Haridi. Low-Bandwidth
                                                                      Topology Maintenance for Robustness in Structured Overlay
   Based on a stringent analysis of current overlay net-              Networks. In HICSS, 2005.
                                                                  [9] S. Girdzijauskas, A. Datta, and K. Aberer. On Small-World
works, we discussed and formally described the key design
                                                                      Graphs in Non-uniformly Distributed Key Spaces. In 1st
aspects in this domain. We then used our assessments to               IEEE International Workshop on Networking Meets Data-
define a reference architecture for overlay networks specif-           bases (NetDB), 2005.
ically addressing API and interoperability aspects. To the       [10] L. Gong. JXTA: A Network Programming Environment.
best of our knowledge this is the first reference architec-            IEEE Internet Computing, 5(3):88–95, May/June 2001.
ture for overlay networks which includes a formal codifica-       [11] K. Gummadi, R. Gummadi, S. Ratnasamy, S. Shenker, and
tion and definition of all design aspects. Previous attempts           I. Stoica. The Impact of DHT Routing Geometry on Re-
                                                                      silience and Proximity. In ACM SIGCOMM, 2003.
have focused on high-level component aspects or on low-          [12] J. Kleinberg. The Small-World Phenomenon: An Algorith-
level API issues or on the comparison of functionalities, but         mic Perspective. In ACM Symposium on the Theory of Com-
none of these works has properly addressed the “internals”            puting (STOC), 2000.
of overlay networks in a formal way so far. To validate          [13] G. S. Manku, M. Bawa, and P. Raghavan. Symphony: Dis-
the correctness and general applicability of our approach we          tributed Hashing in a Small World. In 4th USENIX Sympo-
applied it to model a representative set of overlay networks.         sium on Internet Technologies and Systems (USITS), 2003.
                                                                 [14] M. Mitzenmacher. A Brief History of Generative Models
Our reference architecture establishes a standardized vocab-
                                                                      for Power Law and Lognormal Distributions. Draft man-
ulary and facilitates the assessment of properties of overlays        uscript.∼michaelm/postscripts/
for qualitative comparison and can serve as the basis for   , 2005.
the definition of a standardized API. A standardized API          [15] C. Qu, W. Nejdl, and M. Kriesell. Cayley DHTs - A Group-
backed up by well-founded abstractions as presented in this           Theoretic Framework for Analyzing DHTs Based on Cayley
paper will enable fine-grained interoperability among het-             Graphs. In International Symposium on Parallel and Distrib-
erogeneous overlay networks which was one of our primary              uted Processing and Applications (ISPA), 2004.
                                                                 [16] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and
goals in the definition of the architecture. As a first step in
                                                                      S. Shenker. A Scalable Content-Addressable Network. In
this direction we have already changed the implementation             ACM SIGCOMM, 2001.
of some overlay approaches to match our reference archi-         [17] A. Rowstron and P. Druschel. Pastry: Scalable, distributed
tecture and have defined a reference API which is already              object location and routing for large-scale peer-to-peer sys-
being applied in three EU projects to enable interoperability         tems. In IFIP/ACM International Conference on Distributed
among different overlay networks.                                     Systems Platforms (Middleware), 2001.
                                                                 [18] I. Stoica, R. Morris, D. Karger, F. Kaashoek, and H. Balakr-
References                                                            ishnan. Chord: A Scalable Peer-To-Peer Lookup Service for
                                                                      Internet Applications. In ACM SIGCOMM, 2001.
                                                                 [19] G. Tel. Introduction to Distributed Algorithms. Cambridge
[1] K. Aberer. P-Grid: A self-organizing access structure for
                                                                      University Press, 1994.
    P2P information systems. In Sixth International Conference   [20] B. Y. Zhao, J. Kubiatowicz, and A. D. Joseph. Tapestry: An
    on Cooperative Information Systems (CoopIS), 2001.                infrastructure for fault-tolerant wide-area location and rout-
[2] K. Aberer, A. Datta, and M. Hauswirth. Route maintenance
                                                                      ing. Technical Report UCB/CSD-01-1141, UC Berkeley,
    overheads in DHT overlays. In 6th Workshop on Distributed
    Data and Structures (WDAS), 2004.

To top