Learning Center
Plans & pricing Sign in
Sign Out
Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>

Dynamo Amazon's Highly Available Key-value Store


									Dynamo: Amazon‟s Highly
Available Key-value Store
   Giuseppe DeCandia,,
          SOSP „07
• Dynamo: used to manage applications that
  require only primary-key access to data
• Dynamo applications need scalability, high
  availability, fault tolerance, but don‟t need
  the complexity of a relational DB
  – ACID properties => little parallelism, low
• Applications perform simple read/write ops on
  single, small ( < 1MB) data objects which are
  identified by a unique key.
  – Example: the shopping cart
• Replace ACID properties with weaker
  guarantees: eventual consistency, no isolation
• Services must operate efficiently on commodity
• Used only by internal services, so security isn‟t
  an issue
 Service Level Agreements (SLA)
• Clients and servers negotiate SLAs to
  establish the kind of service and the
  expected performance
• Amazon expects the guarantees to apply
  to 99.9% of requests
  – Claim that most industry systems express
    SLAs in terms of “average”, “median”, and
    “expected variance” – much weaker than
    Amazon‟s requirements
        Design Considerations
• Services control properties such as durability
  and consistency, evaluate tradeoffs (cost v
  performance, for example)
• Replicated databases cannot guarantee
  strong consistency and high availability at the
  same time
  – Optimistic replication updates replicas as a
    background process to get eventual consistency
    Design Considerations:
  Resolving Conflicting Updates
• When
 – Since Dynamo targets services that require
   “always writeable” data storage; e.g., users
   must always be able to add/delete from the
   shopping cart; resolve conflicts during reads,
   not writes
• By Whom
 – Let each application decide for itself
 – But … default is “last write wins”.
   Other Key Design Principles
• Incremental scalability: adding a single
  node should not affect the system
• Symmetry: all nodes have the same
• Decentralization: favor P2P techniques
  over centralized control
• Heterogeneity: take advantage of
  differences in server capabilities.
 Comparison to Other Systems
• Peer-to-Peer (Freenet, Chord, …)
  – Structured v unstructured: access times
  – Conflict resolution for concurrent updates
    without wide-area file locking
• Distributed File Systems and Databases
  (Google, Bayou, Coda, …)
  – Treatment of system partitions
  – Conflict resolution, eventual consistency
  – Strong consistency v eventual consistency
   Dynamo v Other Decentralized
        Storage Systems
• “always writeable”;
  – updates won‟t be rejected because of failure
    or concurrent updates
• One administrative domain; nodes are
  assumed to be trustworthy
• Don‟t require hierarchical name spaces or
  relational schema
• Operations must be performed within a
  few hundred milliseconds.
        System Architecture
• The Dynamo data storage system
  contains items that are associated with a
  single key
• Operations that are implemented: get( )
  and put( ).
  – get(key)
  – put(key, context, object) where context refers
    to various kinds of system metadata
Problem             Technique                   Advantage

Partitioning        Consistent Hashing          Incremental scalability

High availability   Vector clocks, reconciled   Version size is decoupled
for writes          during reads                from update rates

Temporary           Sloppy Quorum,              Provides high availability &
failures            hinted handoff              durability guarantee when
                                                some of the replicas are
                                                not available

Permanent           Anti-entropy using          Synchronizes divergent replicas
failures            Merkle trees                in the background

Membership &      Gossip-based protocol         Preserves symmetry and avoids
failure detection                               having a centralized registry for
                                                storing membership and node
                                                liveness information

Table 1: Summary of techniques used in Dynamo and their advantages
       Partitioning Algorithm
• Partitioning = dividing data storage across
  all nodes. Supports scalability
• Very similar to Chord-based schemes
• Consistent hashing scheme distributes
  content across multiple nodes
  – In consistent hashing the effect of adding a
    node is localized – on average, K/n objects
    must be remapped (K = # of keys, n = # of
       Partitioning Algorithm
• Hash function produces an m-bit number
  which defines a circular name space (like
• Nodes are assigned numbers randomly in
  the name space
• Hash(data key) and assign to node using
  successor function like Chord
           Load Distribution
• Random assignment of node to position in
  ring may produce non-uniform distribution
  of data.
• Solution: virtual nodes
  – Assign several random numbers to each
    physical node; now it is responsible for itself
    and data that would be stored on the virtual
    nodes, if they existed
• Data is replicated at N nodes
• Succ(key) = coordinator node
  – The coordinator replicates the object at the N-1
    successor nodes in the ring, skipping virtual
    nodes to increase fault tolerance
  – Preference list: the list of nodes that store a
    particular key
  – There are actually > N nodes on the preference
    list, in order to ensure N “healthy” nodes at all
            Data Versioning
• Updates can be propagated to replicas
  asynchronously – the put( ) call may return
  before all updates have been applied.
  – Implication: a subsequent get( ) may return
    stale data.
• Barring failure, most updates are applied
  within bounded time, but server or network
  failure can delay updates “for an extended
  period of time”.
            Data Versioning
• Some app‟s can be designed to work in
  this environment; e.g., the “add-to/delete-
  from cart” operation.
  – It‟s okay to add to an old cart, as long as all
    versions of the cart are eventually reconciled
• Dynamo treats each modification as a new
  (& immutable) version of the object.
  – Multiple versions can exist at the same time
• Usually, new versions contain the old
  versions – no problem
• Sometimes concurrent updates and
  failures generate conflicting versions
• Typically this is handled by merging
  – For add-to-cart operations, nothing is lost
  – For delete-from cart, deleted items might
    reappear after the reconciliation
     Parallel Version Branches
• There may be multiple versions of the same
  data, each coming from a different path (e.g., if
  there‟s been a network partition)
• Vector clocks are used to identify causally
  related versions and parallel (concurrent)
   – For causally related versions, accept the final version
     as the “true” version
   – For parallel (concurrent) versions, use some
     reconciliation technique to resolve the conflict
   Execution of get( ) and put( )
• Operations can originate at any node in the
• Clients may
  – Route request through a load-balancing coordinator
  – Use client software that routes the request directly to
    the coordinator for that object
• The coordinator contacts R nodes for reading
  and W nodes for writing, where R + W > N
                “Sloppy Quorum”
• put( ): the coordinator writes to the first N healthy
  nodes on the preference list. If W writes succeed,
  the write is considered to be successful
• get( ): coordinator reads from N nodes; waits for R
   – If they agree, return value.
   – If they disagree, but are causally related, return the
     most recent value
   – If they are causally unrelated apply reconciliation
     techniques and write back the corrected version
              Hinted Handoff
• What if a write operation can‟t reach some of the
  nodes on the preference?
• To preserve availability and durability, store the
  replica temporarily on another node,
  accompanied by a metadata “hint” that
  remembers where the replica should be stored.
• Hinted handoff ensures that read and write
  operations don‟t fail because of network
  partitioning or node failures.
  Handling Permanent Failures
• Hinted replicas may be lost before they
  can be returned to the original node. Other
  problems may cause replicas to be lost or
  fall out of agreement
• Merkle trees allow two nodes to compare
  a set of replicas and determine fairly easily
  – Whether or not they are consistent
  – Where the inconsistencies are
  Handling Permanent Failures
• Merkle trees have leaves whose values are
  hashes of the values associated with keys (one
  – Parent nodes contain hashes of their children
  – Eventually, root contains a hash that represents
    everything in that replica
• To detect inconsistency between two sets of
  replicas, compare the roots
  – Source of inconsistency can be detected by looking at
    internal nodes
• Like Google, Amazon has a number of
  data centers, each with many commodity
  – Individual machines fail regularly
  – Sometimes entire data centers fail due to
    power outages, network partitions, tornados,
• To handle failure of entire centers, replicas
  are spread across multiple data centers.
Membership and Failure Detection
• Temporary failures or accidental additions
  of nodes are possible but shouldn‟t cause
  load re-balancing.
• Additions and deletions of nodes are
  explicitly executed by an administrator.
• A gossip-based protocol is used to ensure
  that every node eventually has a
  consistent view of the membership list.
      Gossip-based Protocol
• Periodically, each node contacts another
  node in the network, randomly selected.
• Nodes compare their membership
  histories and reconcile them.
          Load Balancing for
        Additions and Deletions
• When a node is added, it acquires key values
  from other nodes in the network.
  – Nodes learn of the addition through the gossip
    protocol, contact the node to offer their keys, which
    are then transferred after being accepted
  – When a node is removed, a similar process happens
    in reverse
• Experience has shown that this approach leads
  to a relatively uniform distribution of key/value
  pairs across the system
• Experience with Dynamo indicates that it meets
  the requirements of scalability and availability.
• Service owners are able to customize their
  storage system to emphasize performance,
  durability, or consistency. The primary
  parameters are N, R, and W.
• The developers conclude that decentralization
  and eventual consistency can provide a
  satisfactory platform for hosting highly-available

To top