VIEWS: 0 PAGES: 123 POSTED ON: 9/30/2013
Byzantine Techniques II Justin W. Hart CS 614 12/01/2005 Papers n BAR Fault Tolerance for Cooperative Services. Amitanand S. Aiyer, et. al. (SOSP 2005) n Fault-scalable Byzantine Fault-Tolerant Services. Michael Abd-El-Malek et.al. SOSP 2005 BAR Fault Tolerance for Distributed Services n BAR Model n General Three-Level Architecture n BAR-B Motivation n “General approach to constructing cooperative services that span multiple administrative domains (MADs)” Why is this difficult? n Nodes are under control of multiple administrators n Broken – Byzantine behaviors. n Misconfigured, or configured with malicious intent. n Selfish – Rational behaviors n Alter the protocol to increase local utility Other models? n Byzantine Models – Account for Byzantine behavior, but do not handle rational behavior. n Rational Models – Account for rational behavior, but may break with Byzantine behavior. BAR Model n Byzantine n Behaving arbitrarily or maliciously n Altruistic n Execute the proposed program, whether it benefits them or not n Rational n Deviate from the proposed program for purposes of local benefit BART – BAR Tolerant n It’s a cruel world n At most (n-2)/3 nodes in the system are Byzantine n The rest are rational Two classes of protocols n Incentive-Compatible Byzantine Fault Tolerant (IC-BFT) n Guarantees a set of safety and liveliness properties n It is in the best interest of rational nodes to follow the protocol exactly n Byzantine Altruistic Rational Tolerant n Guarantees a set of safety and liveliness properties despite the presence of rational nodes n IC-BFT is a subset of BART An important concept n It isn’t enough for a protocol to survive drills of a handful of attacks. It must provably provide its guarantees. A flavor of things to come n Protocol builds on Practical Byzantine Fault Tolerance in order to combat Byzantine behavior n Protocol uses game theoretical concepts in order to combat rational behavior A taste of Nash Equilibrium Swerve Go Straight Swerve 0, 0 -1,+1 Go Straight +1,-1 X_X,X_X -100,-100 …and the nodes are starving! n Nodes require access to a state machine in order to complete their objectives n Protocol contains methods for punishing rational nodes, including denying them access to the state machine An expensive notion of identity n Identity is established through cryptographic keys assigned through a trusted authority n Prevents Sybil attacks n Bounds the number of Byzantine nodes n Gives rational nodes reason to consider long- term consequences of their actions n Gives real world grounding to identity Assumptions about rational nodes n “Receive long-term benefit from staying in the protocol” n “Conservative when computing the impact of Byzantine nodes on their utility” n “If the protocol provides a Nash equilibrium, then all rational nodes will follow it” n “Rational nodes do not collude…colluding nodes are classified as Byzantine” Byzantine nodes n Byzantine fault model n Strong adversary n Adversary can coordinate collusion attacks Important concepts n Promptness principal n Proof of Misbehavior (POM) n Cost balancing Promptness principal n If a rational node gains no benefit from delaying a message, it will send it as soon as possible Proof of Misbehavior (POM) n Self-contained, cryptographic proof of wrongdoing n Provides accountability to nodes for their actions Example of POM n Node A requests that Node B store a chunk n Node B replies that it has stored the chunk n Later Node A requests that chunk back n Node B sends back random garbage (it hadn’t stored the chunk) and a signature n Because Node A stored a hash of the chunk, it can demonstrate misbehavior on part of Node B …but it’s a bit more complicated than that! n This corresponds to a rather simple behavior to combat. “Aggressively Byzantine” behavior. Passive-aggressive behaviors n Harder cases than “aggressively Byzantine” n A malicious Node A could merely lie about misbehavior on the part of Node B n A node could exploit non-determinism in order to shirk work Cost Balancing n If two behaviors have the same cost, there is no reason to choose the wrong one Three-Level Architecture Level 1 n Unilaterally deny service to nodes that fail to deliver messages n “Tit-for-Tat” n Balance costs n No incentive to make the wrong choice n Penance n Unilaterally impose extra work on nodes with untimely responses Level 2 n Failure to respond to a request by a state machine will generate a POM from a quorum of nodes in the state machine Level 3 n Makes use of reliable work assignment n Needs only to provide sufficient information to identify valid request/response pairs Nuts and Bolts n Level 1 n Level 2 Level 1 n Ensure long-term benefit to participants n The RSM rotates the leadership role to participants. n Participants want to stay in the system in order to control the RSM and complete their protocols n Limit non-determinism n Self interested nodes could hide behind non- determinism to shirk work n Use Terminating Reliable Broadcast, rather than consensus. n In TRB, only the sender can propose a value n Other nodes can only adopt this value, or choose a default value Level 1 n Mitigate the effects of residual non- determinism n Cost balancing n The protocol preferred choice is no more expensive than any other n Encouraging timeliness n Nodes can inflict sanctions on untimely messages n Enforce predictable communication patterns n Nodes have to have participated at every step in order to have the opportunity to issue a command Terminating Reliable Broadcast 3f+2 nodes, rather than 3f+1 n Suppose a sender “s” is slow n The same group of nodes now want to determine that “s” is slow n A new leader is elected n Every node but “s” wants a timely conclusion to this, in order to get their turn to propose a value to the state machine n “s” is not allowed to participate in this quorum TRB provides a few guarantees n They differ during periods of synchrony and periods of asynchrony In synchrony n Termination n Every non-Byzantine process delivers exactly one message n Agreement n If on non-Byzantine process delivers a message m, then all non-Byzantine processes eventually deliver m In asynchrony n Integrity n If a non-Byzantine process delivers m, then the sender sent m n Non-Triviality n If the sender is non-Byzantine and sends m, then the sender eventually delivers m Message Queue n Enforces predictable communication patterns n Bubbles n A simple retaliation policy n Node A’s message queue is filled with messages that it intends to send to Node B n This message queue is interleaved with bubbles. n Bubbles contain predicates indicating messages expected from B n No message except the expected predicate from B can fill the bubble n No messages in A’s queue will go to B until B fills the bubble Balanced Messages n We’ve already discussed this quite a bit n We assure this at this level of the protocol n This is where we get our gigantic timeout message Penance n Untimely vector n Tracks a nodes perception of the responsiveness of other nodes n When a node becomes a sender, it includes its untimely vector with the message Penance n All nodes but the sender receive penance messages from each node. n Because of bubbles, each untimely node must sent a penance message back in order to continue using the system n This provides a penalty to those nodes n The sender is excluded from this process, because it may be motivated to lie in its penance vector, in order to avoid the work of transmitting penance messages Timeouts and Garbage Collection n Set-turn timeout n Timeout to take leadership away from the sender n Initially 10 seconds in this implementation, in order to overcome all expected network delays n Can only be changed by the sender n Max_response_time n Time at which a node is removed from the system, its messages discarded and its resources garbage collected n Set to 1 week or 1 month in the prototypes Global Punishment n Badlists n Transform local suspicion into POMs n Suspicion is recorded in a local nodes badlist n Sender includes its badlist with its message n If, over time, recipients see a node in f + 1 different senders badlists, then they too, consider that node to be faulty Proof n Real proofs do not appear in this paper, they appear in the technical report …but here’s a bit n Theorem 1: The TRB protocol satisfies Termination, Agreement, Integrity and Non-Triviality …and a bit more n Theorem 2: No node has a unilateral incentive to deviate from the protocol n Lemma 1: No rational node r benefits from delaying sending the “set-turn” message n Follows from penance n Lemma 2: No rational node r benefits from sending the “set-turn” message early n Sending early could result in senderTO to be sent (this protocol uses synchronized clocks, and all messages are cryptographically signed) …and the rest that’s mentioned in the paper n Lemma 3: No rational node r benefits from sending a malformed “set-turn” message. n The “set-turn” message only contains the turn number. Because of this, doing so reduces to either sending early (dealt with in Lemma 1) or sending late (dealt with in Lemma 2) Level 2 n State machine replication is sufficient to support a backup service, but the overhead is unacceptable n 100 participants… 100 MB backed up… 10 GB of drive space n Assign work to individual nodes, using arithmetic codes to provide low- overhead fault-tolerant storage Guaranteed Response n Direct communication is insufficient when nodes can behave rationally n We introduce a “witness” that overhears the conversation n This eliminates ambiguity n Messages are routed through this intermediary Guaranteed Response Guaranteed Response n Node A sends a request to Node B through the witness n The witness stores the request, and enters RequestReceived state n Node B sends a response to Node A through the witness n The witness stores the response, and enters ResponseReceived Guaranteed Response n Deviation from this protocol will cause the witness to either notice the timeout from Node B or lying on the part of Node A Implementation n The system must remain incentive-compatible n Communication with the witness node is not in the form of actual message sending, it is in the form of a command to the RSM n Theorem 3: If the witness node enters the request received state, for some work w to rational node b, then b will execute w n Holds if sufficient sanctions exist to cause it to be motivated to do this State limiting n State is limited by limiting the number of slots (nodes with which a node can communicate) available to a node n Applies a limit to the memory overhead n Limits the rate at which requests are inserted into the system n Forces nodes to acknowledge responses to requests n Nodes want their slots back Optimization through Credible Threats Optimization through Credible Threats n Returns to game theory n Protocol is optimized so nodes can communicate directly. Add a fast path n Nodes register “vows” with the witness n If recipient does not respond, nodes proceed to the unoptimized case n Analogous to a driver in “chicken” throwing their steering wheel out the window Periodic Work Protocol n Witness checks that periodic tasks, such as system maintenance are performed n It is expected that, with a certain frequency, each node in the system will perform such a task n Failure to perform one will generate a POM from the witness Authoritative Time Service n Maintains authoritative time n Binds messages sent to that time n Guaranteed response protocol relies on this for generating NoResponses Authoritative Time Service n Each submission to the state machine contains the timestamp of the proposer n Timestamp is taken to be the maximum of the median of timestamps of the previous f+1 decisions n If “no decision” is decided, then the timestamp is the previous authoritative time Level 3 BAR-B n BAR-B is a cooperative backup system n Three operations n Store n Retrieve n Audit Storage n Nodes break files up into chunks n Chunks are encrypted n Chunks are stored on remote nodes n Remote nodes send signed receipts and store StoreInfos Retrieval n A node storing a chunk can respond to a request for a chunk with n The chunk n A demonstration that the chunk’s lease has expired n A more recent StoreInfo Auditing n Receipts constitute audit records n Nodes will exchange receipts in order to verify compliance with storage quotas Arithmetic Coding n Arithmetic coding is used to keep storage size reasonable n 1 GB of storage requires 1.3 GB of overhead n Keeping this ratio reasonable is crucial to motivate self-interested nodes to participate Request-Response pattern n Store n Retrieve n Audit Retrieve n Originator sends a Receipt for the StoreInfo to be retrieved n Storage node can send n A RetrieveConfirm n Containing the data and the receipt n A RetrieveDeny n Containing a receipt and a proof regarding why n Anything else n Generates a POM Store n Originator sends a StoreInfo to be stored n Storage node can send n A receipt n A StoreReject n Demonstrates that the node has reached its storage commitment n Anything else n Generates a POM Audit n Three phases n Auditor requests both OwnList and StoreList from auditee n Does this for random nodes in the system n Lists are checked for inconsistencies n Inconsistencies result in a POM Time constraints n Data is stored for 30 days n After this, it is garbage collected n Nodes must renew their leases on stored chunks, in order to keep them in the system, prior to this expiration Sanctions n Periodic work protocol forces generation of POMs or special NoPOMs n POMs and NoPOMs are balanced n POMs evict nodes from the system Recovery n Nodes must be able to recover after failures n Chained membership certificates are used in order to allow them to retrieve their old chunks n Use of certificate later in the chain is regarded as a new node entering the system n The old node is regarded as dead n The new node is allowed to view the old nodes chunks Recovery n This forces nodes to redistribute their chunks that were on that node n Length of chains is limited, in order to prevent nodes from shirking work by using a certificate later in the chain Guarantees n Data on BAR-B can be retrieved within the lease period n No POM can be gathered against a node that does not deviate from the protocol n No node can store more than its quota n A time window is available to nodes with catastrophic failures for recovery Evaluation n Performance is inferior to protocols that do note make these guarantees, but acceptable Impact of additional nodes Impact of rotating leadership Impact of fast path optimization Fault-Scalable Byzantine Fault- Tolerant Services n Query/Update (Q/U) protocol n Optimistic quorum based protocol n Better throughput and fault-scalability than Replicated State Machines n Introduces preferred quorum as an optimization on quorum protocols Motivation n Compelling need for services and distributed data structures to be efficient and fault-tolerant n In Byzantine fault-tolerant systems, performance drops off sharply as more faults are tolerated Fault Scalability n A fault-scalable service is one in which performance degrades gracefully as more server faults are tolerated Operations-based interface n Provides an interface similar to RSMs n Exports interfaces comprised of deterministic methods n Queries n Do not modify data n Updates n Modify data n Multi-object updates n Allow a set of objects to be updated together Properties n Operates correctly under an asynchronous model n Queries and updates are strictly serializable n In benign execution, they are obstruction-free n Cost is an increase in the number of required servers 5b + 1 servers, rather than 3b + 1 servers Optimism n Servers store a version history of objects n Updates are non-destructive to the objects n Use of logical timestamps based on contents of update and object state upon which the update is conditioned Speedups n Preferred quorum, rather than random quorum n Addressed later n Efficient cryptographic techniques n Addressed later Efficiency and Scalability Efficiency n Most failure atomic protocols require at least a 2 phase commit n Prepare n Commit n The optimistic approach does not need a prepare phase n This introduces the need for clients to repair inconsistent objects n The optimistic approach also obviates the need for locking! Versioning Servers n In order to allow for this, versioning servers are employed n Each update creates a new version on the server n Updates contain information about the version to be updated. n If no update has been committed since that version, the update goes through unimpeded. Throughput-scalability n Additional servers, beyond those necessary to provide the desired fault tolerance, can provide additional throughput Scaleup pitfall? n Encourage the use of fine-grained objects, which reduce per-object contention n If majority of accesses access individual objects, or few objects, then scaleup pitfall can be avoided n In the example applications, this holds. No need to partition n Other systems achieve throughput- scalability by partitioning services n This is unnecessary in this system The Query/Update Protocol System model n Asynchronous timing n Clients and servers may be Byzantine faulty n Clients and servers assumed to be computationally bounded, assuring effectiveness of cryptography n Failure model is a hybrid failure model n Benign n Malevolent n Faulty System model n Extends definition of “fail prone system” given by Malkhi and Reiter System model n Point-to-point authenticated channels exist between all clients and servers n Infrastructure deploying symmetric keys on all channels n Channels are assumed unreliable n …but, of course, they can be made reliable Overview n Clients update objects by issuing requests stamped with object versions to version servers. n Version servers evaluate these requests. n If the request is over an out of date version, the clients version is corrected and the request reissued n If an out of date server is required to reach a quorum, it retrieves an object history from a group of other servers n If the version matches the server version, of course, it is executed n Everything else is a variation upon this theme Overview n Queries are read only methods n Updates modify an object n Methods exported take arguments and return answers n Clients perform operations by issuing requests to a quorum n A server receives a request. If it accepts it it invokes a method n Each update creates a new object version Overview n The object version is kept with its logical timestamp in a version history called the replica history n Servers return replica histories in response to requests n Clients store replica histories in their object history set, an array of replicas indexed by server Overview n Timestamps in these histories are candidates for future operations n Candidates are classified in order to determine which object version a method should be executed upon Overview n In non-optimistic operation, a client may need to perform a repair n Addressed later n To perform an operation, a client first retrieves an object history set. The clients operation is conditioned on this set, which is transmitted with the operation. Overview n The client sends this operation to a quorum of servers. n To promote efficiency, the client sends the request to a preferred quorum n Addressed later n Single phase operation hinges on the availability of a preferred quorum, and on concurrency-free access. Overview n Before executing a request, servers first validate its integrity. n This is important, servers do not communicate object histories directly to each other, so the client’s data must be validated. n Servers use authenticators to do this, lists of HMACs that prevent malevolent nodes from fabricating replica histories. n Servers cull replica histories from the conditioned on OHS that they cannot validate Overview – the last bit n Servers validate that they do not have a higher timestamp in their local replica histories n Failing this, the client repairs n Passing this, the method is executed, and the new timestamp created n Timestamps are crafted such that they always increase in value Preferred Quorums n Traditional quorum systems use random quorums, but this means that servers frequently need to be synced n This is to distribute the load n Preferred quorums choose to access servers with the most up to date data, assuring that syncs happen less often Preferred Quorums n If a preferred quorum cannot be met, clients probe for additional servers to add to the quorum n Authenticators make it impossible to forge object histories for benign servers n The new host syncs with b+1 host servers, in order to validate that the data is correct n In the prototype, probing selects servers such that the load is distributed using a method parameterized on object ID and server ID Concurrency and Repair n Concurrent access to an object may fail n Two operations n Barrier n Barrier candidates have no data associated with them, and so are safe to select during periods of contention n Barrier advances the logical clock so as to prevent earlier timestamps from completing n Copy n Copies the latest object data past the barrier, so it can be acted upon Concurrency and Repair n Clients may repeatedly barrier each other, to combat this, an exponential backoff strategy is enforced Classification and Constraints n Based on partial observations of the global system state, an operation may be n Complete n Repairable n Can be repaired using the copy and barrier strategy n Incomplete Multi-Object Updates n In this case, servers lock their local copies, if they approve the OHS, the update goes through n If not, a multi-object repair protocol goes through n In this case, repair depends on the ability to establish all objects in the set n Objects in the set are only repairable if all are repairable. If objects in the set that would be repairable are reclassified as incomplete. An example of all of this Implementation details Cached object history set n Clients cache object history sets during execution, and execute updates without first querying. n Failing the request based on an out of date OHS, the server returns an up-to-date OHS with the failure Optimistic query execution n If a client has not accessed an object recently, it is still possible to complete in a single phase. n Servers execute the update on the latest object that they store. Clients then evaluate the result normally. Inline repair n Does not require a barrier and copy n Repairs the candidate “in-place,” obviating the need for a round trip n Only possible in cases where there is no contention Handling repeated requests n Mechanisms may cause requests to be repeated n In order to shortcut other checks, the timestamp is checked first Retry and backoff policies n Update-update requires retry, and backoff to avoid livelock n Update-query does not, the query can be updated in place Object syncing n Only 1 server needs to send the entire object version state n Others send hashes n Syncing server then calculates hash and comparers against all others Other speedups n Authenticators n Authenticators use HMACs rather than digital signatures n Compact timestamps n Hashes are used rather than object histories in timestamps using a collision resistant hash n Compact replica histories n Replica histories are prune based on the conditioned-on timestamp after updates Malevolent components n The astute among you must have noticed the possibility of DOS attacking by refusing exponential backoff n Servers could rate-limit clients n Clients could also issue updates to a subset of a quorum, forcing incomplete updates n Lazy verification can be used to verify correctness of client operations in the background n The amount of unverified work by a client can then be limited Correctness n Operations are strictly serializable n To understand, consider the conditioned-on chain. n All operations chain back to the initial candidate, and a total order is imposed through on all established operations n Operations occur atomically, including those spanning multiple objects n If no operations span multiple objects, then correct operations that complete are also linearizable Tests n Tests performed on a rack of 76 Intel Pentium 4 2.8 GHz machines n Implemented an “increment” method and an NFSv3 metadata service Fault Scalability More fault-scalability Isolated vs Contending NFSv3 metadata References n Text and images have been borrowed directly from both papers.
Pages to are hidden for
"Justin"Please download to view full document