Towards a Unified Theory of Replication

Reviews
Shared by: theoryman
Categories
Tags
Stats
views:
27
rating:
not rated
reviews:
0
posted:
10/30/2008
language:
English
pages:
0
Towards a Unified Theory of Replication Mike Dahlin, Lei Gao, Amol Nayate, Praveen Yalagandula, Jiandan Zheng Department of Computer Sciences University of Texas at Austin August 1, 2005 Department of Computer Sciences, UT Austin Why a Unified Theory of Replication? (1) Better way to build replication systems (2) Way to build better replication systems August 1, 2005 Department of Computer Sciences, UT Austin Better Way to Build Replication Systems Separate mechanism from policy § Continuum of policies v. point solutions Simpler to design and deploy § Replication microkernel or toolkit Integrate disparate theories/protocols § Quorums, client-server, leases, server replication, p2p, … Simplify teaching § A few principles v. a bunch of case studies Goal: Reduce the development effort for a new replication system by an order of magnitude Department of Computer Sciences, UT Austin August 1, 2005 A Way to Build Better Replication Systems Sync Palmtop/Laptop Synchronization time (normalized) 120 100 100 80 60 41 40 20 1 0 PR AC TI Cl ien t/S er ve r PR AC TI Cl ien t/S er ve r PR AC TI Cl ien t/S er ve r PR AC TI Cl ien t/S er ve r Ba yo u Ba yo u Ba yo u Ba yo u 100 100 100 1 1 3.04 1 1.04 Plane (None) Hotel (Modem) Home (DSL) Office (802.11g) Synchronize palmtop to laptop • Client-server: Limited by network to server • Bayou: Limited by fraction of shared data (1%) Department of Computer Sciences, UT Austin Order of magnitude improvements available! August 1, 2005 Outline Case for a unified theory of replication PRACTI: A first step Evaluation Future directions August 1, 2005 Department of Computer Sciences, UT Austin Case for a Unified Theory of Replication* Current systems entangle mechanism with policy • E.g., Coda v. Bayou • 14 OSDI/SOSP papers in 10 years § New environment à new trade-offs à new mechanisms § Not clear new systems dominate old ones (or that 14 is “enough”) Current literature fragmented Impact • Client-server v. quorums v. server replication v. p2p v. … • E.g., Coda and Bayou each have separate server-replication and client-server caching protocols • Systems narrowly tailored for specific environments • Significant effort to develop system for new environment * Scope: “Large scale” replication • WAN, mobile, enterprise, etc. • File systems, tuple stores, databases, distributed objects, … Department of Computer Sciences, UT Austin August 1, 2005 Vision: Replication Microkernel/Toolkit WAN Personal Enterprise Universal Policy FS FS FS Replication Core … Policy Mechanism Grand Challenges: • Each large-scale FS from OSDI/SOSP 1990-2005 as <1000-line “policy layer” • “Universal policy” – self-tuning replication • Control replication to meet high level goals • e.g., “Minimize response time and maximize availability while providing causal consistency and less than 1 minute staleness to all replicas while using less than 2x demand-read traffic.” Department of Computer Sciences, UT Austin August 1, 2005 Outline Case for a unified theory of replication PRACTI: A first step Evaluation Future directions August 1, 2005 Department of Computer Sciences, UT Austin “Towards” a Unified Theory Not there yet • Today: PRACTI • Unify large part of design space (almost) § Client-server (e.g., NFS, Coda, AFS) § Server replication (e.g., Bayou, TACT) § Object replication (e.g., Ficus, Pangea) • Future work to incorporate § Quorums, general model of security, DHT-based P2P, content-keyed identifiers, … August 1, 2005 Department of Computer Sciences, UT Austin Challenge: PRACTI Replication Client-Server AFS NFS Provide guarantees Replicate any PRACTIsubset of data required by application Ficus to any node Don’t pay for moreTACT Pangea GFS Bayou guarantees than needed WinFS (?) Server Replication Arbitrary Consistency CODA Partial Replication Topology Independence Any node can communicate with any other node Object Replication August 1, 2005 Department of Computer Sciences, UT Austin PRACTI Design Overview (0) Start with Bayou • Log-based p2p update exchange • (Could also go in other direction – generalize client/server…) • Separate streams for invalidations and bodies • Challenge: Synchronize these streams • Imprecise invalidations • Challenge: Track “precise” and “imprecise” data (1) Separate data from metadata (2) Summarize unneeded metadata (3) Separate mechanism from policy • Core: PRACTI mechanisms • Controller: Policy August 1, 2005 Department of Computer Sciences, UT Austin Step 0: Start With Bayou Write = Node A Node B Log Checkpoint … … Updates to log Log exchange for updates • Local checkpoint for random access üTI: Pairwise exchange with any peer üAC: Prefix property, causal consistency, eventual consistency ÒPR: All nodes store all data, see all updates Department of Computer Sciences, UT Austin August 1, 2005 Step 1: Separate Data and Metadata Node =A accept = <10,A>> bar=<11,A> > Node B baz, accept = <20,B>> foo=<10,A> bar=<11,A> baz=<20,B> bur=<21,B> INVALID INVALID INVALID INVALID bur=<21,B> > Separate data and metadata Log exchange: • Metadata: Log invalidations • Data: Store update bodies in checkpoint • Send invalidations separate from bodies → Client-server/Server-replication hybrid Department of Computer Sciences, UT Austin August 1, 2005 Issue: Reading Bodies Node A foo=<10,A> bar=<11,A> Node B Node C baz=<20,B> Prepush bar Bar=<11,A> foo=<10,A> bar=<11,A> baz=<20,B> bur=<21,B> INVALID INVALID INVALID INVALID Read bur bur=<21,B> bur=<21,B> Mechanism: Block until data VALID Policy: Your choice • Demand read miss • Prefetch • VALID = body matches latest invalidation § Target is policy choice: client/server, DHT directory, original writer, random, … § TCP-Nice based self-tuning prefetch Department of Computer Sciences, UT Austin August 1, 2005 Issue: Synchronization of Separate Streams Node A foo=<10,A> bar=<11,A> Node B Node C baz=<20,B> foo=<10,A> INVALID bar=<11,A> baz=<20,B> INVALID bur=<21,B> bur=<21,B> Read bar Bar=<11,A> Prepush bur <21,B> Node D foo=<3,Q> foo=<10,A> INVALID bar=<2,A> INVALID bar=<11,A>INVALID baz=<1,B> baz=<20,B> INVALID bur=<1,Q> bur=<21,B> Retrieved body may be newer than metadata → Violate causality → Buffer body until apply associated inval Department of Computer Sciences, UT Austin August 1, 2005 Step 1 Helps… Keep good Bayou properties • Topology independence • Arbitrary consistency § Prefix property § Causal consistency § Eventual consistency Step towards partial replication • Nodes only see bodies of interest § Order of magnitude improvement! • Nodes still see all invalidations § Limits scalability – E.g., Enterprise file system in which every palmtop sees every update by any node Department of Computer Sciences, UT Austin August 1, 2005 Step 2: Imprecise Invalidations Nodes subscribe for Precise invalidation • Precise invalidations for interest sets • Imprecise invalidations for other data Imprecise invalidation • Metadata for one write • Summary of multiple writes • “One or more objects in objectSet were modified between start and end” Department of Computer Sciences, UT Austin August 1, 2005 Imprecise Invalidations à • Nodes subscribe to invalidation streams § Specify which Interest Sets node wants to keep precise § Imprecise Interest Set → Replace collection of invalidations with conservative approximation – Recvr. treats all objects in objSet as if invalidated between start and end • Bookkeeping details (see paper) § Track which Interest Sets are missing invalidations § Block reads to imprecise Interest Sets § Make interest set precise when missing invalidations applied August 1, 2005 Department of Computer Sciences, UT Austin Step 3: Separate Mechanism v. Policy WAN FS Personal Enterprise FS FS PRACTI … Policy Mechanism Goal: Common core mechanism • “Replication microkernel” • Vision: § Implement replication system for new environment in <1000 lines of policy code August 1, 2005 Department of Computer Sciences, UT Austin Core v. Controller Local API (read(), write(), delete()) PRACTI Core Body Streams Apply Body Interface Apply Inval Interface Local Interface Random Access State Send Body Body Streams Inval Streams Log Control Interface Send Inval Inval Streams Inform Requests to remote cores Mgmt. Requests from remote controllers Controller Core: Mechanism • Safety: Any message can be processed at any time § Asynchronous message passing style Controller: Policy • Liveness: Trigger messages between nodes Department of Computer Sciences, UT Austin August 1, 2005 Controller Interface Notified of key events • • • • • Stream begin/end Invalidation arrival Body arrival Local read miss … Directs communication among cores Local housekeeping • Subscribe to inval or body stream • Request demand read body • Log garbage collection • Cache replacement Department of Computer Sciences, UT Austin August 1, 2005 Example: Client-Server Controller Subscriptions • Precise invalidations § Forall f in subscribe to f from server • Bodies § Forall h in subscribe to h from server Local read miss on file f if(f is imprecise) request metadata + body from server else /* f is precise but invalid */ request body from server (read blocks until f is precise and valid) Point of interest perhaps only to me • Client/server crash recovery really natural/elegant Department of Computer Sciences, UT Austin August 1, 2005 Example: EnterpriseFS Controller Support thousands of devices • Handful of big, geographically distributed servers • Many desktops, laptops, palmtops, etc. Read miss • Use DHT to find nearest copy of data Replication policy • DHT tracks file popularity § Self-tuning prefetch important updates to where they are/will be needed • Enforce minimum replication degree for reliability and availability Details TBD… Department of Computer Sciences, UT Austin August 1, 2005 PRACTI Design Summary Result: Subsume many existing mechanisms • Client/server*: Coda, NFS, AFS, … • Server replication: Bayou, TACT • Object replication: Ficus, Pangea, … Key ideas (1) Separate data from metadata (2) Summarize unneeded metadata § Separate streams for invalidations and bodies § Challenge: Synchronize these streams (3) Separate mechanism from policy § Core: PRACTI mechanisms § Controller: Policy § Imprecise invalidations § Challenge: Track “precise” and “imprecise” data August 1, 2005 Department of Computer Sciences, UT Austin Additional Details Efficient, continuous update exchange Garbage collect logs • Incremental log exchange • Incremental checkpoint exchange using lpVV data structures Self-tuning replication Continuous consistency (e.g., TACT) • Prefetch/pre-push bodies over low-priority network channel • Causal consistency by default • Weaken: Imprecise reads (causal coherence) • Strengthen: Constraints layer • Flexible conflict detection and resolution • Bound invalidations § Order error, temporal error, numerical error Enforce minimum replication for availability See paper for details Department of Computer Sciences, UT Austin August 1, 2005 Outline Case for a unified theory of replication PRACTI: A first step Evaluation • Methodology • Benefits of partial replication • Benefits of topology independence • Cost of supporting flexible consistency Future directions August 1, 2005 Department of Computer Sciences, UT Austin Methodology How to evaluate “Unified theory”? August 1, 2005 Department of Computer Sciences, UT Austin Partial Replication 1e+07 Full Replication Bytes Transferred 1e+06 Separate Invalidations/Data 100000 10000 0.1 Imprecise Invalidations 1 Files of Interest (%) 10 100 Order of magnitude improvements • Both separate inval v. body AND imprecise inval • Storage requirements see similar improvements Department of Computer Sciences, UT Austin August 1, 2005 Topology Independence 10 Mb/s 56 Kb/s 0 Mb/s 1 Mb/s Machines Places • Laptop, palmtop, home desktop, office server • Office, home, hotel, plane Department of Computer Sciences, UT Austin August 1, 2005 Palmtop/Laptop Sync Time Sync Palmtop/Laptop Synchronization time (normalized) 120 100 100 80 60 41 40 20 1 0 Ba yo u Ba yo u Ba yo u PR AC TI Cl ien t/S er ve r PR AC TI Cl ien t/S er ve r PR AC TI Cl ien t/S er ve r PR AC TI Cl ien t/S er ve r Ba yo u 100 100 100 1 1 3.04 1 1.04 Plane (None) Hotel (Modem) Home (DSL) Office (802.11g) Synchronize palmtop to laptop • Client-server: Limited by network to server • Bayou: Limited by fraction of shared data (1%) Department of Computer Sciences, UT Austin August 1, 2005 PlanetLabFS Simplify running experiments • Track current locations of files via DHT • Flood initial data, programs from server to clients via cooperative caching • Direct transfer of data updates among clients via cooperative caching • Future: Self-tuning prefetching Benchmark • Phase 1 Disseminate: • Phase 2 Process: § Disseminate 10MB from server to all clients § 10x pairwise exchange 1MB between random clients • Phase 3 Post-Process: § Gather 1MB from each client to server Department of Computer Sciences, UT Austin August 1, 2005 PlanetLabFS 1000 915 250 221 800 600 475 Post-process Process 200 150 100 71 Time(s) 400 282 Disseminate 177 Time(s) 200 0 50 24 Post-process Process Disseminate 37 PLFS Bayou CoopClient/Server 0 Distributed Nodes Remote Cluster PLFS Bayou Coop Client/Server • 3x-5x v. client-server (dissemination) • 2.4x-9x v. server replication (process, postprocess) • 1.5x v. cooperative caching (process) • TBD: Add self-tuning prefetching Department of Computer Sciences, UT Austin August 1, 2005 Cost of Consistency Tunable consistency • Causal, causal + TACT, sequential, linearizable • Consistent or coherent § Consistency: Order writes across all objects § Coherence: Order writes to individual objects PRACTI benefits • Semantics specified on per-read, per-write basis § What information must a read or write wait for to complete? → No unnecessary read delay or write delay • Separation of invalidations from bodies → Minimize delay (hence inconsistency) August 1, 2005 Department of Computer Sciences, UT Austin Improved Consistency Trade-Offs 1 Periodic (500s) 0.1 Average Unavailability 0.01 TACT-Aggressive PRACTI-Demand 0.001 PRACTI-Prefetch 0.0001 0 0.5 1 1.5 2 Available Bandwidth/Write Bandwidth Bayou How Batch When Periodic Invals All Bodies All TACT-Aggressive Batch Frequent All All Department of Computer Sciences, UT Austin PRACTI Incremental Continuous All* Self-tuning August 1, 2005 Cost of Consistency v. Coherence Suppose I care about subset of data • /A/* but not /B/*, /C/*, or /D/* PRACTI • Precise invalidations for /A/* • Imprecise invalidations for the rest Imprecise invalidations: “Placeholders” • Allow future reads/writes to be consistently ordered with writes to /B/*, /C/*, /D/* if desired § Locally or at other nodes • System that only guarantees coherence and never provides option of consistency could omit imprecise invalidations • Worst case: Each precise invalidation paired with imprecise invalidation summarizing writes on which it depends • How much overhead do these imprecise invalidations impose on nodes that don’t use them? Department of Computer Sciences, UT Austin August 1, 2005 Cost of Consistency 60 All precise 50 Prec+Imp (burst=10) Inval Bytes Per Write 40 Prec+Imp (no locality) 30 Coherence only 20 10 0 0 0.2 0.4 0.6 0.8 1 Interest Set Fraction Imprecise invalidations save v. all-precise Imprecise invalidations cost v. coherence only • Worst case 2:1 (messages) • Locality reduces cost August 1, 2005 Department of Computer Sciences, UT Austin Performance Summary Better trade-offs • Partial replication of data • Partial replication of metadata • Topology independence • Minimal consistency cost Additional benefits (see paper) • Self-tuning replication of bodies • Incremental checkpoint transfer August 1, 2005 Department of Computer Sciences, UT Austin Outline Motivation PRACTI Protocol Evaluation Future Work/Conclusions • Towards a unified theory and practice August 1, 2005 Department of Computer Sciences, UT Austin Questions PRACTI doesn’t answer • Does PRACTI reduce development costs by 10x? • Can we support quorums, client-server, server replication, p2p on the same substrate? • Can we efficiently support callbacks and leases? • How do various consistency paradigms relate? § FIFO, causal, sequential, linearizable, etc. v. Reads follow writes, monotonic reads, etc. v. Safe, regular, atomic, etc. § Can we support 14 OSDI/SOSP papers in <1000 LOC each? • What are the “core mechanisms” for security? • Can we support FS, tuple store, and DB on same substrate? • Can we unify other “large scale” replication systems (e.g., cluster)? Department of Computer Sciences, UT Austin August 1, 2005 Conclusion Build your next large-scale replication system using PRACTI • A better way to build replication systems • A way to build better replication systems Details on my web page “PRACTI Replication for Large-Scale Systems,” M. Dahlin, L. Gao, A. Nayate, A. Venkataramani, P. Yalagandula J. Zheng "Dual-Quorum Replication for Edge Services," L. Gao, M. Dahlin, J. Zheng, L. Alvisi, A. Iyengar, Middleware 2005 "Transparent Information Dissemination," A. Nayate, M. Dahlin, A. Iyengar, Middleware 2004. "A Non-interfering Deployable Web Prefetching System," R. Kokku, P. Yalagandula, A. Venkatramani, M. Dahlin,USITS 2003 August 1, 2005 Department of Computer Sciences, UT Austin
Related docs
Towards a Unified Theory of Economic Growth
Views: 191  |  Downloads: 16
Towards a unified theory of the process
Views: 34  |  Downloads: 0
Towards a unified theory of the process
Views: 2  |  Downloads: 0
Towards a unified theory of the process
Views: 1  |  Downloads: 0
Towards a Unified Theory of Economic Growth Oded Galor on the ...
Views: 55  |  Downloads: 5
Towards a Unified Theory of State Abstraction for MDPs
Views: 21  |  Downloads: 0
towards goce
Views: 0  |  Downloads: 0
Data-Oriented Belief Revision Towards a Unified Theory of ...
Views: 7  |  Downloads: 0
Towards a Theory of Volatility Trading
Views: 126  |  Downloads: 14
Towards the theory of alter-globalism
Views: 42  |  Downloads: 1
Towards a Theory of Digital Preservation
Views: 27  |  Downloads: 1
Towards a Stakeholder Theory of Strategic Management
Views: 203  |  Downloads: 14
TOWARDS A DYNAMIC THEORY OF STRATEGY
Views: 371  |  Downloads: 17
A Essay Towards a New Theory of Vision
Views: 2  |  Downloads: 1
premium docs
Perfect Competition and Its Impact on business$131.40
Case Study in Gender and Family Health$43.80
Coca Cola Marketing Mix$76.65
Chemistry Comes Alive - Quick Review$2.00
Human Skeletal System - Quick Review$2.00
Lab Techniques: Introduction to Organic Chemistry$2.25
Fifteen Amendments Impact and Influence$65.70
Other docs by theoryman
Sexual Harassment Policy2
Views: 242  |  Downloads: 4
Burlington Northern Santa Fe Inc Ammendments and By laws
Views: 159  |  Downloads: 0
Telecommuting Checklist and Agreement
Views: 412  |  Downloads: 13
CorpDocs-Board Resolution Advising Approval of Merger
Views: 220  |  Downloads: 2
Contract Checklist
Views: 567  |  Downloads: 40
ASSIGNMENT OF MONEY DUE
Views: 250  |  Downloads: 2
0206 Form W-3C (PR) (PDF) Transmision de Comprobantes de Retencion Corregidos
Views: 259  |  Downloads: 1
Loan Application Bank Review Form
Views: 543  |  Downloads: 10
Overtime Policy Guidance
Views: 769  |  Downloads: 18
Form 5305-SEP Simplified Employee Pension - Individual Retirement Accounts Contribution Agreement
Views: 568  |  Downloads: 2
Employee Complaint Procedures Re Sarbanes Oxley
Views: 293  |  Downloads: 5
Instinctive Technologies Inc Ammendments and Bylaws
Views: 185  |  Downloads: 0
Articles of Incorporation California
Views: 365  |  Downloads: 9
Shareholders Resolution For Blanket Authority to Sell Corporate Property
Views: 259  |  Downloads: 5
Batmobile Top
Views: 596  |  Downloads: 4