VIEWS: 11 PAGES: 20 POSTED ON: 12/18/2009
File System Access (XRootd) Andrew Hanushevsky Stanford Linear Accelerator Center 13-Jan-03 Goals High Performance File-Based Access Scalable Extensible Mass Storage integration Flexible Security Dynamically Configurable Rootd Compatable Jauuary 13, 2003 2: XRootd Achieving High Performance Multi-threaded server Ability to run multiple servers on one host Last active connections serviced first Socket/thread bound for “active” connections Unlocked I/O, sync reduction, file descriptor partitioning 3: XRootd MRU polling Sticky sockets Using high-performance oofs layer Jauuary 13, 2003 Achieving Scalability Socket multiplexing One connection per client/host Requires stream id protocol extensions Dynamic Load Balancing Requires client redirection protocol extensions File reuse, idle closeouts, cached file system Using scalable oofs layer Jauuary 13, 2003 4: XRootd Achieving Extensibility Opaque Information Protocol Protocol extension allows client to relay outboard information to server Messaging interface Cache group Request priority Other things we haven’t thought of yet… Jauuary 13, 2003 5: XRootd Achieving MSS Integration Using oofs layer Already integrates with almost any MSS No brainer Jauuary 13, 2003 6: XRootd Achieving Flexible Security Negotiated Security Protocol Extensions Allows client/server to agree on protocol E.g., Kerberos, GSI, AFS Kerberos, etc. Using oofs layer Multi-protocol authentication support Scalable authorization Jauuary 13, 2003 7: XRootd Achieving Configurability Servers must come up/down without harm Part of dynamic load balancing Requires client retry extension Jauuary 13, 2003 8: XRootd Achieving Compatability Client sensitive to server version number Rootd servers default to base TNetFile class XRootd servers use derived TNetFile class Server requires client to turn on extensions Base TNetFile class restricted to rootd ops Derived TNetFile enables extensions Multi-streaming, Redirection, Request Deferral, Generic Security, Opaque Information 9: XRootd Jauuary 13, 2003 xrootd Architecture Database Protocol Layer Filesystem Logical Layer Filesystem Physical Layer Filesystem Implementation Jauuary 13, 2003 10: XRootd Some Details Compatible Protocol Allows transparent use of rootd & xrootd Allows multiple files per connection Multi-Streaming Cache File System Dynamic Load Balancing Client directed to “appropriate” server Configurability & Recoverability Jauuary 13, 2003 11: XRootd Compatible Protocol Application XTNetFile TNetFile xrootd xrootd rootd compability rootd Jauuary 13, 2003 12: XRootd Multi-Streaming Each open() assigned a stream number Four character id that starts with 0-9 All requests/responses identify relevant stream Non-open file requests use arbitrary id Any four characters not starting with 0-9 E.g., stat(), mv(), unlink(), etc. Parallel Multi-Streaming allowed Client must explicitly enable it 13: XRootd Jauuary 13, 2003 Cache File System /databases/mydbfile Index Area Optional data cache Default data area symlink Naming convention allows for audit and index recovery Multiple Independent Filesystems /cache1/databases:mydbfile /cache2 /cache3 Jauuary 13, 2003 14: XRootd Data Area Any number Any Size Chosen based on free space in LRU order Dynamic Load Balancing Dynamic Selection Jauuary 13, 2003 15: XRootd DLB Implementation xrootd dlbd subscribe open again open Client wait try host:port Jauuary 13, 2003 xrootd dlbd I do xrootd dlbd (any number) who has the file? dlbd xrootd (any number) 16: XRootd Configurability & Recoverability Go hand in hand Configuration changes may be disruptive Must be able to recover from dynamic configuration changes Dynamic Load Balancing essential part Jauuary 13, 2003 17: XRootd Recoverability Whenever client looses r/o connection Back to distinguished xrootd(s) for reselection Whenever client looses r/w connection Limited wait/retry loop on the same server We will be working to improve this next year! All handled in the XTNetFile class Disruptions merely delay the client Jauuary 13, 2003 18: XRootd Configurability Since we have recoverability… Servers may come and go New servers can be added at any time Servers may be brought down for maintenance Files can be moved around in real-time Client will recover and adjust to the new configuration Jauuary 13, 2003 19: XRootd Conclusion Goals for rootd file access can be met Most of this work relies on leveraging work done for the AMS Compatability allows us to lobby the community for xrootd to become the default “rootd” Jauuary 13, 2003 20: XRootd
"File System Access"