"Objectivity - HPSS Integration"
Marcin Nowak RD45 Project IT / Object Databases Objectivity/DB - HPSS Interface Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 1 Interface Features Direct (POSIX) interface to HPSS abandoned due to low performance Staging interface currently used version requires staging pool management AMS-like performance for databases in the staging pool HPSS-like performance for databases on tape RPC timeouts need to be addressed Extended (extensible) AMS 5.2 AMS protocol extended with GAP, OIP, RRP and DRP multi-threaded only one AMS version - no special linkable AMS uses shared libraries (liboofs.so, libooss.so, librfio.so) Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 2 Direct HPSS Interface Standard Objectivity/DB configuration in distributed environment Objy Page Server Objectivity Network Layer AMS Client Network transfer Application in DB pages Filesystem DB Access Layer files DB pages Objectivity/DB with HPSS FS level interface Objectivity Network AMS DB pages Layer Client Application INTERFACE HPSS DB pages HPSS client Server Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 3 Staging Interface Overview Migration Migration (write) request daemon DB files transfer Stage Pool HPSS UNIX FS Server DB pages transfer AMS with HPSS Staging (read) request interface Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 4 Software Components OOAMS 5.2 executable liboofs.so liboss.so librfio.so (multithreaded, HPSS aware version) other shared libraries e.g. security ooss_Stage.pl script moving files from HPSS to the local disk (stage pool) Migration/Purge daemon ooss_MigrPurg.pl migrating idle database files from the local disk to HPSS maintaining stage pool free disk space RFIO commands rfcp, rfmv Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 5 AMS Structure AMS 4.0.10 AMS 5.0 AMS 5.0 AMS 5.2 build 56 build 58 OOFS v2 OOFS v3 OOFS v4 OOFS v5 OOSS HPSS RFIO Other interface RFIO UNIX FS Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 6 AMS/HPSS Interface at CERN CERN version of AMS/HPSS interface is using RFIO HPSS servers are running a special RFIO daemon used by SHIFT and Castor RFIO eliminates the need to link with HPSS libraries no need for DCE on the host running the interface porting to different platforms is easier no performance degradation compared to PFTP full use of the HPSS parallel, point-to-point transfer protocol 2GB file size limit current RFIO library does not support HPSS SetCOS call Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 7 Staging Interface with RFIO AMS File & catalog RFIO calls RFIO management daemon Stage-in requests HPSS UNIX FS I/O Server DB pages Disk Migration HPSS Mover Server daemon (Solaris) Tape Robot Disk Pool RFCP (RFIO copy) Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 8 Space Management in the Staging Pool Space in the disk pool is controlled by the migration and purge daemon (Perl) Parameters of migration and purge are set via configuration file The process is inactive between space checks, but can be activated by AMS if free space is low File migration is done in parallel File locking is used to synchronize with AMS additional lock file for every database Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 9 Control Scripts startMaster - start master script starts AMS and migration/purge daemon stopAll - stops master script stops migration/purge daemon and tires to stop AMS rereadConf - reread configuration file suspendMigr - stop migrating files suspendPurg - stop deleting files restartMigr - start migrating files restartPurg - start deleting files Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 10 Configuration Single configuration file for AMS and migration daemon specified by OOAMS_CONFIG environmental variable Dynamic configuration refresh on the SIGCONT signal Configurable parameters: local and remote root paths list of directories to scan for files to migate file purge thresholds (in percent of disk space) migration and purge frequency and delays selection of file transfer method (PFTP/RFCP) debug levels, log file location and other Timeout problems set OO_RPC_TIMEOUT environment variable Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 11 Database File Names Elements from the interface configuration file ($OOAMS_CONFIG) hpss_server local_root remote_root Elements from FDB catalog: host_name db_pathname Database name in: FDB catalog: host_name::db_pathname „host_name‟: db_pathname HPSS: hpss_server:remote_root/ db_pathname local_root is removed from db_pathname Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 12 Example of Path Handling Configuration values local_root: /datapool hpss server: hpsssrv1 remote root: /hpss/cern.ch/cms Command oonewdb -host dbsrv1 -filepath /datapool/test/mydb.DB -db mydb Database names FDB catalog: dbsrv1::/datapool/test/mydb.DB on dbsrv1: /datapool/test/mydb.DB in HPSS: hpsssrv1:/hpss/cern.ch/cms/test/mydb.DB Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 13 Protocol Enhancements in 5.2 Generic Authentication Protocol (GAP) allows security information to be transferred from client application to the AMS supports private and public key mechanisms information is used by most AMS operations Opaque Information Protocol (OIP) allows arbitrary information to be transferred from client application to the AMS useful for passing hints to HPSS (like Class Of Service) used when a database is opened Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 14 Protocol Enhancements in 5.2 Defer Request Protocol (DRP) allows an AMS to control the timeout of a client‟s request designed to deal with filesystems with variable latencies handled internally by Objectivity should make the use of AMS/HPSS interface completely transparent to the client application Request Redirection Protocol (RRP) allows a group of AMS‟ s to perform load balancing using a redirection scheme assumes that such a group has a single “master” AMS that is always contacted first and redirects the request Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 15 Current Activities Testing Objy/HPSS interface with CMS production software ~Feb 14 starts large scale simulation on ~40 CPUs ~Mar 7 starts ORCA production. Initial step is transfer of cmsim fz zebra format into ooHit format in Objy, ~2TB Linux version Synchronization with SLAC latest release remote run-time AMS monitor load-balancing disk pool manager (?) Implement request deferring Test other AMS protocol enhancements security Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 16 Distribution Complete package for Solaris 2.6 /afs/cern/ch/rd45/amshpss Self-sufficient does not need local Objectivity installation uses /usr/spool/objy directory and /etc/services Objectivity Workshop 02.2000 Marcin Nowak IT/DB/RD45 17